{"id":12162,"date":"2019-04-27T13:20:50","date_gmt":"2019-04-27T18:20:50","guid":{"rendered":"https:\/\/rud.is\/b\/?p=12162"},"modified":"2019-04-29T09:15:24","modified_gmt":"2019-04-29T14:15:24","slug":"quick-hit-scraping-javascript-enabled-sites-with-htmlunit","status":"publish","type":"post","link":"https:\/\/rud.is\/b\/2019\/04\/27\/quick-hit-scraping-javascript-enabled-sites-with-htmlunit\/","title":{"rendered":"Quick Hit: Scraping javascript-“enabled” Sites with {htmlunit}"},"content":{"rendered":"

I’ve mentioned {htmlunit}<\/code><\/a> in passing before, but did not put any code in the blog post. Since I just updated {htmlunitjars}<\/code> to the latest and greatest version, now might be a good time to do a quick demo of it.<\/p>\n

The {htmlunit}<\/code>\/{htmunitjars}<\/code> packages make the functionality of the HtmlUnit Java libray<\/a> available to R. The TLDR on HtmlUnit is that it can help you scrape a site that uses javascript to create DOM elements. Normally, you’d have to use Selenium\/{Rselenium}<\/code>, Splash\/{splashr}<\/code> or Chrome\/{decapitated}<\/code> to try to work with sites that generate the content you need with javascript. Those are fairly big external dependencies that you need to trudge around with you, especially if all you need is a quick way of getting dynamic content. While {htmlunit}<\/code> does have an {rJava}<\/code> dependency, I haven’t had any issues getting Java working with R on Windows, Ubuntu\/Debian or macOS in a very long while—even on freshly minted systems—so that should not be a show stopper for folks (Java+R guaranteed ease of installation is still far from perfect, though).<\/p>\n

To demonstrate the capabilities of {htmlunit}<\/code> we’ll work with a site that’s dedicated to practicing web scraping—toscrape.com<\/code>—and, specifically, the javascript generated sandbox site<\/a>. It looks like this:<\/p>\n

\"\"<\/a><\/p>\n

Now bring up both the “view source” version of the page on your browser and the developer tools “elements” panel and you’ll see that the content is in javascript right there on the site but the source has no <div><\/code> elements because they’re generated dynamically after the page loads.<\/p>\n\n\t\t