<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/diving-into-dynamic-website-content-with-splashr/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/02/09/diving-into-dynamic-website-content-with-splashr/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>If you do enough web scraping, you’ll eventually hit a wall that the trusty <code>httr</code> verbs (that sit beneath <code>rvest</code>) cannot really overcome: dynamically created content (via javascript) on a site. If the site was nice enough to use XHR requests to load the dynamic content, you can generally still stick with <code>httr</code> verbs — if you can figure out what those requests are — and code-up the right parameters (browser “Developer Tools” menus/views and my <code>curlconverter</code> package are super handy for this). Unfortunately, some sites require actual in-page rendering and that’s when scraping turns into a modest chore. </p>
<p>For dynamic sites, the <code>RSelenium</code> and/or <code>seleniumPipes</code> packages are super-handy tools to have in the toolbox. They interface with <a target="_blank" rel="nofollow" href="http://www.seleniumhq.org/">Selenium</a> which is a feature-rich environment/ecosystem for automating browser tasks. You can programmatically click buttons, press keys, follow links and extract page content because you’re scripting actions in an actual browser or a browser-like tool such as <code>phantomjs</code>. Getting the server component of Selenium running was often a source of pain for R folks, but the new <a target="_blank" rel="nofollow" href="https://hub.docker.com/r/selenium/">docker images</a> make it much easier to get started. For truly gnarly scraping tasks, it should be your go-to solution.</p>
<p>However, sometimes all you need is the rendering part and for that, there’s a new light[er]weight alternative dubbed <a target="_blank" rel="nofollow" href="http://splash.readthedocs.io/en/stable/">Splash</a>. It’s written in python and uses QT webkit for rendering. To avoid deluging your system with all of the <code>Splash</code> dependencies you can use the <a target="_blank" rel="nofollow" href="http://splash.readthedocs.io/en/stable/install.html#linux-docker">docker images</a>. In fact, I made it dead easy to do so. Read on!</p>
<h3>Going for a dip</h3>
<p>The intrepid Winston Chang at RStudio <a target="_blank" rel="nofollow" href="http://github.com/wch/harbor">started a package</a> to wrap Docker operations and I’ve recently joind in the fun to add some tweaks &amp; enhancements to it that are necessary to get it on CRAN. Why point this out? Since you need to have <code>Splash</code> running to work with it in <code>splashr</code> I wanted to make it as easy as possible. So, if you <a target="_blank" rel="nofollow" href="https://www.docker.com/">install Docker</a> and then <code>devtools::install_github(&quot;wch/harbor&quot;)</code> you can then <code>devtools::install_github(&quot;hrbrmstr/splashr&quot;)</code> to get <code>Splash</code> up and running with:</p>
<pre><code>library(splashr)

install_splash()
splash_svr &lt;- start_splash()</code></pre>
<p>The <code>install_splash()</code> function will pull the correct image to your local system and you’ll need that <code>splash_svr</code> object later on to stop the container. Now, you can have <code>Splash</code> running on any host, but this post assumes you’re running it locally.</p>
<p>We can test to see if the server is active:</p>
<pre><code>splash(&quot;localhost&quot;) %>% splash_active()
## Status of splash instance on [http://localhost:8050]: ok. Max RSS: 70443008</code></pre>
<p>Now, we’re ready to scrape!</p>
<p>We’ll use this site — <a target="_blank" rel="nofollow" href="http://www.techstars.com/companies/">http://www.techstars.com/companies/</a> — mentioned over at <a target="_blank" rel="nofollow" href="https://www.datacamp.com/community/tutorials/scraping-javascript-generated-data-with-r#gs.dZEqev8">DataCamp’s tutorial</a> since it doesn’t use XHR but does require rendering and it doesn’t prohibit scraping in the Terms of Service (don’t violate Terms of Service, it is both unethical and could get you blocked, fined or worse). </p>
<p>Let’s scrape the “Summary by Class” table. Here’s an excerpt along with the Developer Tools view:</p>
<p><a target="_blank" rel="nofollow" href="https://i1.wp.com/rud.is/b/wp-content/uploads/2017/02/Techstars_Alumni_Companies.png?ssl=1"><img alt="" class="wp-image-5005" src="https://i1.wp.com/rud.is/b/wp-content/uploads/2017/02/Techstars_Alumni_Companies.png?resize=456%2C169&ssl=1" data-recalc-dims="1"></a></p>
<p>You’re saying <em>“HEY. That has <code>&lt;table></code> in the HTML so why not just use <code>rvest</code></em>? Well, you can validate the lack of <code>&lt;table></code>s in the “view source” view of the page or with:</p>
<pre><code>library(rvest)

pg &lt;- read_html(&quot;http://www.techstars.com/companies/&quot;)
html_nodes(pg, &quot;table&quot;)
## {xml_nodeset (0)}</code></pre>
<p>Now, let’s do it with <code>splashr</code>:</p>
<pre><code>splash(&quot;localhost&quot;) %>% 
  render_html(&quot;http://www.techstars.com/companies/&quot;, wait=5) -> pg
  
html_nodes(pg, &quot;table&quot;)
## {xml_nodeset (89)}
##  [1] &lt;table class=&quot;table75&quot;>&lt;tbody>\n&lt;tr>\n&lt;th>Status&lt;/th>\n        &lt;th>Number of Com ...
##  [2] &lt;table class=&quot;table75&quot;>&lt;tbody>\n&lt;tr>\n&lt;th colspan=&quot;2&quot;>Impact&lt;/th>\n      &lt;/tr>\n ...
##  [3] &lt;table class=&quot;table75&quot;>&lt;tbody>\n&lt;tr>\n&lt;th>Class&lt;/th>\n        &lt;th>#Co's&lt;/th>\n   ...
##  [4] &lt;table>&lt;tbody>\n&lt;tr>\n&lt;th class=&quot;batch_class&quot; colspan=&quot;4&quot;>Anywhere 2017 Q1&lt;/th>\ ...
##  [5] &lt;table>&lt;tbody>\n&lt;tr>\n&lt;th class=&quot;batch_class&quot; colspan=&quot;4&quot;>Atlanta 2016 Summer&lt;/t ...
##  [6] &lt;table>&lt;tbody>\n&lt;tr>\n&lt;th class=&quot;batch_class&quot; colspan=&quot;4&quot;>Austin 2013 Fall&lt;/th>\ ...
##  [7] &lt;table>&lt;tbody>\n&lt;tr>\n&lt;th class=&quot;batch_class&quot; colspan=&quot;4&quot;>Austin 2014 Summer&lt;/th ...
##  [8] &lt;table>&lt;tbody>\n&lt;tr>\n&lt;th class=&quot;batch_class&quot; colspan=&quot;4&quot;>Austin 2015 Spring&lt;/th ...
##  [9] &lt;table>&lt;tbody>\n&lt;tr>\n&lt;th class=&quot;batch_class&quot; colspan=&quot;4&quot;>Austin 2016 Spring&lt;/th ...
## [10] &lt;table>&lt;tbody>\n&lt;tr>\n&lt;th class=&quot;batch_class&quot; colspan=&quot;4&quot;>Barclays 2014&lt;/th>\n   ...
## [11] &lt;table>&lt;tbody>\n&lt;tr>\n&lt;th class=&quot;batch_class&quot; colspan=&quot;4&quot;>Barclays 2015 Spring&lt;/ ...
## [12] &lt;table>&lt;tbody>\n&lt;tr>\n&lt;th class=&quot;batch_class&quot; colspan=&quot;4&quot;>Barclays 2016 Winter&lt;/ ...
## [13] &lt;table>&lt;tbody>\n&lt;tr>\n&lt;th class=&quot;batch_class&quot; colspan=&quot;4&quot;>Barclays Cape Town 201 ...
## [14] &lt;table>&lt;tbody>\n&lt;tr>\n&lt;th class=&quot;batch_class&quot; colspan=&quot;4&quot;>Barclays NYC 2015 Summ ...
## [15] &lt;table>&lt;tbody>\n&lt;tr>\n&lt;th class=&quot;batch_class&quot; colspan=&quot;4&quot;>Barclays NYC 2016 Summ ...
## [16] &lt;table>&lt;tbody>\n&lt;tr>\n&lt;th class=&quot;batch_class&quot; colspan=&quot;4&quot;>Barclays Tel Aviv 2016 ...
## [17] &lt;table>&lt;tbody>\n&lt;tr>\n&lt;th class=&quot;batch_class&quot; colspan=&quot;4&quot;>Berlin 2015 Summer&lt;/th ...
## [18] &lt;table>&lt;tbody>\n&lt;tr>\n&lt;th class=&quot;batch_class&quot; colspan=&quot;4&quot;>Berlin 2016 Summer&lt;/th ...
## [19] &lt;table>&lt;tbody>\n&lt;tr>\n&lt;th class=&quot;batch_class&quot; colspan=&quot;4&quot;>Boston 2009 Spring&lt;/th ...
## [20] &lt;table>&lt;tbody>\n&lt;tr>\n&lt;th class=&quot;batch_class&quot; colspan=&quot;4&quot;>Boston 2010 Spring&lt;/th ...
## ...##</code></pre>
<p>We need to set the <code>wait</code> parameter (5 seconds was likely overkill) to give the javascript callbacks time to run. Now you can go crazy turning that into data.</p>
<h3>Candid Camera</h3>
<p>You can also take snapshots (pictures) of websites with <code>splashr</code>, like this (apologies if you start drooling on your keyboard):</p>
<pre><code>splash(&quot;localhost&quot;) %>% 
  render_png(&quot;https://www.cervelo.com/en/triathlon/p-series/p5x&quot;)</code></pre>
<p><a target="_blank" rel="nofollow" href="https://i0.wp.com/rud.is/b/wp-content/uploads/2017/02/cerv.png?ssl=1"><img alt="" class="wp-image-5009" src="https://i0.wp.com/rud.is/b/wp-content/uploads/2017/02/cerv.png?resize=456%2C342&ssl=1" data-recalc-dims="1"></a></p>
<p>The snapshot functions return <code>magick</code> objects, so you can do anything you’d like with them.</p>
<h3>HARd Work</h3>
<p>Since <code>Splash</code> is rendering the entire site (it’s a real browser), it knows all the information about the various components of a page and can return that in <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/.har">HAR format</a>. You can retrieve this data and use John Harrison’s spiffy <code>HARtools</code> package to visualize and further analyze the data. For the sake of brevity, here’s just the main <code>print()</code> output from a site:</p>
<pre><code>splash(&quot;localhost&quot;) %>% 
  render_har(&quot;https://www.r-bloggers.com/&quot;)

## --------HAR VERSION-------- 
## HAR specification version: 1.2 
## --------HAR CREATOR-------- 
## Created by: Splash 
## version: 2.3.1 
## --------HAR BROWSER-------- 
## Browser: QWebKit 
## version: 538.1 
## --------HAR PAGES-------- 
## Page id: 1 , Page title: R-bloggers | R news and tutorials contributed by (750) R bloggers 
## --------HAR ENTRIES-------- 
## Number of entries: 130 
## REQUESTS: 
## Page: 1 
## Number of entries: 130 
##   -  https://www.r-bloggers.com/ 
##   -  https://www.r-bloggers.com/wp-content/themes/magazine-basic-child/style.css 
##   -  https://www.r-bloggers.com/wp-content/plugins/mashsharer/assets/css/mashsb.min.cs... 
##   -  https://www.r-bloggers.com/wp-content/plugins/wp-to-twitter/css/twitter-feed.css?... 
##   -  https://www.r-bloggers.com/wp-content/plugins/jetpack/css/jetpack.css?ver=4.4.2 
##      ........ 
##   -  https://scontent.xx.fbcdn.net/v/t1.0-1/p50x50/10579991_10152371745729891_26331957... 
##   -  https://scontent.xx.fbcdn.net/v/t1.0-1/p50x50/14962601_10210947974726136_38966601... 
##   -  https://scontent.xx.fbcdn.net/v/t1.0-1/c0.8.50.50/p50x50/311082_286149511398044_4... 
##   -  https://scontent.xx.fbcdn.net/v/t1.0-1/p50x50/11046696_917285094960943_6143235831... 
##   -  https://static.xx.fbcdn.net/rsrc.php/v3/y2/r/0iTJ2XCgjBy.png</code></pre>
<h3>FIN</h3>
<p>You can also do some basic scripting in <code>Splash</code> with <code>lua</code> and coding up an interface with that capability is on the TODO as is adding final tests and enabling tweaking the Docker configurations to support more fun things that <code>Splash</code> can do.</p>
<p>File an issue <a target="_blank" rel="nofollow" href="https://github.com/hrbrmstr/splashr">on github</a> if you have feature requests or problems and feel free to jump on board with a PR if you’d like to help put the finishing touches on the package or add some features.</p>
<p>Don’t forget to <code>stop_splash(splash_svr)</code> when you’re finished scraping!</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://rud.is/b/2017/02/09/diving-into-dynamic-website-content-with-splashr/"> R – rud.is</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/CBPZ3n1zsc4"></a></div>