<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2018/08/13/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>The <a rel="nofollow" href="https://gitlab.com/hrbrmstr/splashr" target="_blank">development version of <code>splashr</code></a> now support authenticated connections to <a rel="nofollow" href="https://splash.readthedocs.io/en/stable/" target="_blank">Splash</a> API instances. Just specify <code>user</code> and <code>pass</code> on the initial <code>splashr::splash()</code> call to use your scraping setup a bit more safely. For those not familiar with <code>splashr</code> and/or Splash: the latter is a lightweight alternative to tools like Selenium and the former is an R interface to it. Unlike <code>xml2::read_html()</code>, <code>splashr</code> renders a URL exactly as a browser does (because it uses a virtual browser) and can return far more than just the HTML from a web page. Splash does need to be running and it’s best to use it in a Docker container.</p>
<p>If you have a large number of sites to scrape, working with <code>splashr</code> and Splash “as-is” can be a bit frustrating since there’s a limit to what a single instance can handle. Sure, it’s possible to setup your own highly available, multi-instance Splash cluster and use it, but that’s <em>work</em>. Thankfully, the folks behind TeamHG-Memex created <a rel="nofollow" href="https://github.com/TeamHG-Memex/aquarium" target="_blank">Aquarium</a> which uses <code>docker</code> and <code>docker-compose</code> to stand up a multi-Splash instance behind a pre-configured <a rel="nofollow" href="https://www.haproxy.org/" target="_blank">HAProxy</a> instance so you can take advantage of parallel requests the Splash API. As long as you have <code>docker</code> and <code>docker-compose</code> handy (and Python) following the steps on the aforelinked GitHub page should have you up and running with Aquarium in minutes. You use the same default port (<code>8050</code>) to access the Splash API and you get a bonus port of <code>8036</code> to watch in your browser (the HAProxy stats page).</p>
<p>This works well when combined with <a rel="nofollow" href="https://github.com/DavisVaughan/furrr" target="_blank"><code>furrr</code><img alt="🔗" src="https://i0.wp.com/s.w.org/images/core/emoji/11/72x72/1f517.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"></a> which is an R package that makes parallel operations very tidy.</p>
<p>One way to use <code>purrr</code>, <code>splashr</code> and Aquarium might look like this:</p>
<pre><code>library(splashr)
library(HARtools)
library(urltools)
library(furrr)
library(tidyverse)

list_of_urls_with_unique_urls &lt;- c(&quot;http://...&quot;, &quot;http://...&quot;, ...)

make_a_splash &lt;- function(org_url) {
  splash(
    host = &quot;ip/name of system you started aquarium on&quot;, 
    user = &quot;your splash api username&quot;, 
    pass = &quot;your splash api password&quot;
  ) %>% 
    splash_response_body(TRUE) %>% # we want to get all the content 
    splash_user_agent(ua_win10_ie11) %>% # splashr has many pre-configured user agents to choose from 
    splash_go(org_url) %>% 
    splash_wait(5) %>% # pick a reasonable timeout; modern web sites with javascript are bloated
    splash_har()
}

safe_splash &lt;- safely(make_a_splash) # splashr/Splash work well but can throw errors. Let's be safe

plan(multiprocess, workers=5) # don't overwhelm the default setup or your internet connection

future_map(sites, ~{
  
  org &lt;- safe_splash(.x) # go get it!
  
  if (is.null(org$result)) {
    sprintf(&quot;Error retrieving %s (%s)&quot;, .x, org$error$message) # this gives us good error messages
  } else {
    
    HARtools::writeHAR( # HAR format saves *everything*. the files are YUGE
      har = org$result, 
      file = file.path(&quot;/place/to/store/stuff&quot;, sprintf(&quot;%s.har&quot;, domain(.x))) # saved with the base domain; you may want to use a UUID via uuid::UUIDgenerate()
    )
    
    sprintf(&quot;Successfully retrieved %s&quot;, .x)
    
  }
  
}) -> results</code></pre>
<p>(Those with a keen eye will grok why <code>splashr</code> supports Splash API basic authentication, now)</p>
<p>The parallel iterator will return a list we can flatten to a character vector (I don’t do that by default since it’s safer to get a list back as it can hold anything and <code>map_chr()</code> likes to check for proper objects) to check for errors with something like:</p>
<pre><code>flatten_chr(results) %>% 
  keep(str_detect, &quot;Error&quot;)
## [1] &quot;Error retrieving www.1.example.com (Service Unavailable (HTTP 503).)&quot;
## [2] &quot;Error retrieving www.100.example.com (Gateway Timeout (HTTP 504).)&quot;
## [3] &quot;Error retrieving www.3000.example.com (Bad Gateway (HTTP 502).)&quot;
## [4] &quot;Error retrieving www.a.example.com (Bad Gateway (HTTP 502).)&quot;
## [5] &quot;Error retrieving www.z.examples.com (Gateway Timeout (HTTP 504).)&quot;</code></pre>
<p>Timeouts would suggest you may need to up the timeout parameter in your Splash call. Service unavailable or bad gateway errors may suggest you need to tweak the Aquarium configuration to add more workers or reduce your <code>plan(…)</code>. It’s not unusual to have to create a scraping process that accounts for errors and retries a certain number of times.</p>
<p>If you were stuck in the <code>splashr</code>/Splash slow-lane before, give this a try to help save you some time and frustration.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://rud.is/b/2018/08/13/in-brief-splashr-update-high-performance-scraping-with-splashr-furrr-teamhg-memexs-aquarium/"> R – rud.is</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/XpTwDcvYO1w" height="1"></a></div>