<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/speeding-up-digital-arachnids/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/09/25/speeding-up-digital-arachinds/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<pre>
spiderbar, spiderbar 
Reads robots rules from afar.
Crawls the web, any size; 
Fetches with respect, never lies.
Look Out! 
Here comes the spiderbar.

Is it fast? 
Listen bud, 
It's got C++ under the hood.
Can you scrape, from a site?
Test with can_fetch(), TRUE == alright
Hey, there 
There goes the spiderbar. 
</pre>
<p>(Check the end of the post if you don’t recognize the lyrical riff.)</p>
<h3>Face front, true believer!</h3>
<p>I’ve used and blogged about Peter Meissner’s most excellent <a rel="nofollow" href="https://cran.rstudio.com/web/packages/robotstxt/" target="_blank"><code>robotstxt</code></a> package <a rel="nofollow" href="https://rud.is/b/2017/09/19/pirating-web-content-responsibly-with-r/" target="_blank">before</a>. It’s an essential tool for any ethical web scraper.</p>
<p>But (there’s always a “<em>but</em>“, right?), it was a definite bottleneck for an <a rel="nofollow" href="https://rud.is/b/2017/07/28/analyzing-wait-delay-settings-in-common-crawl-robots-txt-data-with-r/" target="_blank">unintended package use case</a> earlier this year (yes, I still have not rounded out the corners on my “crawl delay” forthcoming post).</p>
<p>I needed something faster for my bulk <code>Crawl-Delay</code> analysis which led me to <a rel="nofollow" href="https://github.com/seomoz/rep-cpp" target="_blank">this small, spiffy C++ library</a> for parsing <code>robots.txt</code> files. After a tiny bit of wrangling, that C++ library has turned into a small R package <a rel="nofollow" href="https://cran.r-project.org/web/packages/spiderbar/" target="_blank"><code>spiderbar</code></a> which is now hitting a CRAN mirror near you, soon. (CRAN — rightly so — did not like the unoriginal name <code>rep</code>).</p>
<h3>How much faster?</h3>
<p>I’m glad you asked!</p>
<p>Let’s take a look at one benchmark: parsing <code>robots.txt</code> and extracting <code>Crawl-delay</code> entries. Just how much faster is <code>spiderbar</code>?</p>
<pre><code>library(spiderbar)
library(robotstxt)
library(microbenchmark)
library(tidyverse)
library(hrbrthemes)

rob &lt;- get_robotstxt(&quot;imdb.com&quot;)

microbenchmark(

  robotstxt = {
    x &lt;- parse_robotstxt(rob)
    x$crawl_delay
  },

  spiderbar = {
    y &lt;- robxp(rob)
    crawl_delays(y)
  }

) -> mb1

update_geom_defaults(&quot;violin&quot;, list(colour = &quot;#4575b4&quot;, fill=&quot;#abd9e9&quot;))

autoplot(mb1) +
  scale_y_comma(name=&quot;nanoseconds&quot;, trans=&quot;log10&quot;) +
  labs(title=&quot;Microbenchmark results for parsing 'robots.txt' and extracting 'Crawl-delay' entries&quot;,
       subtitle=&quot;Compares performance between robotstxt &amp; spiderbar packages. Lower values are better.&quot;) +
  theme_ipsum_rc(grid=&quot;Xx&quot;)</code></pre>
<p><a rel="nofollow" href="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/09/spiderbar_1-1.png?ssl=1" target="_blank"><img alt="" src="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/09/spiderbar_1-1.png?resize=780%2C468&ssl=1" class="wp-image-6475" data-recalc-dims="1"></a></p>
<p>As you can see, it’s just <em>a tad bit faster</em> <img alt="😎" src="https://s.w.org/images/core/emoji/2/72x72/1f60e.png" class="wp-smiley">.</p>
<p>Now, you won’t notice that temporal gain in an interactive context but you absolutely will if you are cranking through a few million of them across a few thousand WARC files from the Common Crawl. </p>
<h3>But, I don’t care about <code>Crawl-Delay</code>!</h3>
<p>OK, fine. Do you care about fetchability? We can speed that up, too!</p>
<pre><code>rob_txt &lt;- parse_robotstxt(rob)
rob_spi &lt;- robxp(rob)

microbenchmark(

  robotstxt = {
    robotstxt:::path_allowed(rob_txt$permissions, &quot;/Vote&quot;)
  },

  spiderbar = {
    can_fetch(rob_spi, &quot;/Vote&quot;)
  }

) -> mb2

autoplot(mb2) +
  scale_y_comma(name=&quot;nanoseconds&quot;, trans=&quot;log10&quot;) +
  labs(title=&quot;Microbenchmark results for testing resource 'fetchability'&quot;,
       subtitle=&quot;Compares performance between robotstxt &amp; spiderbar packages. Lower values are better.&quot;) +
  theme_ipsum_rc(grid=&quot;Xx&quot;)</code></pre>
<p><a rel="nofollow" href="https://i0.wp.com/rud.is/b/wp-content/uploads/2017/09/spiderbar_2-1.png?ssl=1" target="_blank"><img alt="" src="https://i0.wp.com/rud.is/b/wp-content/uploads/2017/09/spiderbar_2-1.png?resize=780%2C468&ssl=1" class="wp-image-6476" data-recalc-dims="1"></a></p>
<h3>Vectorized or it didn’t happen.</h3>
<p>(<em>Gosh, even Spider-Man got more respect!</em>) </p>
<p>OK, this is a tough crowd, but we’ve got vectorization covered as well:</p>
<pre><code>microbenchmark(

  robotstxt = {
    paths_allowed(c(&quot;/ShowAll/this/that&quot;, &quot;/Soundtracks/this/that&quot;, &quot;/Tsearch/this/that&quot;), &quot;imdb.com&quot;)
  },

  spiderbar = {
    can_fetch(rob_spi, c(&quot;/ShowAll/this/that&quot;, &quot;/Soundtracks/this/that&quot;, &quot;/Tsearch/this/that&quot;))
  }

) -> mb3

autoplot(mb3) +
  scale_y_comma(name=&quot;nanoseconds&quot;, trans=&quot;log10&quot;) +
  labs(title=&quot;Microbenchmark results for testing multiple resource 'fetchability'&quot;,
       subtitle=&quot;Compares performance between robotstxt &amp; spiderbar packages. Lower values are better.&quot;) +
  theme_ipsum_rc(grid=&quot;Xx&quot;)</code></pre>
<p><a rel="nofollow" href="https://i0.wp.com/rud.is/b/wp-content/uploads/2017/09/spiderbar_3-1.png?ssl=1" target="_blank"><img alt="" src="https://i0.wp.com/rud.is/b/wp-content/uploads/2017/09/spiderbar_3-1.png?resize=780%2C468&ssl=1" class="wp-image-6477" data-recalc-dims="1"></a></p>
<h3>Excelsior!</h3>
<p>Peter’s package does more than this one since it helps find the <code>robots.txt</code> files and provides helpful data frames for more robots exclusion protocol content. And, <a rel="nofollow" href="https://github.com/hrbrmstr/spiderbar/issues/1#issuecomment-331636193" target="_blank">we’ve got some plans</a> for package interoperability. So, stay tuned, true believer, for more spider-y goodness.</p>
<p>You can check out the code and leave package questions or comments <a rel="nofollow" href="https://github.com/hrbrmstr/spiderbar" target="_blank">on GitHub</a>.</p>
<p><iframe width="427" allowfullscreen src="https://www.youtube.com/embed/SUtziaZlDeE" height="240"></iframe></p>
<p><i>(Hrm…<b>Peter</b> Parker was Spider-Man and <b>Peter</b> Meissner wrote <code>robotstxt</code> which is all about spiders. Coincidence?! I think not!)</i></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://rud.is/b/2017/09/25/speeding-up-digital-arachinds/"> R – rud.is</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/vUed5LY--WI" height="1"></a></div>