<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/analyzing-wait-delay-settings-in-common-crawl-robots-txt-data-with-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/07/28/analyzing-wait-delay-settings-in-common-crawl-robots-txt-data-with-r/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>One of my tweets that referenced an excellent post about the ethics of web scraping garnered some interest:</p>
<p>
<blockquote class="twitter-tweet" data-lang="en">
<p lang="en" dir="ltr">Apologies for a Medium link but if you do ANY web scraping, you need to read this <a rel="nofollow" href="https://twitter.com/hashtag/rstats?src=hash" target="_blank">#rstats</a> // Ethics in Web Scraping <a rel="nofollow" href="https://t.co/y5YxvzB8Fd" target="_blank">https://t.co/y5YxvzB8Fd</a></p>
<p>— boB Rudis (<a rel="nofollow" href="http://twitter.com/hrbrmstr" target="_blank">@hrbrmstr</a>) <a rel="nofollow" href="https://twitter.com/hrbrmstr/status/890226176383475712" target="_blank">July 26, 2017</a></p></blockquote>
<p> </p>
<p>If you load that up that tweet and follow the thread, you’ll see a <em>really good</em> question by <a rel="nofollow" href="http://twitter.com/kennethrose82" target="_blank">@kennethrose82</a> regarding what an appropriate setting should be for a delay between crawls.</p>
<p>The answer is a bit nuanced as there are some written and unwritten “rules” for those who would seek to scrape web site content. For the sake of brevity in this post, we’ll only focus on “best practices” (ugh) for being kind to web site resources when it comes to timing requests, after a quick mention that “Step 0” <em>must</em> be to validate that the site’s terms &amp; conditions or terms of service allow you to scrape &amp; use data from said site. </p>
<h3>Robot Roll Call</h3>
<p>The absolute first thing you should do before scraping a site should be to check out their <code>robots.txt</code> file. What’s that? Well, I’ll let you read about it first from the <a rel="nofollow" href="https://cran.rstudio.com/web/packages/robotstxt/vignettes/using_robotstxt.html" target="_blank">vignette of the package</a> we’re going to use to work with it.</p>
<p>Now that you know what such a file is, you also likely know how to peruse it since the vignette has excellent examples. But, we’ll toss one up here for good measure, focusing on one field that we’re going to talk about next:</p>
<pre><code>library(tidyverse)
library(rvest)

robotstxt::robotstxt(&quot;seobook.com&quot;)$crawl_delay %>% 
  tbl_df()
## # A tibble: 114 x 3
##          field       useragent value
##          </code></pre>