<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/pirating-web-content-responsibly-with-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/09/19/pirating-web-content-responsibly-with-r/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>International <del datetime="2017-09-19T11:31:05+00:00">Code</del> Talk Like A Pirate Day almost slipped by without me noticing (September has been a crazy busy month), but it popped up in the calendar notifications today and I was glad that I had prepped the meat of a post a few weeks back.</p>
<p>There will be no ‘rrrrrr’ abuse in this post, I’m afraid, but there will be plenty of R code.</p>
<p>We’re going to combine pirate day with “pirating” data, in the sense that I’m going to show one way on how to use the web scraping powers of R <em>responsibly</em> to collect data on and explore modern-day pirate encounters.</p>
<h3>Scouring The <del datetime="2017-09-19T11:31:05+00:00">Seas</del> Web For Pirate Data</h3>
<p>Interestingly enough, there are many of sources for pirate data. I’ve blogged a few in the past, but I came across a new (to me) one by the International Chamber of Commerce. Their Commercial Crime Services division has something called the <a rel="nofollow" href="https://www.icc-ccs.org/index.php/piracy-reporting-centre/live-piracy-report/details/169/1345" target="_blank">Live Piracy &amp; Armed Robbery Report</a>:</p>
<p><a rel="nofollow" href="https://i0.wp.com/rud.is/b/wp-content/uploads/2017/09/preview.png?ssl=1" target="_blank"><img alt="" src="https://i0.wp.com/rud.is/b/wp-content/uploads/2017/09/preview.png?resize=780%2C730&ssl=1" class="wp-image-6386" data-recalc-dims="1"></a></p>
<p>(site png snapshot taken with <code>splashr</code>)</p>
<p>I fiddled a bit with the URL and — sure enough — if you work a bit you can get data going back to late 2013, all in the same general format, so I jotted down base URLs and start+end record values and filed them away for future use:</p>
<pre><code>library(V8)
library(stringi)
library(httr)
library(rvest)
library(robotstxt)
library(jwatr) # github/hrbrmstr/jwatr
library(hrbrthemes)
library(purrrlyr)
library(rprojroot)
library(tidyverse)

report_urls &lt;- read.csv(stringsAsFactors=FALSE, header=TRUE, text=&quot;url,start,end
https://www.icc-ccs.org/index.php/piracy-reporting-centre/live-piracy-report/details/169/, 1345, 1459
https://www.icc-ccs.org/piracy-reporting-centre/live-piracy-report/details/151/, 1137, 1339
https://www.icc-ccs.org/piracy-reporting-centre/live-piracy-map/details/146/, 885, 1138
https://www.icc-ccs.org/piracy-reporting-centre/live-piracy-report/details/144/, 625, 884
https://www.icc-ccs.org/index.php/piracy-reporting-centre/live-piracy-report/details/133/, 337, 623&quot;)

by_row(report_urls, ~sprintf(.x$url %s+% &quot;%s&quot;, .x$start:.x$end), .to=&quot;url_list&quot;) %>%
  pull(url_list) %>%
  flatten_chr() -> target_urls

head(target_urls)
## [1] &quot;https://www.icc-ccs.org/index.php/piracy-reporting-centre/live-piracy-report/details/169/1345&quot;
## [2] &quot;https://www.icc-ccs.org/index.php/piracy-reporting-centre/live-piracy-report/details/169/1346&quot;
## [3] &quot;https://www.icc-ccs.org/index.php/piracy-reporting-centre/live-piracy-report/details/169/1347&quot;
## [4] &quot;https://www.icc-ccs.org/index.php/piracy-reporting-centre/live-piracy-report/details/169/1348&quot;
## [5] &quot;https://www.icc-ccs.org/index.php/piracy-reporting-centre/live-piracy-report/details/169/1349&quot;
## [6] &quot;https://www.icc-ccs.org/index.php/piracy-reporting-centre/live-piracy-report/details/169/1350&quot;</code></pre>
<p>Time to pillage some details!</p>
<h3>But…Can We Really Do It?</h3>
<p>I poked around the site’s terms of service/terms and conditions and automated retrieval was not discouraged. Yet, those aren’t the only sea mines we have to look out for. Perhaps they use their <code>robots.txt</code> to stop pirates. Let’s take a look:</p>
<pre><code>robotstxt::get_robotstxt(&quot;https://www.icc-ccs.org/&quot;)
## # If the Joomla site is installed within a folder such as at
## # e.g. www.example.com/joomla/ the robots.txt file MUST be
## # moved to the site root at e.g. www.example.com/robots.txt
## # AND the joomla folder name MUST be prefixed to the disallowed
## # path, e.g. the Disallow rule for the /administrator/ folder
## # MUST be changed to read Disallow: /joomla/administrator/
## #
## # For more information about the robots.txt standard, see:
## # http://www.robotstxt.org/orig.html
## #
## # For syntax checking, see:
## # http://www.sxw.org.uk/computing/robots/check.html
##
## User-agent: *
## Disallow: /administrator/
## Disallow: /cache/
## Disallow: /cli/
## Disallow: /components/
## Disallow: /images/
## Disallow: /includes/
## Disallow: /installation/
## Disallow: /language/
## Disallow: /libraries/
## Disallow: /logs/
## Disallow: /media/
## Disallow: /modules/
## Disallow: /plugins/
## Disallow: /templates/
## Disallow: /tmp/</code></pre>
<p>Ahoy! We’ve got a license to pillage!</p>
<p>But, we don’t have a license to abuse their site. </p>
<p>While I still haven’t had time to follow up on an <a rel="nofollow" href="https://rud.is/b/2017/07/28/analyzing-wait-delay-settings-in-common-crawl-robots-txt-data-with-r/" target="_blank">earlier post about ‘crawl-delay’</a> settings across the internet I <em>have</em> done enough work on it to know that a 5 or 10 second delay is the most common setting (when sites bother to have this directive in their <code>robots.txt</code> file). ICC’s site does not have this setting defined, but we’ll still <del datetime="2017-09-19T11:31:05+00:00">pirate</del> crawl responsibly and use a 5 second delay between requests:</p>
<pre><code>s_GET &lt;- safely(GET)

pb &lt;- progress_estimated(length(target_urls))
map(target_urls, ~{
  pb$tick()$print()
  Sys.sleep(5)
  s_GET(.x)
}) -> httr_raw_responses

write_rds(httr_raw_responses, &quot;data/2017-icc-ccs-raw-httr-responses.rds&quot;)

good_responses &lt;- keep(httr_raw_responses, ~!is.null(.x$result))

jwatr::response_list_to_warc_file(good_responses, &quot;data/icc-good&quot;)</code></pre>
<p>There are more “safety” measures you can use with <code>httr::GET()</code> but this one is usually sufficient. It just prevents the iteration from dying when there are hard retrieval errors.</p>
<p>I also like to save off the crawl results so I can go back to the raw file (if needed) vs re-scrape the site (this crawl takes a while). I do it two ways here, first using raw <code>httr</code> <code>response</code> objects (including any “broken” ones) and then filtering out the “complete” responses and saving them in WARC format so it’s in a more common format for sharing with others who may not use R.</p>
<h3>Digging For Treasure</h3>
<p>Did I mention that while the site looks like it’s easy to scrape it’s really not easy to scrape? That nice looking table is a sea mirage ready to trap unwary <del datetime="2017-09-19T11:31:05+00:00">sailors</del> crawlers in a pit of despair. The UX is built dynamically from on-page javascript content, a portion of which is below:</p>
<p><a rel="nofollow" href="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/09/view-source_https___www_icc-ccs_org_index_php_piracy-reporting-centre_live-piracy-report_details_169_1345.png?ssl=1" target="_blank"><img alt="" src="https://i2.wp.com/rud.is/b/wp-content/uploads/2017/09/view-source_https___www_icc-ccs_org_index_php_piracy-reporting-centre_live-piracy-report_details_169_1345.png?resize=780%2C431&ssl=1" class="wp-image-6392" data-recalc-dims="1"></a></p>
<p>Now, you’re likely thinking: <em>“Don’t we need to re-scrape the site with <code>seleniumPipes</code> or <code>splashr</code>?”</em></p>
<p>Fear not, stout yeoman! We can do this with the content we have if we don’t mind swabbing the decks first. Let’s put the <del datetime="2017-09-19T11:31:05+00:00">map</del> code up first and then dig into the details:</p>
<pre><code># make field names great again
mfga &lt;- function(x) {
  x &lt;- tolower(x)
  x &lt;- gsub(&quot;[[:punct:][:space:]]+&quot;, &quot;_&quot;, x)
  x &lt;- gsub(&quot;_+&quot;, &quot;_&quot;, x)
  x &lt;- gsub(&quot;(^_|_$)&quot;, &quot;&quot;, x)
  x &lt;- make.unique(x, sep = &quot;_&quot;)
  x
}

# I know the columns I want and this makes getting them into the types I want easier
cols(
  attack_number = col_character(),
  attack_posn_map = col_character(),
  date = col_datetime(format = &quot;&quot;),
  date_time = col_datetime(format = &quot;&quot;),
  id = col_integer(),
  location_detail = col_character(),
  narrations = col_character(),
  type_of_attack = col_character(),
  type_of_vessel = col_character()
) -> pirate_cols

# iterate over the good responses with a progress bar
pb &lt;- progress_estimated(length(good_responses))
map_df(good_responses, ~{

  pb$tick()$print()

  # `safely` hides the data under `result` so expose it
  doc &lt;- content(.x$result)

  # target the `		
		
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://rud.is/b/2017/09/19/pirating-web-content-responsibly-with-r/"> R – rud.is</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/gQxdLGU0eSw" height="1"></a></div></code></pre>