<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/pirating-pirate-data-for-pirate-day/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2017/09/pirating-pirate-data-for-pirate-day.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>This past Tuesday was Talk Like A Pirate Date, the unofficial holiday of R (aRRR!) users worldwide. In recognition of the day, Bob Rudis used R to create this <a rel="nofollow" href="https://rud.is/b/2017/09/19/pirating-web-content-responsibly-with-r/amp/" target="_blank">map of worldwide piracy incidents from 2013 to 2017</a>. </p>
<p><a rel="nofollow" href="http://a7.typepad.com/6a0105360ba1c6970c01bb09c5cde7970d-pi" target="_blank"><img border="0" alt="Pirates" title="Pirates" src="http://a7.typepad.com/6a0105360ba1c6970c01bb09c5cde7970d-800wi"></a></p>
<p>The post provides a useful and practical example of extracting data from a website without an API, otherwise known as &quot;scraping&quot; data. In this case the website was the <a rel="nofollow" href="https://www.icc-ccs.org/index.php/piracy-reporting-centre/live-piracy-report/details/169/1345" target="_blank">International Chamber of Commerce</a>, and the provided R code demonstrates several useful R packages for scraping:</p>
<ul>
<li><a rel="nofollow" href="https://mran.microsoft.com/package/purrrlyr/" target="_blank">rvest</a>, for extracting data from a web pages' HTML source</li>
<li><a rel="nofollow" href="https://mran.microsoft.com/package/purrr/" target="_blank">purrr</a>, and specifically the <a rel="nofollow" href="https://www.rdocumentation.org/packages/purrr/versions/0.2.3/topics/safely?tap_a=5644-dce66f&tap_s=10907-287229" target="_blank">safely</a> function, for streamlining the process of iterating over pages that may return errors</li>
<li><a rel="nofollow" href="https://mran.microsoft.com/package/purrrlyr/" target="_blank">purrrly</a>, for iterating over the rows of scraped data</li>
<li><a rel="nofollow" href="https://mran.microsoft.com/package/splashr/" target="_blank">splashr</a>, for automating the process of taking a screenshot of a webpage</li>
<li>and <a rel="nofollow" href="https://mran.microsoft.com/package/robotstxt" target="_blank">robotstxt</a>, for checking whether automated downloads of the website content are allowed </li>
</ul>
<p>That last package is an important one, because while it's almost always technically possible to automate the process of extracting data from a website, it's not always allowed or even legal. Inspecting the robots.txt file is one check you should definitely make, and you should also check the terms of service of the website which may also prohibit the practice. Even then, you should be respectful and take care not to overload the server with frequent and/or repeated calls, as Bob demonstrates by spacing requests by 5 seconds. Finally — and most importantly! — even if scraping isn't expressly forbidden, using scraped data may not be <strong>ethical</strong>, especially when the data is about people who are unable to give their individual consent to your use of the data. This <em>Forbes</em> article <a rel="nofollow" href="https://www.forbes.com/sites/kalevleetaru/2017/09/16/ai-gaydar-and-how-the-future-of-ai-will-be-exempt-from-ethical-review/amp/" target="_blank">about analyzing data scraped from a dating website</a> offers an instructive tale in that regard.</p>
<p>This piracy data example however provides a case study of using websites and the data it provides in the <em>right</em> way. Follow the link below all for the details.</p>
<p>rud.is: <a rel="nofollow" href="https://rud.is/b/2017/09/19/pirating-web-content-responsibly-with-r/amp/" target="_blank">Pirating Web Content Responsibly With R</a></p>
</div>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2017/09/pirating-pirate-data-for-pirate-day.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/KeZAUcYoPhA" height="1"></a></div>