<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/downloading-files-from-a-webserver-and-failing/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://rmhogervorst.nl/cleancode/blog/2017/12/08/downloading-multiple-files-and-failing.html"> Clean Code</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Recently I wanted to download all the transcripts of a podcast (600+ episodes).<br>
The transcripts are simple txt files so in a way<br>
I am not even ‘web’-scraping but just reading in 600 or so text files which is<br>
not really a big deal. I thought.</p>
<p>This post shows you where I went wrong</p>
<p>Also here is a <a rel="nofollow" href="https://www.flickr.com/photos/32123311@N00/502155430" title="source: flickr, cc-by 2.0 jbloom" target="_blank">picture</a> I found of scraping.</p>
<p><img alt="Scraping a plate " src="https://i0.wp.com/rmhogervorst.nl/cleancode/images/scraping_a_plate.jpg?w=456" data-recalc-dims="1"></p>
<h3>Webscraping general</h3>
<p>For every download you ask the server for a file and it returns the file (this is also how you normally browse the web btw, your browser requests the pages).</p>
<p>In general it is nice if you ask permission (I did, on twitter and the author was really nice! I recommend it!) and don’t<br>
push the website to its limit. The servers where these files are hosted are quite beefy and I will<br>
probably not even make a dent in them, when I’m downloading these files. But still, be gentle.</p>
<p><strong>No really, be a responsible scraper and tell the website owners you are scraping (in person or by identifying in the header) and check if it is allowed</strong></p>
<p>I recently witnessed a demo where someone explained a lot of dirty tricks on how to get over those pesky servers denying them access and generally ignoring good practices and it made me sick…</p>
<p>Here are some general guides:</p>
<ul>
<li><a rel="nofollow" href="https://rud.is/b/2017/07/28/analyzing-wait-delay-settings-in-common-crawl-robots-txt-data-with-r/" target="_blank">Bob Rudis (@hrbrmstr) about scraping and robots.txt</a></li>
<li><a rel="nofollow" href="https://cran.r-project.org/web/packages/httr/vignettes/api-packages.html" target="_blank">excellent vignette of the httr package, read the part about user-agent</a></li>
</ul>
<h3>Downloading non-html files</h3>
<p>There are multiple ways I could do this downloading:<br>
if I had used rvest to scrape a website I would have set a user-agent<br>
header^[a piece of information we snd with every request that describes who we are]<br>
and I would have used incremental backoff: when the server refuses a connection<br>
we would wait and retry again, if it still refuses we would wait twice as long<br>
and retry again etc.</p>
<p>However, since these are txt files I can just use read_lines^[This is the readr variant of readLines from base-R, it is much faster then the original]<br>
to read the txt file of a transcript and apply further work downstream.</p>
<h4>A first, failing approach, tidy but wrong</h4>
<p>This was my first approach:</p>
<ul>
<li>all episodes are numbered and the transcript files are sequental too, so just a paste0 of “https://first-part-of-link” number”.txt” would work.</li>
<li>put all links as row into dataframe</li>
<li>apply some purrr magic by mapping every link to a read_lines function (<em>what? use the read_lines() function on every link</em> ).</li>
</ul>
<div>
<div>
<pre><code><span>latest_episode</span><span> </span><span>&lt;-</span><span> </span><span>636</span><span>
</span><span>system.time</span><span>(</span><span>
</span><span>df_sn</span><span> </span><span>&lt;-</span><span> </span><span>data_frame</span><span>(</span><span>link</span><span> </span><span>=</span><span> </span><span>paste0</span><span>(</span><span>&quot;https:linktowebsite.com/firstpart-&quot;</span><span>,</span><span>
</span><span>formatC</span><span>(</span><span>1</span><span>:</span><span>latest_episode</span><span>,</span><span> </span><span>width</span><span> </span><span>=</span><span> </span><span>3</span><span>,</span><span>flag</span><span> </span><span>=</span><span> </span><span>0</span><span>),</span><span>&quot;.txt&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>transcript</span><span> </span><span>=</span><span> </span><span>map</span><span>(</span><span>link</span><span>,</span><span> </span><span>read_lines2</span><span>))</span><span>
</span><span>)</span><span>
</span></code></pre>
</div>
</div>
<p>This failed.</p>
<p>Some episodes don’t exists or have no transcript (I didn’t know). Sometimes the internet connection didn’t want to work and just threw me out. Sometimes the server stopped my requests.</p>
<p>On every of those occasions the process would stop, give an informative error^[really, it did]. But the R-process would stop and I had no endresult.</p>
<h4>Getting more information to my eyeballs and pausing in between requests</h4>
<p>Also I didn’t know where it failed. So I created a new function that also sometimes waited (to not overwhelm the server)</p>
<div>
<div>
<pre><code><span>## to see where we are this function wraps read_lines and prints the episodenumber</span><span>
</span><span>read_lines2</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>file</span><span>){</span><span>
</span><span>print</span><span>(</span><span>file</span><span>)</span><span>
</span><span>if</span><span>(</span><span>runif</span><span>(</span><span>1</span><span>,</span><span>0</span><span>,</span><span>1</span><span>)</span><span> </span><span>></span><span>0.008</span><span>)</span><span>Sys.sleep</span><span>(</span><span>5</span><span>)</span><span>
</span><span>read_lines</span><span>(</span><span>file</span><span>)</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
</div>
<p>This one also failed, but more informatively, I now knew if it failed on a certain episode.</p>
<p>But ultimately, downloading files from the internet is a somewhat unpredictable process. And it is much easier to just first download all the files and read them in afterwards.</p>
<h4>A two step approach, download first, parse later.</h4>
<p>Also I wanted to let the logs show that I was the one doing the scraping and how to reach me if I was overwhelming the service.</p>
<p>Enter curl.<br>
Curl is a library that helps you download stuff, it is used by the httr package and is a wrapper around the c++ package with the same name, <em>wrapped by Jeroen ‘c-plus-plus’ Ooms</em>.</p>
<p>Since I ran this function a few times I downloaded some of the files, and didn’t really want to download every file again, so I also added a check to see if the file wasn’t already downloaded^[I thought that was really clever, didn’t you?]. And I wanted it to print to the screen, because I like moving text over the screen when I’m debugging.</p>
<div>
<div>
<pre><code><span>download_file</span><span> </span><span>&lt;-</span><span> </span><span>function</span><span>(</span><span>file</span><span>){</span><span>
</span><span>filename</span><span> </span><span>&lt;-</span><span> </span><span>basename</span><span>(</span><span>file</span><span>)</span><span>
</span><span>if</span><span>(</span><span>file.exists</span><span>(</span><span>paste0</span><span>(</span><span>&quot;data/&quot;</span><span>,</span><span>filename</span><span>))){</span><span>
</span><span>print</span><span>(</span><span>paste</span><span>(</span><span>&quot;file exists: &quot;</span><span>,</span><span>filename</span><span>))</span><span>
</span><span>}</span><span>else</span><span>{</span><span>
</span><span>print</span><span>(</span><span>paste0</span><span>(</span><span>&quot;downloading file:&quot;</span><span>,</span><span> </span><span>file</span><span>))</span><span>
</span><span>h</span><span> </span><span>&lt;-</span><span> </span><span>new_handle</span><span>(</span><span>failonerror</span><span> </span><span>=</span><span> </span><span>FALSE</span><span>)</span><span>
</span><span>h</span><span> </span><span>&lt;-</span><span> </span><span>handle_setheaders</span><span>(</span><span>h</span><span>,</span><span> </span><span>&quot;User-Agent&quot;</span><span>=</span><span> </span><span>&quot;scraper by RM Hogervorst, @rmhoge, gh: rmhogervorst&quot;</span><span>)</span><span>
</span><span>curl_download</span><span>(</span><span>url</span><span> </span><span>=</span><span> </span><span>file</span><span>,</span><span>destfile</span><span> </span><span>=</span><span> </span><span>paste0</span><span>(</span><span>&quot;data/&quot;</span><span>,</span><span>filename</span><span>),</span><span>mode</span><span> </span><span>=</span><span> </span><span>&quot;wb&quot;</span><span>,</span><span> </span><span>handle</span><span> </span><span>=</span><span> </span><span>h</span><span>)</span><span>
</span><span>Sys.sleep</span><span>(</span><span>sample</span><span>(</span><span>seq</span><span>(</span><span>0</span><span>,</span><span>2</span><span>,</span><span>0.5</span><span>),</span><span> </span><span>1</span><span>))</span><span> </span><span># copied this from Bob Rudis(@hrbrmstr)</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span></code></pre>
</div>
</div>
<p>I set the header (I think…) and I tell <code>curl</code> not to worry if it fails, <em>we all need reassurance sometimes</em>, but just to continue.</p>
<p>And the downloading begins:</p>
<div>
<div>
<pre><code><span># we choose walk here, because we don't expect output (we do get prints)</span><span>
</span><span># We specificaly do this for the side-effect: downloading to a folder.</span><span>
</span><span>latest_episode</span><span> </span><span>&lt;-</span><span> </span><span>636</span><span>
</span><span>#downloading</span><span>
</span><span>walk</span><span>(</span><span>paste0</span><span>(</span><span>&quot;https://first-part-of-link.com/episodenr-&quot;</span><span>,</span><span>
</span><span>formatC</span><span>(</span><span>1</span><span>:</span><span>latest_episode</span><span>,</span><span> </span><span>width</span><span> </span><span>=</span><span> </span><span>3</span><span>,</span><span>flag</span><span> </span><span>=</span><span> </span><span>0</span><span>),</span><span>&quot;.txt&quot;</span><span>),</span><span> </span><span>download_file</span><span>)</span><span>
</span></code></pre>
</div>
</div>
<h2>Conclusion</h2>
<p>So in general, don’t be a dick, ask permission and take it easy.</p>
<p>The final download approach works great! And it doesn’t matter if you stop it halfway. In the future you can see why I wanted all of these files.</p>
<p>I thought this would be the easy step, would the rest be even harder? Tune in next time!</p>
<h4>Cool things that I could have done:</h4>
<ul>
<li>use purrr::safely ? I think it will continue to work after a fail then?</li>
<li>use a trycatch in the download</li>
<li>first check if the file exists</li>
<li>Do something more with curl, honestly it has many many options that I just didn’t explore.</li>
<li>use some CLI spinners for every download, way cooler</li>
<li>write to a log, and not to the console.</li>
</ul>
<p><a rel="nofollow" href="http://rmhogervorst.nl/cleancode/blog/2017/12/08/downloading-multiple-files-and-failing.html" target="_blank">Downloading files from a webserver, and failing.</a> was originally published by at <a rel="nofollow" href="http://rmhogervorst.nl/cleancode" target="_blank">Clean Code</a> on December 08, 2017.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://rmhogervorst.nl/cleancode/blog/2017/12/08/downloading-multiple-files-and-failing.html"> Clean Code</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/U9dDRyRnJ-g" height="1"></a></div>