<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/caching-httr-requests-this-means-warc/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2017/08/22/caching-httr-requests-this-means-warc/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I’ve blathered about my <a rel="nofollow" href="https://rud.is/b/2017/07/28/analyzing-wait-delay-settings-in-common-crawl-robots-txt-data-with-r/" target="_blank"><code>crawl_delay</code> project before</a> and am just waiting for a rainy weekend to be able to crank out a follow-up post on it. Working on that project involved sifting through thousands of <a rel="nofollow" href="https://www.loc.gov/preservation/digital/formats/fdd/fdd000236.shtml" target="_blank">Web Archive (WARC)</a> files. While I have a nascent package on github to work with WARC files it’s a tad fragile and improving it would mean reinventing many wheels (i.e. there are longstanding solid implementations of WARC libraries in many other languages that could be tapped vs writing a C++-backed implementation). </p>
<p>One of those implementations is <a rel="nofollow" href="https://sbforge.org/display/JWAT/JWAT" target="_blank">JWAT</a>, a library written in Java (as many WARC use-cases involve working in what would traditionally be called map-reduce environments). It has a small footprint and is structured well-enough that I decided to take it for a spin as a set of R packages that wrap it with <code>rJava</code>. There are two packages since it follows a recommended CRAN model of having one package for the core Java Archive (JAR) files — since they tend to not change as frequently as the functional R package would and they tend to take up a modest amount of disk space — and another for the actual package that does the work. They are:</p>
<ul>
<li><a rel="nofollow" href="https://github.com/hrbrmstr/jwatjars" target="_blank"><code>jwatjars</code></a></li>
<li><a rel="nofollow" href="https://github.com/hrbrmstr/jwatr" target="_blank"><code>jwatr</code></a></li>
</ul>
<p>I’ll exposit on the full package at some later date, but I wanted to post a snippet showng that you may have a use for WARC files that you hadn’t considered before: pairing WARC files with <code>httr</code> web scraping tasks to maintain a local cache of what you’ve scraped.</p>
<p>Web scraping consumes network &amp; compute resources on the server end that you typically don’t own and — in many cases — do not pay for. While there are scraping tasks that need to access the latest possible data, many times tasks involve scraping data that won’t change.</p>
<p>The same principle works for caching the results of API calls, since you may make those calls and use some data, but then realize you wanted to use more data and make the same API calls again. Caching the raw API results can also help with reproducibility, especially if the site you were using goes offline (like the U.S. Government sites that are being taken down by the anti-science folks in the current administration). </p>
<p>To that end I’ve put together the beginning of some “WARC wrappers” for <code>httr</code> verbs that make it seamless to cache scraping or API results as you gather and process them. Let’s work through an example using the <a rel="nofollow" href="https://data.police.uk/" target="_blank">U.K. open data portal on crime and policing</a> API.</p>
<p>First, we’ll need some helpers:</p>
<pre><code>library(rJava)
library(jwatjars) # devtools::install_github(&quot;hrbrmstr/jwatjars&quot;)
library(jwatr) # devtools::install_github(&quot;hrbrmstr/jwatr&quot;)
library(httr)
library(jsonlite)
library(tidyverse)</code></pre>
<p>Just doing <code>library(jwatr)</code> would have covered much of that but I wanted to show some of the work R does behind the scenes for you.</p>
<p>Now, we’ll grab some neighbourhood and crime info:</p>
<pre><code>wf &lt;- warc_file(&quot;~/Data/wrap-test&quot;)

res &lt;- warc_GET(wf, &quot;https://data.police.uk/api/leicestershire/neighbourhoods&quot;)

str(jsonlite::fromJSON(content(res, as=&quot;text&quot;)), 2)
## 'data.frame':	67 obs. of  2 variables:
##  $ id  : chr  &quot;NC04&quot; &quot;NC66&quot; &quot;NC67&quot; &quot;NC68&quot; ...
##  $ name: chr  &quot;City Centre&quot; &quot;Cultural Quarter&quot; &quot;Riverside&quot; &quot;Clarendon Park&quot; ...

res &lt;- warc_GET(wf, &quot;https://data.police.uk/api/crimes-street/all-crime&quot;,
                query = list(lat=52.629729, lng=-1.131592, date=&quot;2017-01&quot;))

res &lt;- warc_GET(wf, &quot;https://data.police.uk/api/crimes-at-location&quot;,
                query = list(location_id=&quot;884227&quot;, date=&quot;2017-02&quot;))

close_warc_file(wf)</code></pre>
<p>As you can see, the standard <code>httr</code> <code>response</code> object is returned for processing, and the HTTP response itself is being stored away for us as we process it.</p>
<pre><code>file.info(&quot;~/Data/wrap-test.warc.gz&quot;)$size
## [1] 76020</code></pre>
<p>We can use these results later and, pretty easily, since the WARC file will be read in as a tidy R tibble (fancy data frame):</p>
<pre><code>xdf &lt;- read_warc(&quot;~/Data/wrap-test.warc.gz&quot;, include_payload = TRUE)

glimpse(xdf)
## Observations: 3
## Variables: 14
## $ target_uri                 </code></pre>