<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/vcr-record-and-replay-http-requests/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://ropensci.org/technotes/2018/05/25/vcr-http-request-cacahing/"> rOpenSci - open tools for open science</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>R package test suites that include HTTP requests are dependent on an internet connection being up, the internet connection speed, changing behavior of the remote server, as well as changing response formats/data from a remote server. We ideally want to test functionality of our package relative to some known data that isn’t intermittently unavailable or changing. Caveat is that we do want to make sure the package fails well, including fails well in response to server failures, but these responses can be cached.</p>
<p>One way to deal with the problems above is to store responses from HTTP requests and use them in subsequent requests so that test suites are using a predictable set of data on each run of the test suite. We don’t have to reinvent the wheel to look for a solution, and the solution has been around for a while. The programming language <a rel="nofollow" href="https://www.ruby-lang.org/en/" target="_blank">Ruby</a> has had <a rel="nofollow" href="https://github.com/vcr/vcr" target="_blank">vcr</a> since <a rel="nofollow" href="https://rubygems.org/gems/vcr/versions" target="_blank">2010</a> (and ported to 15 other languages).</p>
<p>Ruby’s <a rel="nofollow" href="https://github.com/vcr/vcr" target="_blank">vcr</a> plugs in to many other HTTP gems via <a rel="nofollow" href="https://github.com/bblimke/webmock" target="_blank">webmock</a>, another Ruby gem. <code>webmock</code> does the actual matching of HTTP requests (but does not store any response), while <code>vcr</code> leverages <code>webmock</code> but then stores the HTTP response.</p>
<p>About 3.5 years ago I made my first commits on ports of Ruby’s vcr to R called <a rel="nofollow" href="https://github.com/ropensci/vcr/commit/6690a23" target="_blank">vcr</a>, and one for Ruby’s webmock, called <a rel="nofollow" href="https://github.com/ropensci/webmockr/commit/7fe4225" target="_blank">webmockr</a>. It took me a long time to get these out on CRAN, which I’ll talk about below.</p>
<p></p>
<h1>vcr</h1>
<p>Let’s get right into it. How do we use <code>vcr</code>?</p>
<p></p>
<h2>Setup</h2>
<p>The setup within your package goes like so. In the <code>tests/</code> folder, you’ll end up with a new folder called <code>fixtures</code>. You can control what this is called by configuring vcr via <code>vcr::vcr_configuration()</code>. Let’s assume we’re working with a package called <code>foo</code>.</p>
<pre><code>foo/tests
├── fixtures/vcr_cassettes
├── test-all.R
└── testthat
</code></pre>
<p>Inside of <code>testthat/</code> we need to insert some configuration for <code>vcr</code>, which can go in a helper file like so:</p>
<pre><code>foo/tests/testthat
├── helper-foo.R
</code></pre>
<p>Within the <code>helper-foo.R</code> file we need something like so to tell <code>vcr</code> where to store cached HTTP responses:</p>
<pre><code>library(&quot;vcr&quot;)
vcr_configure(dir = &quot;../fixtures/vcr_cassettes&quot;)
</code></pre>
<p><code>vcr</code> will set up the new folders given in the <code>dir</code> parameter of <code>vcr_configure</code> if they don’t already exist.</p>
<p>You may have figured this out already, but to be sure: “cassette” means essentially a thing to which you are recording HTTP interactions. There’s other terminology in <code>vcr</code> like “record” and “eject”. See the <a rel="nofollow" href="https://ropensci.github.io/http-testing-book/intro.html#terminology" target="_blank">terminology book section</a> for more.</p>
<p></p>
<h2>Basic usage</h2>
<p>Load <a rel="nofollow" href="https://github.com/ropensci/vcr" target="_blank">vcr</a>, <a rel="nofollow" href="https://github.com/ropensci/crul" target="_blank">crul</a>, and <code>testthat</code></p>
<pre><code>library(vcr)
library(crul)
library(testthat)
</code></pre>
<p>Make a <code>crul</code> connection client.</p>
<pre><code>cli &lt;- crul::HttpClient$new(url = &quot;https://api.crossref.org&quot;)
</code></pre>
<p>Then wrap the actual HTTP request call in a call to <code>vcr::use_cassette()</code>.</p>
<pre><code>use_cassette(name = &quot;helloworld&quot;, {
  res &lt;- cli$get(&quot;works&quot;, query = list(rows = 3))
  expect_is(res, &quot;HttpResponse&quot;)
})
#> CrulAdapter enabled!
#> net connect allowed
#> net connect disabled
#> ejecting cassette: helloworld
#> CrulAdapter disabled!
</code></pre>
<p>The first time the above code block is run real HTTP connections are allowed because it doesn’t match any previous requests, and the response is cached. The second time the reqeust is made, the cached response is used.</p>
<p>The cached response is a yaml file – an example is show below. The advante of the format is that it’s machine readable but humans can also read them so you can inspect your cached HTTP responses to better be able to construct/debug your test suite.</p>
<pre><code>http_interactions:
- request:
    method: get
    uri: https://api.crossref.org/works?rows=3
    body:
      encoding: ''
      string: ''
    headers:
      User-Agent: libcurl/7.54.0 r-curl/3.2 crul/0.5.2
      Accept-Encoding: gzip, deflate
      Accept: application/json, text/xml, application/xml, */*
  response:
    status:
      status_code: '200'
      message: OK
      explanation: Request fulfilled, document follows
    headers:
      status: HTTP/1.1 200 OK
      content-type: application/json;charset=UTF-8
      vary: Accept
      access-control-allow-origin: '*'
      access-control-allow-headers: X-Requested-With
      content-length: '5360'
      server: http-kit
      date: Sat, 28 Apr 2018 15:12:29 GMT
      x-rate-limit-limit: '50'
      x-rate-limit-interval: 1s
      connection: close
    body:
      encoding: UTF-8
      string: '{&quot;status&quot;:&quot;ok&quot;,&quot;message-type&quot;:&quot;work-list&quot;,&quot;message-version&quot;:&quot;1.0.0&quot;,&quot;message&quot;:{&quot;facets&quot;:{}},&quot;items-per-page&quot;:3,&quot;query&quot;:{&quot;start-index&quot;:0,&quot;search-terms&quot;:null}}'
  recorded_at: 2018-04-28 15:12:29 GMT
  recorded_with: vcr/0.1.0, webmockr/0.2.6, crul/0.5.2
</code></pre>
<p></p>
<h2>Why vcr and webmockr?</h2>
<p>The power of both vcr and webockr is they abstract away the gory details, and present a single user interface to (eventually) many different HTTP R libraries (only <code>crul</code> right now, but <code>curl</code> and <code>httr</code> are in development). In addition, though not available yet, vcr will have many different serialization formats; it has only YAML now, but could include JSON and other formats. Both packages will handle a relatively complex set of use cases that come from many years of input from lots of users.</p>
<p></p>
<h2>http testing book</h2>
<p>To integrate documentation for <code>vcr</code> and <code>webmockr</code> and related tools we’ve started a new <a rel="nofollow" href="https://bookdown.org/home/" target="_blank">bookdown</a> called the <a rel="nofollow" href="https://ropensci.github.io/http-testing-book/" target="_blank">HTTP Testing Book</a>.</p>
<p>The book will cover high level concepts of testing R packges that make HTTP requests, how to approach the general problem, how to use webmockr by itself, and how to use vcr. As we add more integrations with http R packages we’ll add more content/chapters to document the new stuff.</p>
<p>Do let us know if you have any feedback on the book! You can send a PR if you want to suggest a change by clicking on the edit button.</p>
<p></p>
<h2>vcr in the wild</h2>
<p>R’s <code>webmockr</code> and <code>vcr</code> are on CRAN. In addition, there are two packages on CRAN using <code>vcr</code> for test caching. They are:</p>
<ul>
<li><a rel="nofollow" href="https://cran.rstudio.com/web/packages/ritis/" target="_blank">ritis</a></li>
<li><a rel="nofollow" href="https://cran.rstudio.com/web/packages/worrms/" target="_blank">worrms</a></li>
</ul>
<p>There are a number more on GitHub, soon to head to CRAN:</p>
<ul>
<li><a rel="nofollow" href="https://github.com/ropensci/rredlist/tree/vcr-integration" target="_blank">rredlist</a></li>
<li><a rel="nofollow" href="https://github.com/ropensci/bold/tree/with-vcr" target="_blank">bold</a></li>
<li><a rel="nofollow" href="https://github.com/ropensci/wikitaxa/tree/with-vcr" target="_blank">wikitaxa</a></li>
<li><a rel="nofollow" href="https://github.com/ropensci/microdemic/tree/with-vcr" target="_blank">microdemic</a></li>
<li><a rel="nofollow" href="https://github.com/ropenscilabs/zbank" target="_blank">zbank</a></li>
<li><a rel="nofollow" href="https://github.com/ropensci/rplos" target="_blank">rplos</a></li>
</ul>
<p>You can get a sense for how <code>vcr</code> is used in real packages by looking at the <code>tests</code> directory in any of the packages above.</p>
<p></p>
<h2>Security</h2>
<p>Testing HTTP packages often involves using authentication to the web service. In addition, responses may contain sensitive data pertaining to users or other such things. If you push your stored HTTP responses onto the public web you most definitely DO NOT want your secure tokens/etc. to be included.</p>
<p>We have a way to deal with this already, where you can replace strings matching a pattern with a placeholder. The HTTP responses stored on disk will have the placeholder only. For example:</p>
<pre><code>vcr_configure(
  filter_sensitive_data = list(&quot;</code></pre>