<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/packaging-the-theyworkforyou-api/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://conjugateprior.org/2017/06/packaging-the-twfy-api/"> R – CONJUGATEPRIOR</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a rel="nofollow" href="http://www.theyworkforyou.com/" target="_blank">TheyWorkForYou</a> is a great website for keeping up with British politics and one of the many fine things <a rel="nofollow" href="https://www.mysociety.org/democracy/" target="_blank">mySociety</a> does to make democracy in the UK more transparent.</p>
<p>There’s also an API, accessible via <code>http</code> and wrapped up for a few languages. However, R is not amongst them, so I wrote <strong><a rel="nofollow" href="https://www.github.com/conjugateprior/twfy" target="_blank">twfy</a></strong>.</p>
<p>If you’re interested in using it (and you’ve got <code>devtools</code> installed) you can install it with</p>
<pre>
devtools::install_github(&quot;conjugateprior/twfy&quot;)
</pre>
<p>It was my first proper API package and a bit of a learning experience. If you want to hear more about that, read on.<br>
<span></span></p>
<h3>APIs</h3>
<p>First some recap, for those just joining us.</p>
<p>The TheyWorkForYou API works with parameterized GETs to URLs with a common base:</p>
<pre>
http://theyworkforyou.com/api/
</pre>
<p>and different endpoints, depending on what you want. First you <a rel="nofollow" href="http://theyworkforyou.com/api/" target="_blank">sign up for an API key</a> and then you make the calls.</p>
<p>For example, if you want a list of UK parliamentary constituencies then your endpoint is <code>getConstituency</code>, which takes either a <code>name</code> or a <code>postcode</code>, plus your API key <code>key</code> and an <code>output</code> specification, and returns a structured constituency object.</p>
<p>In a browser, the complete call looks like</p>
<pre>
https://www.theyworkforyou.com/api/getConstituency?name=Keighley&amp;output=js&amp;key=adsfiddscsdlsdlk
</pre>
<p>where of course <code>adsfiddscsdlsdlk</code> isn’t really an API key. It just plays one the web.</p>
<p>The server returns a JSON object:</p>
<pre>
{
  &quot;bbc_constituency_id&quot; : &quot;344&quot;,
  &quot;guardian_election_results&quot; : &quot;http://www.guardian.co.uk/politics/constituency/1050/keighley&quot;,
  &quot;guardian_id&quot; : &quot;1050&quot;,
  &quot;guardian_name&quot; : &quot;Keighley&quot;,
  &quot;pa_id&quot; : &quot;338&quot;,
  &quot;name&quot; : &quot;Keighley&quot;
}
</pre>
<p>Except that’s only sort of true. The server claims to return a Javascript object, as we can tell from its MIME type. <code>&quot;text/javascript; charset=iso-8859-1&quot;</code>. We’ll just treat it like JSON though.</p>
<p>Making this call and decoding the result programmatically is straightforward with the right packages</p>
<pre>
library(httr)
library(jsonlite)

q &lt;- list(output=&quot;js&quot;, key=&quot;adsfiddscsdlsdlk&quot;, name=&quot;Keighley&quot;)
url &lt;- &quot;https://www.theyworkforyou.com/api/getConstituency&quot;

resp &lt;- GET(url, query=q) # make the call
json &lt;- fromJSON(content(resp)) # parse the response
</pre>
<p>where we’ve let the <code>GET</code> function deal with all the URL escaping, and just asked <code>fromJSON</code> to do the right thing. By default, <code>jsonlite</code> generally believes the right thing looks like a data.frame and most of the time that’s fine.</p>
<p>So far so straightforward. </p>
<h3>Creating a general purpose function to call the API</h3>
<p>It’s pretty easy to generalize this code to create a generic API calling function</p>
<pre>
call_api(endpoint, ...)
</pre>
<p>Inside that function we’ll just grab the arguments as <code>list(...)</code>, add our preferred <code>output</code> format and the API <code>key</code>, and hand the whole thing to <code>GET</code>.</p>
<p>With <code>call_api</code> in hand it’s possible to make all the API functions look pretty much the same, e.g. </p>
<pre>
getConstituencies &lt;- function(date=NULL, search=NULL){
  params &lt;- list(data=date, search=search)
  call_api(&quot;getConstituencies&quot;, params)
}
</pre>
<p>but you know, why write ‘getConstituencies’ twice? </p>
<p>Let’s be a bit more general, and use the fact that R functions know what their names are, and that all function parameters have a name, to make a completely general function body.</p>
<p>The actual function in <strong><a rel="nofollow" href="https://www.github.com/conjugateprior/twfy" target="_blank">twfy</a></strong> is</p>
<pre>
getConstituencies &lt;- function(date=NULL, search=NULL){
  params &lt;- params_from_call(match.call())
  do.call(&quot;call_api&quot;, params)
}
</pre>
<p>which has exactly the same body as</p>
<pre>
getMPInfo &lt;- function(id, fields=NULL){
  params &lt;- params_from_call(match.call())
  do.call(&quot;call_api&quot;, params)
}
</pre>
<p>Cute.</p>
<p>If the API adds another endpoint, I’ll create a new function with this body, give it the name of the endpoint, and write the help in roxygen above it.</p>
<p>So how does this work? </p>
<p>Well, inside (any) function <code>as.list(match.call())</code> is a list with an unlabeled first element that is the name of the function, and subsequent labeled components that are its arguments. If we call <code>getConstituencies</code> function above with <code>search=&quot;Keigh&quot;</code> that means<br>
<code><br>
[[1]]<br>
getConstituencies
<p>$search<br>
[1] &quot;Keigh&quot;<br>
<br>
All the package’s <code>params_from_call</code> does is remove the first argument from the list and re-add it (<code>as.character</code>, because it’s actually an R symbol) under the new label <code>endpoint</code>, so that <code>params</code> is<br>
<code><br>
$search<br>
[1] &quot;Keigh&quot;
<p>$endpoint<br>
[1] &quot;getConstituencies&quot;<br>
<br>
I then use <code>do.call</code> to call <code>call_api</code> with these arguments. This works because <code>call_api</code> is looking for an argument called endpoint and zero or more named arguments, and params gives it one of each.</p>
<p>This leads to the question: why even have separate function for each endpoint offered by the API? There are two answers:</p>
<p>First, an important effect of wrapping an API is to have the documentation near to hand. This requires separate R functions to write the roxygen above. </p>
<p>Speaking of documentation, TheyWorkForYou is a little bit vague about what each of its endpoints returns, so if you’re listening, a pointer to some more documentation would be great.</p>
<p>Second, it is sometimes useful to pre- or post-process the arguments to <code>do.call</code>. Here’s an example of how documentation and pre-processing interact:</p>
<pre>
getDebates &lt;- function(type=c(&quot;commons&quot;, &quot;westminsterhall&quot;, &quot;lords&quot;,
                 &quot;scotland&quot;, &quot;northernireland&quot;),
                 date=NULL, search=NULL, person=NULL, gid=NULL, 
                 order=c(&quot;d&quot;, &quot;r&quot;), page=NULL, num=NULL){
  params &lt;- params_from_call(match.call())
  params$type &lt;- match.arg(type)
  params$order &lt;- match.arg(order)
  do.call(&quot;call_api&quot;, params)
}
</pre>
<p>The user must specify a legislative body to search with the <code>type</code> argument, and can specify a results ordering with the <code>order</code> argument. The function definition is a good place to put the small number of argument possibilities, not least because they will get picked up by command completion. </p>
<p>In the code above I process the function’s arguments as usual, but then step in and fix the values of <code>type</code> and <code>order</code> using <code>match.arg</code> in the normal way, before making the call.</p>
<h3>Where did I leave my keys?</h3>
<p>Like most APIs TheyWorkForYou requires a key to use. Here I follow Hadley Wickham’s very useful guidelines (see the links at the end) and store it as an environment variable. </p>
<p>In <strong><a rel="nofollow" href="https://www.github.com/conjugateprior/twfy" target="_blank">twfy</a></strong> there’s an internal function that prompts for a key as necessary</p>
<pre>
get_api_key &lt;- function(){
  key &lt;- Sys.getenv(&quot;TWFY_API_KEY&quot;)
  if (key == &quot;&quot;){
    key &lt;- ask_for_key()
    if (key != &quot;&quot;){
      Sys.setenv(TWFY_API_KEY=key)
      add_key_to_renviron(key)  # and set up for next time
    } else
      stop(&quot;Hint: you can request an API key from http://theyworkforyou.com/api/key&quot;)
  }
  key
}
</pre>
<p>The first time it’s needed, this prompts the user for the key, sets its value in the local environment, and writes a line into the user’s <code>.Renviron</code> file so it’s available in later sessions.</p>
<p>There is a <code>set_api_key</code>, but this is only really needed to reset an existing key.</p>
<h3>Testing with keys</h3>
<p>If you’re a fan of continuous integration, then the next challenge is to set things up in such a way as not to expose the API key in the server logs or hardcode it into the R source. <strong><a rel="nofollow" href="https://www.github.com/conjugateprior/twfy" target="_blank">twfy</a></strong> uses Travis, and for Travis the solution is to set the api key as an environment variable in the repository settings. </p>
<p>By default these variables do not appear in the build logs, and that’s the way we like it.</p>
<p>The current <code>.travis.yml</code> for <strong><a rel="nofollow" href="https://www.github.com/conjugateprior/twfy" target="_blank">twfy</a></strong> looks like</p>
<pre>
language: R
sudo: false
cache: packages

before_install:
 - echo &quot;TWFY_API_KEY=${TWFY_API_KEY}&quot; > ~/.Renviron
</pre>
<p>Actually I’m not sure whether it’s even necessary to drop Travis’s copy of the API key into the <code>.Renviron</code> to get picked up by the package functions. It’s possible that R picks up local environment variables more reliably on Ubuntu than on my OS X box. </p>
<p>Still, this works. The package builds and no third parties (or forks) see the API key.</p>
<h3>Further reading</h3>
<p>If you find yourself wrapping an API I’d thoroughly recommend reading</p>
<ul>
<li>The <a rel="nofollow" href="https://cran.r-project.org/web/packages/httr/vignettes/api-packages.html" target="_blank">httr vignette</a> on API packages.</li>
<li>the <a rel="nofollow" href="https://cran.r-project.org/web/packages/jsonlite/vignettes/json-apis.html" target="_blank">jsonlite vignette</a> on REST APIs.</li>
</ul>
<h3>Somebody always got there first</h3>
<p>It turns out that <a rel="nofollow" href="http://www.jackblumenau.com/" target="_blank">Jack Blumenau</a> saw TheyWorkForYou’s API and thought the same thing. Great minds, and all that. You can see his take on the problem at <a rel="nofollow" href="https://github.com/jblumenau/twfyR" target="_blank">here</a>. He likes XML quite a bit more than me, apparently. </p>
<p>In any case, as Chairman Mao once said: “Let a hundred flowers blossom, let a hundred schools of thought contend. And let the winner drive the losers onto a small volcanic island shaped like a sweet potato”. Or something like that.</p>

				
		
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://conjugateprior.org/2017/06/packaging-the-twfy-api/"> R – CONJUGATEPRIOR</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...

	<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/_6_i0Tlgcsw" height="1"></a></div></code></code>