<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/how-i-became-a-crolute-i-e-an-user-of-the-crul-package/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://maelle.github.io/2017/06/30/crolute/"> Maëlle</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>A few months ago rOpenSci’s Scott Chamberlain asked me for feedback about a new package of his called <a rel="nofollow" href="https://github.com/ropensci/crul" target="_blank"><code>crul</code></a>, an http client like <code>httr</code>, so basically something you use for e.g. writing a package interfacing an API. He told me that a great thing about <a rel="nofollow" href="https://github.com/ropensci/crul" target="_blank"><code>crul</code></a> was that it supports <em>asynchronous requests</em>. I felt utterly uncool because I had no idea what this meant although I had already written quite a few API packages (for instance <a rel="nofollow" href="https://github.com/ropensci/ropenaq" target="_blank"><code>ropenaq</code></a>, <a rel="nofollow" href="https://github.com/ropensci/riem" target="_blank"><code>riem</code></a> and <a rel="nofollow" href="https://github.com/ropensci/opencage" target="_blank"><code>opencage</code></a>).</p>
<p>So I googled the concept, my mind was blown and I decided that I’d trust Scott’s skills (spoiler: you can always do that) and just replace the <code>httr</code><br>
dependency of <code>ropenaq</code> by <a rel="nofollow" href="https://github.com/ropensci/crul" target="_blank"><code>crul</code></a>. Why? First of all note that Crul is a planet in Star Wars whose male inhabitants are called <em>crolutes</em> which sound quite cool (there are female ones as well, called <em>gilliands</em> which doesn’t sound like the package name) and which I now use as a synonym for “user of the <code>crul</code> package”. But I had other reasons to switch… that was the subject of my lightning talk today at the French R conference in Anglet. In this blog post I’ll tell the story again, with a bit more details, in the hope to make you curious about <a rel="nofollow" href="https://github.com/ropensci/crul" target="_blank"><code>crul</code></a>!</p>
<p><img alt="" src="https://i1.wp.com/maelle.github.io/figure/source/2017-06-30-crolute/title.jpg?w=456" data-recalc-dims="1"></p>
<p>Pic by <a rel="nofollow" href="https://twitter.com/thinkR_fr/status/880778815148101632" target="_blank">ThinkR</a>, thanks Colin/Diane/Vincent!</p>
<p><span></span></p>
<h1>Asynchronous requests for dummies like me</h1>
<p>OpenAQ API, which is a fantastic source of open data about air quality (see my blog post about <code>ropenaq</code> <a rel="nofollow" href="https://ropensci.org/blog/blog/2017/02/21/ropenaq" target="_blank">on rOpenSci’s blog</a>), has a pagination system. If you want say all ozone measures for one country, the API will produce several pages of results containing 10,000 measurements each, and for each of these pages you’ll need to make a request.</p>
<p>In theory, you can either do all these requests one after the other (<em>synchronous</em> requests, yes <em>synchronous</em>) or all of them at once or at least in bunches, which corresponds to <em>asynchronous</em> requests. The <a rel="nofollow" href="https://github.com/ropensci/crul" target="_blank"><code>crul</code></a> package supports <em>asynchronous</em> requests while <code>httr</code> doesn’t. As you can imagine, sending several requests at once can increase speed!</p>
<p>Of course, you can’t send too many requests at once to an API, either because it’s not allowed anyway, or at least because it’s not a really nice thing to do: you might put an API in trouble by asking too many things in one go and you probably don’t want to break things. I asked <a rel="nofollow" href="https://twitter.com/joseph_flasher" target="_blank">OpenAQ’s co-founder Joe Flasher</a> whether I could use asynchronous requests, thinking he might say no. That was funny because I actually thought the API was frailer than it is, Joe told me it was fine to do up to 10 requests at once, so I started doing that.</p>
<h1>A big change for <code>ropenaq</code></h1>
<p>So I started removing all calls to <code>httr</code> functions by calls to <code>crul</code>, which wasn’t too hard really because <code>crul</code> has two vignettes, in particular one about asynchronous requests. I like the style of the vignettes, I could recognize the style from <a rel="nofollow" href="https://rforcats.net/" target="_blank">R for cats</a>. Besides, Scott had asked for feedback and I assumed that asking many questions would just help him make the documentation fool-proof. Thanks for your patience Scott!</p>
<p>You can see the <code>crul</code> calls of <code>ropenaq</code> <a rel="nofollow" href="https://github.com/ropensci/ropenaq/blob/master/R/utils.R" target="_blank">here</a>. I have no ambition to write a step by step but basically what happens now in <code>ropenaq</code> is:</p>
<ul>
<li>
<p>if the user provides a page argument, then I do a simple request for that page thanks to an <code>crul::HttpRequest</code> object. In the past as an user of <code>ropenaq</code> for any data wish you had to first had to make a query to get the count of measurements and then write a loop over all pages. Now you don’t, see the following point.</p>
</li>
<li>
<p>if the user doesn’t provide a page, then I retrieve the number of measurements for the query in total, then count the pages and prepare one query by page, which I group in bunches of 10 queries. Each bunch, which is a list, is then used to create an <code>AsyncVaried</code> object via <code>crul::AsyncVaried$new(.list = async)</code>, and then I use methods of that object to retrieve the status and the content.</p>
</li>
</ul>
<p>The only methods I needed to use where <code>new</code> to create objects, <code>request</code> to actually make the requests, <code>status_code</code> to retrieve the request status, <code>get</code> to extract the content of the request output when it had worked and <code>parse</code> to well parse the contents. Their names are quite well chosen. For figuring all of that out I read the vignettes, the documentation of the classes and as mentioned previously asked questions to Scott. In particular at the beginning I wasn’t too sure how to make the 10 requests for different pages at once but as the async vignette of <code>crul</code> says,</p>
<p>“There are two interfaces to asynchronous requests in crul:</p>
<ol>
<li>
<p>Simple async: any number of URLs, all treated with the same curl options, headers, etc., and only one HTTP method type at a time.</p>
</li>
<li>
<p>Varied request async: build any type of request and execute all asynchronously.”</p>
</li>
</ol>
<p>and my case was the second one. I was also a bit puzzled as to how to pass my list of requests to the function creating the <code>AsyncVaried</code> object but it turned out you can just pass a list to the <code>.list</code> argument.</p>
<p>Now the CRAN version of <code>ropenaq</code> is a proud reverse dependency of <code>crul</code> (yes my packages have feelings), so if you do <code>install.packages(&quot;ropenaq&quot;)</code> you can take advantage of my hard work. In my talk I had the results of a small <code>microbenchmark</code> and long story short as you can imagine the total time needed to retrieve data is shorter when you use asynchronous requests.</p>
<h1>Other arguments in favour of <code>crul</code></h1>
<p>So asynchronous requests are a big argument in my opinion to use <code>crul</code> instead of <code>httr</code> as an API package developer but I’ve also listed a few other reasons. Note that I don’t get any money for saying any of this, this is my most honest opinion.</p>
<h2>R6 classes!</h2>
<p>You might have noticed I talked about creating objects and then using methods. <code>crul</code> is based on R6 classes so uses object-oriented programming. You can do things with these objects thanks to the methods. In <code>httr</code>, everything is supported by <em>functions</em>. This is an argument for using <code>crul</code> if:</p>
<ul>
<li>
<p>you like R6 classes or object-oriented programming in general.</p>
</li>
<li>
<p>like me you’ve not used a lot of object-oriented programming but you’re curious to discover them. It was really not scary. Besides I love the fact that once you’ve created an object, in RStudio thanks to auto-completion you can start typing “$”, then press tab and see all the methods of that object.</p>
</li>
</ul>
<p>Furthermore, each time I called an <code>httr</code> function in my package code I wrote <code>httr::</code> before the name of the function to specify the import. With <code>crul</code> you do that only when creating an object, after that when you use the methods of that object you don’t need to declare any more import. So now I can be a bit lazier!</p>
<h2>The promise of mocking made easy quite soon!</h2>
<p>The unit tests of an API package might fail because your code no longer works… or because the API changed its output or is just down. A solution to differentiate these failure reasons is to have tests calling the API, and tests using fake data, the latter being <em>mocked tests</em>. This is discussed <a rel="nofollow" href="https://discuss.ropensci.org/t/best-practices-for-testing-api-packages/460" target="_blank">in this very good thread</a>. Right now <code>crul</code> supports mocking via <a rel="nofollow" href="https://github.com/ropensci/webmockr" target="_blank"><code>webmockr</code></a>, and if I understood correctly will soon support mocking in an easier way via <a rel="nofollow" href="https://github.com/ropensci/vcr" target="_blank"><code>vcr</code></a>. When it does I’ll try to make myself wiser about mocking which I don’t use at all currently.</p>
<h1>Conclusion</h1>
<p>So overall if you’re an API package developper you should consider giving <code>crul</code> a try! I’ll keep using <code>httr</code> in my webscraping and well it is still a dependency of all my other API packages so I like both packages.</p>
<p><img alt="" src="https://i2.wp.com/maelle.github.io/figure/source/2017-06-30-crolute/end.jpg?w=456" data-recalc-dims="1"></p>
<p>Pic by <a rel="nofollow" href="https://twitter.com/alinedeschamps/status/880777453752528896" target="_blank">Aline Deschamps</a>, on that slide I was merely advertising for <a rel="nofollow" href="https://ropensci.org/" target="_blank">rOpenSci</a> and in particular its <a rel="nofollow" href="https://github.com/ropensci/onboarding" target="_blank">onboarding process for packages</a></p>
<p>At the French R conference I was happy to get to present the reasons why I became a crolute. I also enjoyed meeting new people, some of them I knew from Twitter (<a rel="nofollow" href="https://twitter.com/_ColinFay" target="_blank">Colin Fay</a> who actually gave a talk about API packages, <a rel="nofollow" href="https://twitter.com/DianeBELDAME" target="_blank">Diane Beldame</a> who lead a tutorial about the tidyverse, <a rel="nofollow" href="https://twitter.com/VincentGuyader" target="_blank">Vincent Guyader</a> who made the use of Docker to encapsulate a Shiny app look really easy, all of them from <a rel="nofollow" href="https://twitter.com/thinkR_fr" target="_blank">ThinkR</a>, <a rel="nofollow" href="https://twitter.com/joelgombin" target="_blank">Joël Gombin</a> who gave a keynote about data journalism and <a rel="nofollow" href="https://twitter.com/alinedeschamps" target="_blank">Aline Deschamps</a> from <a rel="nofollow" href="https://twitter.com/dacta_fr" target="_blank">Dacta</a>), many others I didn’t know at all, for instance the real guy behind <a rel="nofollow" href="https://twitter.com/R_Graph_Gallery" target="_blank">The R graph gallery</a> who made a cool project using tweets about surfing. I even got to see Nick Tierney again, who gave a talk about his awesome <a rel="nofollow" href="https://github.com/njtierney/narnia" target="_blank"><code>narnia</code> package</a> for handling missing data. The talks were great, some findings were shared via the <a rel="nofollow" href="https://twitter.com/search?q=%23angletr2017&src=tyah" target="_blank">#angletr2017 hashtag on Twitter</a>.</p>
<p>We heard that the next French R conference will be in Rennes which is in Brittany, my beloved home region, so I’m already hoping I can go next year, as an human or a crolute or whatever!</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://maelle.github.io/2017/06/30/crolute/"> Maëlle</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/Axbagj3UuSA" height="1"></a></div>