{"id":3919,"date":"2016-02-10T12:11:02","date_gmt":"2016-02-10T17:11:02","guid":{"rendered":"http:\/\/rud.is\/b\/?p=3919"},"modified":"2018-10-13T15:02:30","modified_gmt":"2018-10-13T20:02:30","slug":"craft-httr-calls-cleverly-with-curlconverter","status":"publish","type":"post","link":"https:\/\/rud.is\/b\/2016\/02\/10\/craft-httr-calls-cleverly-with-curlconverter\/","title":{"rendered":"Craft httr calls cleverly with curlconverter"},"content":{"rendered":"

UPDATE<\/strong> curlconverter<\/code> will now return (as the function return value) a working R function<\/em>. See the README<\/a> for examples<\/p>\n\n

When you visit a site like the LA Times’ NH Primary Live Results site<\/a> and wish you had the data that they used to make the tables & visualizations on the site:<\/p>\n

\"primary\"<\/p>\n

Sometimes it’s as simple as opening up your browsers “Developer Tools” console and looking for XHR (XML HTTP Requests) calls:<\/p>\n

\"XHR\"<\/p>\n

You can actually see a preview of those requests (usually JSON):<\/p>\n

\"Developer_Tools_-_http___graphics_latimes_com_election-2016-new-hampshire-results_\"<\/p>\n

While you could<\/em> go through all the headers and cookies and transcribe them into httr::GET<\/code> or httr::POST<\/code> requests, that’s tedious, especially when most browsers present an option to “Copy as cURL”<\/em>. cURL is a command-line tool (with a corresponding systems programming library) that you can use to grab data from URIs. The RCurl<\/code> and curl<\/code> packages in R are built with the underlying library. The cURL command line captures all of the information necessary to replicate the request the browser made for a resource. The cURL command line for the URL that gets the Republican data is:<\/p>\n

curl 'http:\/\/graphics.latimes.com\/election-2016-31146-feed.json' \n  -H 'Pragma: no-cache' \n  -H 'DNT: 1' \n  -H 'Accept-Encoding: gzip, deflate, sdch'\n  -H 'X-Requested-With: XMLHttpRequest' \n  -H 'Accept-Language: en-US,en;q=0.8' \n  -H 'User-Agent: Mozilla\/5.0 (Macintosh; Intel Mac OS X 10_11_3) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/49.0.2623.39 Safari\/537.36' \n  -H 'Accept: *\/*' \n  -H 'Cache-Control: no-cache' \n  -H 'If-None-Match: \"7b341d7181cbb9b72f483ae28e464dd7\"' \n  -H 'Cookie: s_fid=79D97B8B22CA721F-2DD12ACE392FF3B2; s_cc=true' \n  -H 'Connection: keep-alive' \n  -H 'If-Modified-Since: Wed, 10 Feb 2016 16:40:15 GMT'\n  -H 'Referer: http:\/\/graphics.latimes.com\/election-2016-new-hampshire-results\/' \n  --compressed<\/pre>\n
While that’s easier than manual copy\/paste transcription, these requests are uniform enough that there Has To Be A Better Way. And, now there is, with curlconverter<\/code><\/a>.<\/p>\n
The curlconverter<\/code> package has (for the moment) two main functions:<\/p>\n