<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/web-data-acquisition-understanding-rcurl-from-the-command-line-part-1/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://r-posts.com/web-data-acquisition-understanding-rcurl-from-the-command-line-part-1/"> R posts</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>After the short presentation <a target="_blank" rel="nofollow" href="http://www.linkedin.com/pulse/r-esolutions-2017-marche-roberto-palloni">here</a>, let’s start using R seriously, i.e. with every day data.</p>
<p>Being a frequent flyer, I often search the web to book flights and organise my trip. Being a data analyst, it’s natural to look at price data with interest, especially the most convenient, most expensive and “average” flights. Being a PhD student in software engineering the (study) programme includes some on the structure of applications, client-server data flows and the web languages running the processes.</p>
<p>Nothing like Google flight API <a target="_blank" rel="nofollow" href="http://developers.google.com/qpx-express/">QPX express</a> would have been better to combine all these three. I discovered it by chance delving into the first results of my flight searches on Google and it is perfect for some web scraping and analysis. As described in the manual, QPX Express is part of the Google suite of publicly-available <a target="_blank" rel="nofollow" href="http://en.wikipedia.org/wiki/Representational_state_transfer">REST</a> APIs, designed for easy integration, fast response and high reliability. QPX Express searches combinations of airline schedules, fares and seat availability and includes search parameters such as origin, destination, dates and travel features such as maximum price and earliest departure. The API returns up to 500 itineraries for each request, including all data elements needed for customer display and a booking process. Moreover, it also allows to clearly see the structure of the data in #json both in input (JSON request) and output (Unformatted Response). For a simple test, use the <a target="_blank" rel="nofollow" href="http://qpx-express-demo.itasoftware.com/">Demo</a> with a Google account.</p>
<p>Before start using the QPX Express API, you need to get a Google API key. To get a key, follow these steps: 1. Go to the <a target="_blank" rel="nofollow" href="http://cloud.google.com/console">Google Developers Console</a> / 2. Create or select a project / 3. Click Continue to enable the API and any related services / 4. On the Credentials page, get an <strong>API key. </strong>With a single API key, it is possible to make 50 free requests per day (then the cost per single query is $ 0.035.)</p>
<p>The first thing I tried before writing the code in #R for data acquisition, manipulation and analysis was to test the <strong>client-server connection </strong>through a command line instruction using the cURL library (you can download <a target="_blank" rel="nofollow" href="http://curl.haxx.se/download.html">here</a> – add the certificate in the same directory otherwise the server will deny you access).</p>
<p>Let’s play a bit with the command line (Windows 10) before running the request:</p>
<pre><span># open the command prompt and the default folder should be system32 of the OS</span>
C:\WINDOWS\system32>

<span># set a convenient current directory (cd) for storing data - \ (enter) and / (exit) the path</span>
<span>cd</span> C:\Users\Roberto\Desktop

<span># your new position should be in </span>
C:\Users\Roberto\Desktop>


<span># I prefer using a partition of the HD dedicated to data instead of a messy desktop</span>
<span>cd</span> /D D:\

<span># create a new working directory (use &quot;&quot; for separated names otherwise two folders will be created)</span>
mkdir <span>&quot;flight data&quot;</span>
<span># set the current directory to the newly created folder</span>
<span>cd</span> <span>&quot;flight data&quot;</span>
<span># make sub-folders of the cd</span>
mkdir <span>&quot;curl test 1&quot;</span>
mkdir <span>&quot;curl test 2&quot;</span>
<span># check the cd, its features and its new sub-folders</span>
<span>cd</span>

dir

<span># finally, check the proper installation of curl using help</span>
<span># the list of the function parameters should appear with some description</span>

curl -help
</pre>
<p>After the creation of a repository, cURL can be used for testing the connection to QPX express and the collection of data in json using cURL parameters for HTTP requests and the structure of the JSON request the API proposes:</p>
<pre><span># Parameters</span>
<span># -X POST: parameter for POST request -> see: https://en.wikipedia.org/wiki/POST_(HTTP)</span>
<span># -H: Pass custom header LINE to server (H) -> see: https://en.wikipedia.org/wiki/List_of_HTTP_header_fields</span>
<span># -d: HTTP POST data, i.e. the input data in JSON structure </span>
<span># >: &quot;send to&quot; destination file</span>

curl -X POST <span>&quot;https://www.googleapis.com/qpxExpress/v1/trips/search?key={SERVER_KEY}&amp;alt=json&quot;</span> -H <span>&quot;Content-Type: application/json; charset=UTF-8&quot;</span> <span>-d</span> <span>&quot;{ \&quot;request\&quot;: { \&quot;slice\&quot;: [ { \&quot;origin\&quot;: \&quot;{ORIGIN_CODE}\&quot;, \&quot;destination\&quot;: \&quot;{DESTINATION_CODE}\&quot;, \&quot;date\&quot;: \&quot;{DATE_YYYY-MM-DD}\&quot; } ], \&quot;passengers\&quot;: { \&quot;adultCount\&quot;: 1, \&quot;infantInLapCount\&quot;: 0, \&quot;infantInSeatCount\&quot;: 0, \&quot;childCount\&quot;: 0, \&quot;seniorCount\&quot;: 0 }, \&quot;solutions\&quot;: 500, \&quot;refundable\&quot;: false }}&quot;</span> > <span>&quot;{RESULT_FILE}.json&quot;</span>
<span># Mac syntax</span>
curl -X POST <span>'https://www.googleapis.com/qpxExpress/v1/trips/search?key={SERVER_KEY}&amp;alt=json'</span> -H <span>'Content-Type: application/json; charset=UTF-8'</span> <span>-d</span> <span>'{ &quot;request&quot;: { &quot;slice&quot;: [ { &quot;origin&quot;: &quot;{ORIGIN_CODE}&quot;, &quot;destination&quot;: &quot;{DESTINATION_CODE}&quot;, &quot;date&quot;: &quot;{DATE_YYYY-MM-DD}&quot; } ], &quot;passengers&quot;: { &quot;adultCount&quot;: 1, &quot;infantInLapCount&quot;: 0, &quot;infantInSeatCount&quot;: 0, &quot;childCount&quot;: 0, &quot;seniorCount&quot;: 0 }, &quot;solutions&quot;: 500, &quot;refundable&quot;: false }}'</span>> <span>'{RESULT_FILE}.json'</span>
</pre>
<p>After a second, the prompt should show the downloading process stats and a new json file should appear in the cd “flight data” close to the previously created subfolders.</p>
<p>The connection works!</p>
<p>Next up – a post with code to <strong>translate</strong> the same commands in R – to manipulate the data comfortably after acquisition from the API.</p>
<p>#R #rstats #maRche #json #curl #qpxexpress #Rbloggers</p>
<p><em>This post is also shared in </em><em><a target="_blank" rel="nofollow" href="http://www.r-bloggers.com/">www.r-bloggers.com</a> and <a target="_blank" rel="nofollow" href="https://www.linkedin.com/pulse/web-data-acquisition-understanding-rcurl-from-command-roberto-palloni">LinkedIn</a></em></p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://r-posts.com/web-data-acquisition-understanding-rcurl-from-the-command-line-part-1/"> R posts</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/ASdjnEnSaac"></a></div>