<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/web-data-acquisition-parsing-json-objects-with-tidyjson-part-3/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://r-posts.com/web-data-acquisition-parsing-json-objects-with-tidyjson-part-3/"> R-posts.com</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img data-image-description="" data-orig-file="https://i1.wp.com/r-posts.com/wp-content/uploads/2017/03/json.png?fit=383%2C229" data-orig-size="383,229" data-image-title="json" class="wp-image-69" data-comments-opened="1" data-recalc-dims="1" data-attachment-id="69" data-permalink="http://r-posts.com/web-data-acquisition-parsing-json-objects-with-tidyjson-part-3/json/" alt="" data-medium-file="https://i1.wp.com/r-posts.com/wp-content/uploads/2017/03/json.png?fit=383%2C229" data-large-file="https://i1.wp.com/r-posts.com/wp-content/uploads/2017/03/json.png?fit=383%2C229" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" src="https://i1.wp.com/r-posts.com/wp-content/uploads/2017/03/json.png?resize=383%2C229"></p>
<p>The collection of example flight data in json format available in <a target="_blank" rel="nofollow" href="http://www.linkedin.com/pulse/web-data-acquisition-structure-rcurl-request-part-2-roberto-palloni">part 2</a>, described the libraries and the structure of the POST request necessary to collect data in a json object. Despite the process generated and transferred locally a proper response, the data collected were neither in a suitable structure for data analysis nor immediately readable. They appears as just a long string of information nested and separated according to the JavaScript object notation syntax.</p>
<p>Thus, to visualize the deeply nested json object and make it human readable and understandable for further processing, the json content could be copied and pasted in a common online <a target="_blank" rel="nofollow" href="http://www.jsoneditoronline.org/">parser</a>. The tool allows to select each node of the tree and observe the data structure up to the variables and data of interest for the statistical analysis. The bulk of the relevant information for the purpose of the analysis on flight prices are hidden in the <em>tripOption</em> node as shown in the following figure (only 50 flight solutions were requested).</p>
<p><img sizes="(max-width: 450px) 100vw, 450px" data-image-description="" data-orig-file="https://i1.wp.com/r-posts.com/wp-content/uploads/2017/03/parser.png?fit=1112%2C705" data-orig-size="1112,705" data-image-title="parser" data-comments-opened="1" class="wp-image-68" data-recalc-dims="1" data-attachment-id="68" data-permalink="http://r-posts.com/web-data-acquisition-parsing-json-objects-with-tidyjson-part-3/parser/" alt="" data-medium-file="https://i1.wp.com/r-posts.com/wp-content/uploads/2017/03/parser.png?fit=450%2C285" data-large-file="https://i1.wp.com/r-posts.com/wp-content/uploads/2017/03/parser.png?fit=450%2C285" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" src="https://i1.wp.com/r-posts.com/wp-content/uploads/2017/03/parser.png?resize=450%2C285"></p>
<p>However, looking deeply into the object, several other elements are provided as the distance in mile, the segment, the duration, the carrier, etc. The R parser to transform the json structure in a usable dataframe requires the dplyr library for using the pipe operator (%>%) to streamline the code and make the parser more readable. Nevertheless, the library actually wrangling through the lines is tidyjson and its powerful functions:</p>
<ul>
<li>enter_object: enters and dives into a data object;</li>
<li>gather_array: stacks a JSON array;</li>
<li>spread_values: creates new columns from values assigning specific type (e.g. jstring, jnumber).</li>
</ul>
<pre>

library(dplyr)      # for pipe operator %>% and other dplyr functions
library(tidyjson)   # https://cran.r-project.org/web/packages/tidyjson/vignettes/introduction-to-tidyjson.html

data_items &lt;- datajson %>%
  spread_values(kind = jstring(&quot;kind&quot;)) %>%
  spread_values(trips.kind = jstring(&quot;trips&quot;,&quot;kind&quot;)) %>%
  spread_values(trips.rid = jstring(&quot;trips&quot;,&quot;requestId&quot;)) %>%
  enter_object(&quot;trips&quot;,&quot;tripOption&quot;) %>%
  gather_array %>%
  spread_values(
    id = jstring(&quot;id&quot;),
    saleTotal = jstring(&quot;saleTotal&quot;)) %>%
  enter_object(&quot;slice&quot;) %>% 
  gather_array %>%
  spread_values(slice.kind = jstring(&quot;kind&quot;)) %>%
  spread_values(slice.duration = jstring(&quot;duration&quot;)) %>%
  enter_object(&quot;segment&quot;) %>% 
  gather_array %>%
  spread_values(
    segment.kind = jstring(&quot;kind&quot;),
    segment.duration = jnumber(&quot;duration&quot;),
    segment.id = jstring(&quot;id&quot;),
    segment.cabin = jstring(&quot;cabin&quot;)) %>%
  enter_object(&quot;leg&quot;) %>% 
  gather_array %>%
  spread_values(
    segment.leg.aircraft = jstring(&quot;aircraft&quot;),
    segment.leg.origin = jstring(&quot;origin&quot;),
    segment.leg.destination = jstring(&quot;destination&quot;),
    segment.leg.mileage = jnumber(&quot;mileage&quot;)) %>%
  select(kind, trips.kind, trips.rid,
         saleTotal,id, slice.kind, slice.duration,
         segment.kind, segment.duration, segment.id,
         segment.cabin, segment.leg.aircraft, segment.leg.origin,
         segment.leg.destination, segment.leg.mileage)

head(data_items)
                    kind             trips.kind              trips.rid saleTotal
1 qpxExpress#tripsSearch qpxexpress#tripOptions UnxCOx4nKIcIOpRiG0QBOe EUR178.38
2 qpxExpress#tripsSearch qpxexpress#tripOptions UnxCOx4nKIcIOpRiG0QBOe EUR178.38
3 qpxExpress#tripsSearch qpxexpress#tripOptions UnxCOx4nKIcIOpRiG0QBOe EUR235.20
4 qpxExpress#tripsSearch qpxexpress#tripOptions UnxCOx4nKIcIOpRiG0QBOe EUR235.20
5 qpxExpress#tripsSearch qpxexpress#tripOptions UnxCOx4nKIcIOpRiG0QBOe EUR248.60
6 qpxExpress#tripsSearch qpxexpress#tripOptions UnxCOx4nKIcIOpRiG0QBOe EUR248.60
                         id           slice.kind slice.duration
1 ftm7QA6APQTQ4YVjeHrxLI006 qpxexpress#sliceInfo            510
2 ftm7QA6APQTQ4YVjeHrxLI006 qpxexpress#sliceInfo            510
3 ftm7QA6APQTQ4YVjeHrxLI009 qpxexpress#sliceInfo            490
4 ftm7QA6APQTQ4YVjeHrxLI009 qpxexpress#sliceInfo            490
5 ftm7QA6APQTQ4YVjeHrxLI007 qpxexpress#sliceInfo            355
6 ftm7QA6APQTQ4YVjeHrxLI007 qpxexpress#sliceInfo            355
            segment.kind segment.duration       segment.id segment.cabin
1 qpxexpress#segmentInfo              160 GixYrGFgbbe34NsI         COACH
2 qpxexpress#segmentInfo              235 Gj1XVe-oYbTCLT5V         COACH
3 qpxexpress#segmentInfo              190 Grt369Z0shJhZOUX         COACH
4 qpxexpress#segmentInfo              155 GRvrptyoeTfrSqg8         COACH
5 qpxexpress#segmentInfo              100 GXzd3e5z7g-5CCjJ         COACH
6 qpxexpress#segmentInfo              105 G8axcks1R8zJWKrN         COACH
  segment.leg.aircraft segment.leg.origin segment.leg.destination segment.leg.mileage
1                  320                FCO                     IST                 859
2                  77W                IST                     LHR                1561
3                  73H                FCO                     ARN                1256
4                  73G                ARN                     LHR                 908
5                  319                FCO                     STR                 497
6                  319                STR                     LHR                 469

</pre>
<p>Data are now in an R-friendly structure despite not yet ready for analysis. As can be observed from the first rows, each record has information on a single <em>segment</em> of the flight selected. A further step of aggregation using some SQL is needed in order to end up with a dataframe of flights data suitable for statistical analysis.</p>
<p>Next up, the aggregation, some <strong>data analysis</strong> and <strong>data visualization</strong> to complete the journey through the web data acquisition using R.</p>
<p>#R #rstats #maRche #json #curl #tidyjson #Rbloggers</p>
<p><em>This post is also shared in </em><a target="_blank" rel="nofollow" href="http://www.r-bloggers.com/"><em>www.r-bloggers.com</em></a><em> and </em><a target="_blank" rel="nofollow" href="https://www.linkedin.com/pulse/web-data-acquisition-parsing-json-objects-tidyjson-part-palloni"><em>LinkedIn</em></a></p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://r-posts.com/web-data-acquisition-parsing-json-objects-with-tidyjson-part-3/"> R-posts.com</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/jJEEWzysmmA"></a></div>