<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/a-package-for-tidying-nested-lists/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://ropensci.org/blog/2018/06/26/roomba/"> rOpenSci - open tools for open science</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Data == knowledge! Much of the data we use, whether it be from<br>
government repositories, social media, GitHub, or e-commerce sites comes<br>
from public-facing APIs. The quantity of data available is truly<br>
staggering, but munging JSON output into a format that is easily<br>
analyzable in R is an equally staggering undertaking. When JSON is<br>
turned into an R object, it usually becomes a deeply nested list riddled<br>
with missing values that is difficult to untangle into a tidy format.<br>
Moreover, every API presents its own challenges; code you’ve written to<br>
clean up data from GitHub isn’t necessarily going to work on Twitter<br>
data, as each API spews data out in its own unique, headache-inducing<br>
nested list structure. To ease and generalize this process, <a rel="nofollow" href="https://github.com/ropensci/unconf18/issues/24" target="_blank">Amanda<br>
Dobbyn proposed</a> an<br>
<a rel="nofollow" href="https://ropensci.org/blog/2018/06/05/unconf18/" target="_blank">unconf18</a> project for a general API response tidier! Welcome <code>roomba</code>,<br>
our first stab at easing the process of tidying nested lists!</p>
<p><center><br>
<img alt="Drawing" src="https://i1.wp.com/ropensci.org/img/blog-images/2018-06-26-roomba/roomba_sticker.png?w=456&ssl=1" data-recalc-dims="1"><br>
</center><br>
<code>roomba</code> will eventually be able to walk nested lists in a variety of<br>
different structures from JSON output, replace <code>NULL</code> or <code>.empty</code> values<br>
with <code>NA</code>s or a user-specified value, and return a <code>tibble</code> with names<br>
matching a user-specified list. Of course, in two days we haven’t<br>
<em>fully</em> achieved this vision, but we’re off to a promising start.</p>
<h2>The birth of roomba</h2>
<p>It was clear Amanda was on to something good by the lively discussion in<br>
the <a rel="nofollow" href="https://github.com/ropensci/unconf18/issues/" target="_blank">#runconf18</a> issues<br>
repository leading up to the unconf. Thanks to input from Jenny Bryan,<br>
Jim Hester, Carl Boettinger, Scott Chamberlain, Bob Rudis, and Noam<br>
Ross, we had a lot of ideas to work with when the unconf began.<br>
Fortunately, Jim already had a function called <code>dfs_idx()</code><br>
(<a rel="nofollow" href="https://github.com/ropenscilabs/roomba/blob/master/R/dfs_idx.R" target="_blank">here</a>)<br>
written to perform depth-first searches of nested lists from the <a rel="nofollow" href="https://developer.github.com/v4/" target="_blank">GitNub<br>
GraphQL API</a>. With the core<br>
list-traversal code out of the way, we split our efforts between<br>
developing a usable interface, stockpiling <code>.JSON</code> files to test on, and<br>
developing a Shiny app.</p>
<h2>What’s working</h2>
<p>We’ve got the basic structure of <code>roomba</code> sorted out, and you should<br>
<a rel="nofollow" href="https://github.com/ropenscilabs/roomba/" target="_blank">install it from GitHub</a> to try out! Here are a few of the examples we’ve<br>
put together.</p>
<pre><code>library(roomba)
#load twitter data example
data(twitter_data)

#roomba-fy!
roomba(twitter_data, c(&quot;created_at&quot;, &quot;name&quot;))

## # A tibble: 24 x 2
##    name                 created_at                    
##    </code></pre>