<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/lmx-ot-nosj-interchanging-classic-data-formats-with-single-blackmagic-incantations/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://rud.is/b/2018/05/18/lmx-ot-nosj-interchanging-classic-data-formats-with-single-blackmagic-incantations/"> R – rud.is</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img data-orig-file="https://i1.wp.com/rud.is/b/wp-content/uploads/2018/05/blackmagic.jpg?fit=700%2C648&amp;ssl=1" data-orig-size="700,648" data-medium-file="https://i1.wp.com/rud.is/b/wp-content/uploads/2018/05/blackmagic.jpg?fit=300%2C278&amp;ssl=1" data-attachment-id="10289" data-image-meta="{" data-image-description="" src="https://i1.wp.com/rud.is/b/wp-content/uploads/2018/05/blackmagic.jpg?resize=700%2C648&ssl=1" data-permalink="https://rud.is/b/2018/05/18/lmx-ot-nosj-interchanging-classic-data-formats-with-single-blackmagic-incantations/blackmagic/" data-large-file="https://i1.wp.com/rud.is/b/wp-content/uploads/2018/05/blackmagic.jpg?fit=530%2C491&amp;ssl=1" alt="" data-image-title="blackmagic" width="450" data-comments-opened="1" class="wp-image-10289" data-recalc-dims="1"></p>
<p>The D.C. Universe magic hero <a rel="nofollow" href="https://en.wikipedia.org/wiki/Zatanna" target="_blank">Zatanna</a> used spells (i.e. incantations) to battle foes and said spells were just sentences said backwards, hence the mixed up jumble in the title. But, now I’m regretting not naming the package <code>zatanna</code> and reversing the function names to help ensure they’re only used deliberately &amp; carefully. You’ll see why in a bit.</p>
<p>Just like their ore-seeking speleological counterparts, workers in our modern day data mines process a multitude of mineralic data formats to achieve our goals of <strike>world domination</strike> finding meaning, insight &amp; solutions to hard problems.</p>
<p>Two formats in particular are common occurrences in many of our <code>$DAYJOB</code>s: XML and JSON. The rest of this (hopefully short-ish) post is going to assume you have at least a passing familiarity with — if not full-on battle scars from working with — them.</p>
<p>XML and JSON are, in many ways, very similar. This similarity is on purpose since JSON was originally created to make is easier to process data in browsers and help make data more human-readable. If your <code>$DAYJOB</code> involves processing small or large streams of nested data, you likely prefer JSON over XML.</p>
<p>There are times, though, that — even if one generally works with only JSON data — one comes across a need to ingest some XML and turn it into JSON. This was the case for a question-poser on Stack Overflow this week (I won’t point-shill with a direct link but it’ll be easy to find if you are interested in this latest <a rel="nofollow" href="https://rud.is/b/2017/09/28/sodd-stackoverflow-driven-development/" target="_blank">SODD</a> package).</p>
<p>Rather than take on the potentially painful task of performing the XML to JSON transformation on their own the <a rel="nofollow" href="https://meta.stackexchange.com/questions/40353/stack-exchange-glossary-dictionary-of-commonly-used-terms" target="_blank">OP</a> wished for a simple incantation to transform the entirety of the incoming XML into JSON.</p>
<p><img data-orig-file="https://i1.wp.com/rud.is/b/wp-content/uploads/2018/05/price.png?fit=527%2C439&amp;ssl=1" data-orig-size="527,439" data-medium-file="https://i1.wp.com/rud.is/b/wp-content/uploads/2018/05/price.png?fit=300%2C250&amp;ssl=1" data-attachment-id="10288" data-image-meta="{" data-image-description="" src="https://i1.wp.com/rud.is/b/wp-content/uploads/2018/05/price.png?resize=527%2C439&ssl=1" data-permalink="https://rud.is/b/2018/05/18/lmx-ot-nosj-interchanging-classic-data-formats-with-single-blackmagic-incantations/price/" data-large-file="https://i1.wp.com/rud.is/b/wp-content/uploads/2018/05/price.png?fit=527%2C439&amp;ssl=1" alt="" data-image-title="price" width="450" data-comments-opened="1" class="wp-image-10288" data-recalc-dims="1"></p>
<p>We’ll switch comic universes for a moment to issue a warning that all magic comes with a price. And, the cost for automatic XML&lt;->JSON conversion can be quite high. XML has namespaces, attributes tags and values and requires schemas to convey data types and help validate content structure. JSON has no attributes, implicitly conveys types and is generally schema-less (though folks have <a rel="nofollow" href="http://json-schema.org/" target="_blank">bolted on</a> that concept).</p>
<p>If one is going to use magic for automatic data conversion there must be rules (no, <em>not</em> <a rel="nofollow" href="https://en.wikipedia.org/wiki/Magic:_The_Gathering_rules" target="_blank">those kind</a> of Magic rules), otherwise how various aspects of XML become encoded into JSON (and the reverse) will generate inconsistency and may even result in significant data corruption. Generally speaking, you are <em>always</em> better off writing <em>your own</em> conversion utility vs rely on specific settings in a general conversion script/function. However, if your need is a one-off (which anyone who has been doing this type of work for a while knows is also generally never the case) you may have cause to throw caution to the wind, get your quick data fix, and move on. If that is the case, the <a rel="nofollow" href="https://github.com/hrbrmstr/blackmagic/" target="_blank"><code>blackmagic</code><img alt="🔗" src="https://i2.wp.com/s.w.org/images/core/emoji/2.4/72x72/1f517.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"></a> package may be of use to you.</p>
<h3>gnitrevnoC eht ANAI sserddA ecapS yrtsigeR ot NOSJ</h3>
<p>One file that’s in XML that I only occasionally have to process is the <a rel="nofollow" href="https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.xml" target="_blank">IANA IPv4 Address Space Registry</a>. If you visited that link you may have uttered <em>“Hey! That’s not XML it’s HTML!”</em>, to wit — I would respond — <em>“Well, HTML is really XML anyway, but use the View Source, Luke! and see that it is indeed XML with some clever XSL style sheet processing being applied in-browser to make the gosh awful XML human readable.”</em>.</p>
<p>With <code>blackmagic</code> we can make quick work of converting this monstrosity into JSON.</p>
<p>The <code>blackmagic</code> package itself uses even <em>darker</em> magic to accomplish its goals. The package is just a thin <code>V8</code> wrapper around the <a rel="nofollow" href="https://www.npmjs.com/package/xml-js" target="_blank"><code>xml-js</code><img alt="🔗" src="https://i2.wp.com/s.w.org/images/core/emoji/2.4/72x72/1f517.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"></a> javascript library. Because of this, it is recommended that you do not try to process gigabytes of XML with it as there is a round trip of <a rel="nofollow" href="https://en.wikipedia.org/wiki/Marshalling_(computer_science)" target="_blank">data marshalling</a> between R and the embedded v8 engine.</p>
<pre><code>
requireNamespace(&quot;jsonlite&quot;) # jsonlite::flatten clobbers purrr::flatten in the wrong order so I generally fully-qualify what I need
## Loading required namespace: jsonlite
library(xml2)
library(blackmagic) # devtools::install_github(&quot;hrbrmstr/blackmagic&quot;)
library(purrr)
requireNamespace(&quot;dplyr&quot;) # I'm going to fully qualify use of dplyr:data_frame() below
## Loading required namespace: dplyr
</code></pre>
<p>You can thank <a rel="nofollow" href="http://twitter.com/yoniceedee" target="_blank">@yoniceedee</a> <a rel="nofollow" href="https://github.com/hrbrmstr/blackmagic/issues/1" target="_blank">for the URL processing capability in <code>blackmagic</code></a>:</p>
<pre><code>
source_url &lt;- &quot;https://www.iana.org/assignments/ipv4-address-space/ipv4-address-space.xml&quot;

iana_json &lt;- blackmagic::xml_to_json(source_url)

# NOTE: cat the whole iana_json locally to see it — perhaps to file=&quot;...&quot; vs clutter your console
cat(substr(iana_json, 1800, 2300))
## me&quot;:&quot;prefix&quot;,&quot;elements&quot;:[{&quot;type&quot;:&quot;text&quot;,&quot;text&quot;:&quot;000/8&quot;}]},{&quot;type&quot;:&quot;element&quot;,&quot;name&quot;:&quot;designation&quot;,&quot;elements&quot;:[{&quot;type&quot;:&quot;text&quot;,&quot;text&quot;:&quot;IANA - Local Identification&quot;}]},{&quot;type&quot;:&quot;element&quot;,&quot;name&quot;:&quot;date&quot;,&quot;elements&quot;:[{&quot;type&quot;:&quot;text&quot;,&quot;text&quot;:&quot;1981-09&quot;}]},{&quot;type&quot;:&quot;element&quot;,&quot;name&quot;:&quot;status&quot;,&quot;elements&quot;:[{&quot;type&quot;:&quot;text&quot;,&quot;text&quot;:&quot;RESERVED&quot;}]},{&quot;type&quot;:&quot;element&quot;,&quot;name&quot;:&quot;xref&quot;,&quot;attributes&quot;:{&quot;type&quot;:&quot;note&quot;,&quot;data&quot;:&quot;2&quot;}}]},{&quot;type&quot;:&quot;element&quot;,&quot;name&quot;:&quot;record&quot;,&quot;elements&quot;:[{&quot;type&quot;:&quot;element&quot;,&quot;name&quot;:&quot;prefix&quot;,&quot;elements&quot;:[{&quot;type&quot;:&quot;
</code></pre>
<p>By by the hoary hosts of Hoggoth that's not very &quot;human readable&quot;! And, it looks super-verbose. Thankfully, Yousuf Almarzooqi <a rel="nofollow" href="https://www.npmjs.com/package/xml-js#options-for-converting-xml-%E2%86%92-js-object--json" target="_blank">knew we'd want to fine-tune the output</a> and we can use those options to make this a bit better:</p>
<pre><code>
blackmagic::xml_to_json(
  doc = source_url, 
  spaces = 2,                # Number of spaces to be used for indenting XML output
  compact = FALSE,           # Whether to produce detailed object or compact object
  ignoreDeclaration = TRUE   # No declaration property will be generated.
) -> iana_json

# NOTE: cat the whole iana_json locally to see it — perhaps to file=&quot;...&quot; vs clutter your console
cat(substr(iana_json, 3000, 3300))
## pe&quot;: &quot;element&quot;,
##               &quot;name&quot;: &quot;prefix&quot;,
##               &quot;elements&quot;: [
##                 {
##                   &quot;type&quot;: &quot;text&quot;,
##                   &quot;text&quot;: &quot;000/8&quot;
##                 }
##               ]
##             },
##             {
##               &quot;type&quot;: &quot;element&quot;,
##               &quot;name&quot;: &quot;designation&quot;,
## 
</code></pre>
<p>One &quot;plus side&quot; for doing the mass-conversion is that we don't really need to do much processing to have it be &quot;usable&quot; data in R:</p>
<pre><code>
blackmagic::xml_to_json(
  doc = source_url, 
  compact = FALSE,        
  ignoreDeclaration = TRUE
) -> iana_json

# NOTE: consider taking some more time to explore this monstrosity than this
str(processed &lt;- jsonlite::fromJSON(iana_json), 3)
## List of 1
##  $ elements:'data.frame':    3 obs. of  5 variables:
##   ..$ type       : chr [1:3] &quot;instruction&quot; &quot;instruction&quot; &quot;element&quot;
##   ..$ name       : chr [1:3] &quot;xml-stylesheet&quot; &quot;oxygen&quot; &quot;registry&quot;
##   ..$ instruction: chr [1:3] &quot;type=\&quot;text/xsl\&quot; href=\&quot;ipv4-address-space.xsl\&quot;&quot; &quot;RNGSchema=\&quot;ipv4-address-space.rng\&quot; type=\&quot;xml\&quot;&quot; NA
##   ..$ attributes :'data.frame':  3 obs. of  2 variables:
##   .. ..$ xmlns: chr [1:3] NA NA &quot;http://www.iana.org/assignments&quot;
##   .. ..$ id   : chr [1:3] NA NA &quot;ipv4-address-space&quot;
##   ..$ elements   :List of 3
##   .. ..$ : NULL
##   .. ..$ : NULL
##   .. ..$ :'data.frame':  280 obs. of  4 variables:

compact(processed$elements$elements[[3]]$elements) %>% 
  head(6) %>% 
  str(3) 
## List of 6
##  $ :'data.frame':    1 obs. of  2 variables:
##   ..$ type: chr &quot;text&quot;
##   ..$ text: chr &quot;IANA IPv4 Address Space Registry&quot;
##  $ :'data.frame':    1 obs. of  2 variables:
##   ..$ type: chr &quot;text&quot;
##   ..$ text: chr &quot;Internet Protocol version 4 (IPv4) Address Space&quot;
##  $ :'data.frame':    1 obs. of  2 variables:
##   ..$ type: chr &quot;text&quot;
##   ..$ text: chr &quot;2018-04-23&quot;
##  $ :'data.frame':    3 obs. of  4 variables:
##   ..$ type      : chr [1:3] &quot;text&quot; &quot;element&quot; &quot;text&quot;
##   ..$ text      : chr [1:3] &quot;Allocations to RIRs are made in line with the Global Policy published at &quot; NA &quot;. \nAll other assignments require IETF Review.&quot;
##   ..$ name      : chr [1:3] NA &quot;xref&quot; NA
##   ..$ attributes:'data.frame':   3 obs. of  2 variables:
##   .. ..$ type: chr [1:3] NA &quot;uri&quot; NA
##   .. ..$ data: chr [1:3] NA &quot;http://www.icann.org/en/resources/policy/global-addressing&quot; NA
##  $ :'data.frame':    3 obs. of  4 variables:
##   ..$ type      : chr [1:3] &quot;text&quot; &quot;element&quot; &quot;text&quot;
##   ..$ text      : chr [1:3] &quot;The allocation of Internet Protocol version 4 (IPv4) address space to various registries is listed\nhere. Origi&quot;| __truncated__ NA &quot; documents most of these allocations.&quot;
##   ..$ name      : chr [1:3] NA &quot;xref&quot; NA
##   ..$ attributes:'data.frame':   3 obs. of  2 variables:
##   .. ..$ type: chr [1:3] NA &quot;rfc&quot; NA
##   .. ..$ data: chr [1:3] NA &quot;rfc1466&quot; NA
##  $ :'data.frame':    5 obs. of  4 variables:
##   ..$ type      : chr [1:5] &quot;element&quot; &quot;element&quot; &quot;element&quot; &quot;element&quot; ...
##   ..$ name      : chr [1:5] &quot;prefix&quot; &quot;designation&quot; &quot;date&quot; &quot;status&quot; ...
##   ..$ elements  :List of 5
##   .. ..$ :'data.frame':  1 obs. of  2 variables:
##   .. ..$ :'data.frame':  1 obs. of  2 variables:
##   .. ..$ :'data.frame':  1 obs. of  2 variables:
##   .. ..$ :'data.frame':  1 obs. of  2 variables:
##   .. ..$ : NULL
##   ..$ attributes:'data.frame':   5 obs. of  2 variables:
##   .. ..$ type: chr [1:5] NA NA NA NA ...
##   .. ..$ data: chr [1:5] NA NA NA NA ...
</code></pre>
<p>As noted previously, all magic comes with a price and we just traded XML processing for some gnarly list processing. This isn't the case for all XML files and you can try to tweak the parameters to <code>xml_to_json()</code> to make the output more usable (NOTE: <a rel="nofollow" href="https://www.npmjs.com/package/xml-js#options-for-changing-key-names" target="_blank">key name transformation parameters</a> still need to be implemented in the package), but this seems a <em>whole</em> lot easier (to me):</p>
<pre><code>
doc &lt;- read_xml(source_url)

xml_ns_strip(doc)

dplyr::data_frame(
  
  prefix = xml_find_all(doc, &quot;.//record/prefix&quot;) %>% xml_text(),
  
  designation = xml_find_all(doc, &quot;.//record/designation&quot;) %>% xml_text(),
  
  date = xml_find_all(doc, &quot;.//record/date&quot;) %>% 
    xml_text() %>% 
    sprintf(&quot;%s-01&quot;, .) %>% 
    as.Date(),
  
  whois = xml_find_all(doc, &quot;.//record&quot;) %>% 
    map(xml_find_first, &quot;./whois&quot;) %>% 
    map_chr(xml_text),
  
  status = xml_find_all(doc, &quot;.//record/status&quot;) %>% xml_text()
  
)
## # A tibble: 256 x 5
##    prefix designation                     date       whois        status  
##    </code></pre>