<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/parsing-metadata-with-r-a-package-story/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://ropensci.org/blog/2018/10/09/jstor/"> rOpenSci - open tools for open science</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Every R package has its story. Some packages are written by experts, some by<br>
novices. Some are developed quickly, others were long in the making. This is the<br>
story of <code>jstor</code>, a package which I developed during my time as a student of<br>
sociology, working in a research project on the scientific elite within<br>
sociology. Writing the package has taught me many things (more on that later)<br>
and it is deeply gratifying to see, that others find the package useful.</p>
<p>Before taking a look at the package’s inception however, I want to give you some<br>
basic<br>
information. <a rel="nofollow" href="https://www.jstor.org/" target="_blank">JSTOR</a> is a large archive for scientific<br>
texts, mainly known for<br>
their coverage of journal articles, although they recently added book chapters<br>
and other sources as well. They make all of their content available for<br>
researchers with their service <a rel="nofollow" href="https://www.jstor.org/dfr/" target="_blank">Data for Research</a>.<br>
With this data you can do text mining, citation analysis, and everything else<br>
you could come up with, using<br>
metadata and data about the content of the articles. The package<br>
I wrote, <code>jstor</code>, provides functions to import the metadata, and it has a few<br>
helper functions for common cleaning tasks. Everything you need to know about<br>
how to use it is on the <a rel="nofollow" href="https://ropensci.github.io/jstor/" target="_blank">package website</a>.<br>
You will find three vignettes, with a general introduction, examples on how to<br>
import many files at once, and a few examples of known quirks of JSTORs data.<br>
There is also a lengthy case study that shows how to combine metadata and<br>
information on the articles’ content. If you still need more information, you<br>
can also watch<br>
<a rel="nofollow" href="https://www.youtube.com/watch?v=kNRbT-ki9tU&t" target="_blank">my presentation at this year’s useR! in Brisbane</a> or dive into the<br>
<a rel="nofollow" href="https://speakerdeck.com/tklebel/jstor-an-r-package-for-analysing-scientific-articles" target="_blank">slides</a>.<br>
But for now, let us turn back the clock<br>
to follow along my journey of developing the package.</p>
<h3>Hacking Away</h3>
<p>Back in March 2017, I was starting out as a MA-student of<br>
sociology in a research project<br>
concerned with the scientific elite within sociology and economics. The project<br>
had many goals but writing an R package was not one of them. At the start,<br>
I was presented with a dataset which was huge, at least for my terms:<br>
around 30GB of data, half of which was text, the other half 500,000 <code>.xml</code>-files.<br>
The dataset was incredible in its depth: we had all articles<br>
from JSTOR which belonged to the topics “sociology” and “economics”. To repeat:<br>
all articles that JSTOR has on those topics for all years JSTOR has data on.</p>
<p>My task was to somehow make this data accessible for our research. Since we are<br>
sociologists and no computer experts, and my knowledge of R was mainly<br>
self-taught, my approach was quite ad-hoc: “Let’s see, how we can extract<br>
relevant information for one file, and then maybe we can <code>lapply</code> over the whole<br>
set of files.” That is what the tidyverse philosophy and <code>purrr</code> tell you to do:<br>
solve it for one case using a function and apply this function to the whole<br>
set of files, cases, nested rows, or whatever. Long story short, you can do it<br>
like that, and I surely did it like that. But if I could start over and write<br>
a new version of the package, I would<br>
probably do a few things differently, one of which I discuss near the end of the<br>
post.</p>
<h4>How (not) to import .xml-files</h4>
<p>So, I had to start somewhere, and that was obviously importing the data into R.<br>
After searching and trying different packages, I settled on<br>
<code>xml2::read_xml()</code>. But then what. I had done a few pet projects with<br>
web-scraping but had no knowledge of XPATH-expressions and how to access<br>
certain parts of the document directly. After some stumbling around, I had found<br>
<code>xml2::as_list()</code> and was very happy: I could turn this unpleasant creature of<br>
XML into a pretty list, which I was accustomed to. My happiness is obvious when<br>
looking at the relevant commit message and the corresponding changes.</p>
<p><img alt="" src="https://i2.wp.com/ropensci.org/img/blog-images/2018-10-09-jstor/lists.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>After turning the XML into a list, I would use something<br>
like <code>listviewer::jsonedit()</code> to inspect the elements, and extract what I<br>
needed. The approach was cumbersome, and the code was not pretty, since the<br>
original documents are deeply nested, and the structure was not always the same.<br>
But it worked, and I was happy with it, for the time being.</p>
<p>My functions looked something like this:</p>
<pre><code>extract_contributors &lt;- function(contributors) {
  if (is.null(contributors)) {
    return(NA)
  } else {
    contributors %>%
      map(extract_name) %>%
      bind_rows() %>%
      rename(given_names = `given-names`) %>%
      mutate(author_number = 1:n())
  }
}


find_meta &lt;- function(meta_list) {
  front &lt;- meta_list$front
  article &lt;- front$`article-meta`

  # authors
  contributors &lt;- article$`contrib-group`
  contributors &lt;- extract_contributors(contributors)

  # piece together all elements
  data_frame(
    journal_id = front$`journal-meta`$`journal-id`[[1]],
    article_id = article$`article-id`[[1]],
    article_title = article$`title-group`$`article-title`[[1]],
    authors = list(contributors),
    article_pages = extract_pages(article)
  )
}
</code></pre>
<p>As you can see, there is a lot of indexing (like in<br>
<code>journal_id = front$journal-meta$journal-id[[1]]</code>) and <code>map</code>ping over<br>
lists.</p>
<p>The function would have been applied like so:</p>
<pre><code># one file
&quot;file_path.xml&quot; %>% 
  read_xml() %>% 
  as_list() %>% 
  find_meta()
  
# many files
c(&quot;mutliple.xml&quot;, &quot;file.xml&quot;, &quot;paths.xml&quot;) %>% 
  map(read_xml) %>% 
  map(as_list) %>% 
  map(find_meta)
</code></pre>
<p>As can be expected when parsing deeply nested lists which do not always have<br>
the same structure, this quickly escalated into<br>
more and more complex and sometimes quite ridiculous functions:</p>
<pre><code>extract_name &lt;- function(x) {
  x %>%
    flatten() %>%
    flatten() %>%
    flatten() %>%
    .[!(names(.) %in% c(&quot;x&quot;, &quot;addr-line&quot;))] %>%
    as_data_frame()
}
</code></pre>
<p>But, as I said, I was solving problems with the tools I already knew, so I was<br>
happy.</p>
<h4>Benchmarking my code</h4>
<p>At some point though, I started benchmarking my solutions. I experimented with<br>
<code>mclapply</code> and it doubled the speed of execution on my machine, but it was still<br>
very slow: Parsing the content <em>after</em> importing and transforming it into a list<br>
took 7.2 seconds for 3,000 files. For 200,000 files of sociological<br>
articles this would amount to 8 hours. And importing and<br>
converting to a list took a lot of time too. Although I didn’t think at that<br>
time, that I would<br>
need to import the files more than once or at most twice<br>
(which I had to do later to update results<br>
after we got new versions with better data), I decided that this was too slow,<br>
and I looked for better options.</p>
<p>The first idea which I had was simply to scale computing power: I had a faster<br>
machine at home and planned to read the data into R at work, save it as <code>.rds</code>,<br>
and then process it at home on the faster machine (for some reason I didn’t want<br>
to carry the original data home). Apparently, this is not as easy as it sounds.<br>
If you parse a file with <code>xml2::read_xml()</code> you don’t get the content of the<br>
file, but only an object that points to the file, and you can then extract<br>
content with <code>xml_child()</code> and other functions.<br>
But you cannot use <code>readr::write_rds</code> to save the file, instead you need<br>
<code>xml2::xml_serialize()</code> and <code>xml2::xml_unserialize()</code> and they require an open<br>
connection, and so on. I can tell you that, since Jim Hester kindly responded to <a rel="nofollow" href="https://stackoverflow.com/questions/44070577/write-xml-object-to-disk" target="_blank">my<br>
question on StackOverflow</a>.<br>
He had a further hint:</p>
<blockquote>
<p>[…] I would seriously consider using XPATH expressions to extract the<br>
desired data, even if you have to rewrite code. […] Xpath extracting just the elements you are interested in is going to run much faster than converting the entire data to a list first then manipulating that. ~Jim Hester</p>
</blockquote>
<p>And this is what I did. Just for the fun of it, I benchmarked two versions: my<br>
function before re-writing to using XPATH expressions, and the function today.<br>
The current version is a lot safer and handles many edge-cases, but it is still<br>
around 5 times faster. Furthermore, the processing time depends less on how complex the original<br>
file is, and more on how much information is being read. With the new version<br>
of the function, around 30% of the time is spent on reading the file from disk,<br>
while the remaining 70% deal with parsing the content. In contrast, the old<br>
version spent around 85% of the time on parsing the file.</p>
<pre><code>microbenchmark::microbenchmark(
  convert = jst_example(&quot;sample_with_references.xml&quot;) %>% 
    read_xml() %>% 
    as_list(),
    
  old = jst_example(&quot;sample_with_references.xml&quot;) %>% 
    read_xml() %>% 
    as_list() %>% 
    find_meta(),
  
  new = jst_get_article(jst_example(&quot;sample_with_references.xml&quot;))
)
</code></pre>
<pre><code>## Unit: milliseconds
##     expr       min        lq     mean    median       uq       max neval
##  convert 35.621488 39.274647 44.29424 43.530006 46.64541 114.12766   100
##      old 42.038368 46.596185 51.49769 49.711072 52.43276 134.12999   100
##      new  8.675674  9.054154 10.33858  9.365577 10.16726  23.88457   100
</code></pre>
<p>I uploaded the old code to a gist: <a rel="nofollow" href="https://gist.github.com/tklebel/11d2664ac2bc754ddc367369eec1a660" target="_blank">old_functions.R</a>.<br>
If you want to run the benchmark on your own, execute the following snippet<br>
first:</p>
<pre><code># load current functions
library(jstor)

# load old function
devtools::source_gist(
  &quot;https://gist.github.com/tklebel/11d2664ac2bc754ddc367369eec1a660&quot;,
  filename = &quot;old_functions.R&quot;
)

library(tidyverse)
library(xml2)
</code></pre>
<p></p>
<h3>Doing It Properly</h3>
<h4>Working with XPATH-expressions</h4>
<p>Rewriting my functions was not that much of a hassle, in the end. I had turned<br>
my functions into a package early on and had already included many test cases<br>
with <a rel="nofollow" href="https://github.com/r-lib/testthat" target="_blank">testthat</a><br>
to make sure everything works as expected. This helped a lot for re-structuring<br>
the code, since I already knew what my output should look like, and I only had<br>
to change the intermediate steps.</p>
<p>The work on re-writing progressed quickly. The first step was to simply extract<br>
the identifier for the journal within which the article was published:</p>
<pre><code>find_meta &lt;- function(xml_file) {
  if (identical(&quot;xml_document&quot; %in% class(xml_file), FALSE)) {
    stop(&quot;Input must be a `xml_document`&quot;, call. = FALSE)
  }

  front &lt;- xml_find_all(xml_file, &quot;front&quot;)

  data_frame(
    journal_id = extract_jcode(front)
  )
}


extract_jcode &lt;- function(front) {
  front %>%
    xml_child(&quot;journal-meta&quot;) %>%
    xml_find_first(&quot;journal-id&quot;) %>%
    xml_text()
}
</code></pre>
<p>I quickly added new parts, although it didn’t feel all too easy: I basically had<br>
to learn how to navigate within the document via XPATH.</p>
<p>Another thing I took up<br>
during this re-write was regularly benchmarking my functions. Since I lacked<br>
the technical expertise to know up front which approach would be faster,<br>
I benchmarked possibilities to figure it out. The following benchmark<br>
compares three versions of extracting the text from the node named <code>volume</code>.<br>
Especially the first two seemed very similar to me and it was not at all<br>
obvious, which one would be faster.</p>
<pre><code>test_file &lt;- jst_example(&quot;sample_with_references.xml&quot;) %>% xml2::read_xml()

fun1 &lt;- function() {
  xml_find_all(test_file, &quot;front/.//volume/.//text()&quot;) %>% as.character()
}
fun2 &lt;- function() {
  xml_find_all(test_file, &quot;front/.//volume&quot;) %>% xml_text()
}
fun3 &lt;- function() {
  xml_find_all(test_file, &quot;front&quot;) %>%
    xml_child(&quot;article-meta&quot;) %>%
    xml_child(&quot;volume&quot;) %>%
    xml_text()
}
  
# `bench::mark` checks if results are consistent, which is usually a good thing 
# to do
bench::mark(fun1(), fun2(), fun3(), min_time = 2)
</code></pre>
<pre><code># A tibble: 3 x 14
  expression   min  mean median    max `itr/sec` mem_alloc  n_gc n_itr total_time
  </code></pre>