<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/fulltext-v1-text-mining-scholarly-works/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://ropensci.org/technotes/2018/01/17/fulltext-v1/"> rOpenSci - open tools for open science</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<h2>The problem</h2>
<p>Text-mining – the art of answering questions by extracting patterns, data, etc. out of the published literature – is not easy.</p>
<p>It’s made incredibly difficult because of publishers. It is a fact that the vast majority of publicly funded research across the globe is published in paywall journals. That is, taxpayers pay twice for research: once for the grant to fund the work, then again to be able to read it. These paywalls mean that every potential person text-mining will have different access: some have access through their university, some may have access through their company, and others may only have access to whatever happens to be open access. On top of that, access for paywall journals often depends on your <a rel="nofollow" href="https://en.wikipedia.org/wiki/IP_address" target="_blank">IP address</a> – something not generally on top of mind for most people.</p>
<p>Another hardship with text-mining is the huge number of publishers together with no standardized way to figure out the URL for full text versions of a scholarly work. There is the DOI (Digital Object Identifier) system used by Crossref, Datacite and others, but those generally help you sort out the location of the scholarly work on a web page – the html version. What one probably wants for text-mining is the PDF or XML version if available. Publishers can optionally choose to include URLs for full text (PDF and/or XML) with Crossref’s metadata (e.g., see this <a rel="nofollow" href="http://api.crossref.org/works?filter=has-full-text:true&rows=3" target="_blank">Crossref API call</a> and search for “link” on the page), but the problem is that it’s <strong>optional</strong>.</p>
<p><strong><code>fulltext</code> is a package to help R users address the above problems, and get published literature from the web in it’s many forms, and across all publishers.</strong></p>

<h2>the fulltext package</h2>
<p><code>fulltext</code> tries to make the following use cases as easy as possible:</p>
<ul>
<li>Search for articles</li>
<li>Fetch abstracts</li>
<li>Fetch full text articles</li>
<li>Get links for full text articles (xml, pdf)</li>
<li>Extract text from articles</li>
<li>Collect sections of articles that you actually need (e.g., titles)</li>
<li>Download supplementary materials</li>
</ul>
<p><code>fulltext</code> organizes functions around the above use cases, then provides flexiblity to query many data sources within that use case (i.e. function). For example <code>fulltext::ft_search</code> searches for articles – you can choose among one or more of many data sources to search, passing options to each source as needed.</p>
<p></p>
<h2>What does a workflow with fulltext look like?</h2>
<ul>
<li>Search for articles with <code>ft_search()</code></li>
<li>Fetch articles with <code>ft_get()</code> using the output of the previous step</li>
<li>Collect the text into an object with <code>ft_collect()</code></li>
<li>Extract sections of articles needed with <code>ft_chunks()</code>, or</li>
<li>Combine texts into a data.frame ready for <code>quanteda</code> or similar text-mining packages</li>
</ul>
<p></p>
<h2>Package overhaul</h2>
<p><code>fulltext</code> has undergone a re-organization, which includes a bump in the major version to <code>v1</code> to reinforce the large changes the package has undergone. Changes include:</p>
<ul>
<li>Function name standardization with the <code>ft_</code> prefix. e,g, <code>chunks</code> is now <code>ft_chunks</code></li>
<li><code>ft_get</code> has undergone major re-organization – biggest of which may be that all full text XML/plain text/PDF goes to disk to simplify the user interface. Along with this we’ve changed to using DOIs/IDs as the file names</li>
<li>We no longer store files as rds – but as the format they are, pdf, txt or xml</li>
<li><code>storr</code> is now imported to manage mapping between real DOIs and file paths that include normalized DOIs – and aids in the function <code>ft_table()</code> for creating a data.frame of text results</li>
<li>Note that with the <code>ft_get()</code> overhaul, the only option is to write to disk. Before we attempted to provide many different options for saving XML and PDF data, but it was too complicated. This has implications for using the output of <code>ft_get()</code> – the output includes the paths to the files – use <code>ft_collect()</code> to collect the text if you want to use <code>ft_chunks()</code> or other <code>fulltext</code> functions downstream.</li>
<li>A number of functions have been removed to further hone the scope of the package</li>
<li>A function <code>ft_abstract</code> is introduced to fetch abstracts for when you just need abstracts</li>
<li>A function <code>ft_table</code> has been introduced to gather all your documents into a data.frame to make it easy to do downstream analyses with other packages</li>
<li>Two new data sources have been added: Scopus and Microsoft Academic – both of which are available via <code>ft_search()</code> and <code>ft_abstract()</code></li>
<li>New functions have been added for the user to find out what plugins are available: <code>ft_get_ls()</code>, <code>ft_links_ls()</code>, and <code>ft_search_ls()</code></li>
</ul>
<p>We’ve battle tested <code>ft_get()</code> on a lot of DOIs – but there still may be errors – let us know if you have any problems.</p>
<p></p>
<h2>Documentation</h2>
<p>Along with an overhual of the package we have made a new manual for <code>fulltext</code>. Check it out at <a rel="nofollow" href="https://ropensci.github.io/fulltext-book/" target="_blank">https://ropensci.github.io/fulltext-book/</a></p>

<h2>Setup</h2>
<p>Install <code>fulltext</code></p>
<blockquote>
<p>at the time of writing binaries are not yet available on CRAN, so you’ll have to install from source from CRAN (which shouldn’t provide any problems since there’s no compiled code in the package), or install from GitHub</p>
</blockquote>
<pre><code>install.packages(&quot;fulltext&quot;)
</code></pre>
<p>Or get the development version:</p>
<pre><code>devtools::install_github(&quot;ropensci/fulltext&quot;)
</code></pre>
<pre><code>library(fulltext)
</code></pre>
<p>Below I’ll discuss some of the new features of the package, and not do an exhaustive tutorial to the package. Check out the manual for more details: <a rel="nofollow" href="https://ropensci.github.io/fulltext-book/" target="_blank">https://ropensci.github.io/fulltext-book/</a></p>
<p></p>
<h2>Fetch abstracts: ft_abstract</h2>
<p><code>ft_abstract()</code> is a new function in <code>fulltext</code>. It gives you access to absracts from the following data sources:</p>
<ul>
<li>crossref</li>
<li>microsoft</li>
<li>plos</li>
<li>scopus</li>
</ul>
<p>A quick example. Search for articles in PLOS.</p>
<pre><code>res &lt;- ft_search(query = 'biology', from = 'plos', limit = 10, 
   plosopts = list(fq = list('doc_type:full', '-article_type:correction',
                  '-article_type:viewpoints')))
</code></pre>
<p>Now pass the DOIs to <code>ft_abstract()</code> to get abstracts:</p>
<pre><code>ft_abstract(x = res$plos$data$id, from = &quot;plos&quot;)
</code></pre>
<pre><code>## </code></pre>