The problem
Text-mining - the art of answering questions by extracting patterns, data, etc. out of the published literature - is not easy.
It’s made incredibly difficult because of publishers. It is a fact that the vast majority of publicly funded research across the globe is published in paywall journals. That is, taxpayers pay twice for research: once for the grant to fund the work, then again to be able to read it. These paywalls mean that every potential person text-mining will have different access: some have access through their university, some may have access through their company, and others may only have access to whatever happens to be open access. On top of that, access for paywall journals often depends on your IP address - something not generally on top of mind for most people.
Another hardship with text-mining is the huge number of publishers together with no standardized way to figure out the URL for full text versions of a scholarly work. There is the DOI (Digital Object Identifier) system used by Crossref, Datacite and others, but those generally help you sort out the location of the scholarly work on a web page - the html version. What one probably wants for text-mining is the PDF or XML version if available. Publishers can optionally choose to include URLs for full text (PDF and/or XML) with Crossref’s metadata (e.g., see this Crossref API call and search for “link” on the page), but the problem is that it’s optional.
fulltext is a package to help R users address the above problems, and get published literature from the web in it’s many forms, and across all publishers.
the fulltext package
fulltext tries to make the following use cases as easy as possible:
Search for articles
Fetch abstracts
Fetch full text articles
Get links for full text articles (xml, pdf)
Extract text from articles
Collect sections of articles that you actually need (e.g., titles)
Download supplementary materials
fulltext organizes functions around the above use cases, then provides flexiblity to query many data sources within that use case (i.e. function). For example fulltext::ft_search searches for articles - you can choose among one or more of many data sources to search, passing options to each source as needed.
What does a workflow with fulltext look like?
Search for articles with ft_search()
Fetch articles with ft_get() using the output of the previous step
Collect the text into an object with ft_collect()
Extract sections of articles needed with ft_chunks(), or
Combine texts into a data.frame ready for quanteda or similar text-mining packages
Package overhaul
fulltext has undergone a re-organization, which includes a bump in the major version to v1 to reinforce the large changes the package has undergone. Changes include:
Function name standardization with the ft_ prefix. e,g, chunks is now ft_chunks
ft_get has undergone major re-organization - biggest of which may be that all full text XML/plain text/PDF goes to disk to simplify the user interface. Along with this we’ve changed to using DOIs/IDs as the file names
We no longer store files as rds - but as the format they are, pdf, txt or xml
storr is now imported to manage mapping between real DOIs and file paths that include normalized DOIs - and aids in the function ft_table() for creating a data.frame of text results
Note that with the ft_get() overhaul, the only option is to write to disk. Before we attempted to provide many different options for saving XML and PDF data, but it was too complicated. This has implications for using the output of ft_get() - the output includes the paths to the files - use ft_collect() to collect the text if you want to use ft_chunks() or other fulltext functions downstream.
A number of functions have been removed to further hone the scope of the package
A function ft_abstract is introduced to fetch abstracts for when you just need abstracts
A function ft_table has been introduced to gather all your documents into a data.frame to make it easy to do downstream analyses with other packages
Two new data sources have been added: Scopus and Microsoft Academic - both of which are available via ft_search() and ft_abstract()
New functions have been added for the user to find out what plugins are available: ft_get_ls(), ft_links_ls(), and ft_search_ls()
We’ve battle tested ft_get() on a lot of DOIs - but there still may be errors - let us know if you have any problems.
Documentation
Along with an overhual of the package we have made a new manual for fulltext. Check it out at https://ropensci.github.io/fulltext-book/
Setup
Install fulltext
at the time of writing binaries are not yet available on CRAN, so you’ll have to install from source from CRAN (which shouldn’t provide any problems since there’s no compiled code in the package), or install from GitHub
install.packages(&quot;fulltext&quot;)
Or get the development version:
devtools::install_github(&quot;ropensci/fulltext&quot;)
library(fulltext)
Below I’ll discuss some of the new features of the package, and not do an exhaustive tutorial to the package. Check out the manual for more details: https://ropensci.github.io/fulltext-book/
Fetch abstracts: ft_abstract
ft_abstract() is a new function in fulltext. It gives you access to absracts from the following data sources:
crossref
microsoft
plos
scopus
A quick example. Search for articles in PLOS.
res % ft_tabularize()
Extract text: ft_extract
ft_extract() used to have options for extracting text from PDF’s with different pieces of software. To simplify the function it only uses the pdftools package.
path