<p>Textual data and natural language processing are still a niche domain within the R ecosytstem. The <a href="https://cran.r-project.org/view=NaturalLanguageProcessing">NLP task view</a> gives an overview of existing work however a lot of basic infrastructure is still missing.
At the rOpenSci <a href="https://ropensci.org/blog/blog/2017/05/03/textworkshop17">text workshop</a> in April we discussed many ideas for improving text processing in R which revealed several core areas that need improvement:</p>
<ul>
<li>Reading: better tools for extracing text and metadata from documents in various formats (doc, rtf, pdf, etc).</li>
<li>Encoding: many text packages work well for ascii text but rapidly break down when text contains Hungarian, Korean or emojis.</li>
<li>Interchange: packages don't work well together due to lack of data classes or conventions for textual data (see also <a href="https://github.com/ropensci/tif">ropensci/tif</a>)</li>
</ul>
<p>Participants also had many good suggestions for C/C++ libraries that text researchers in R might benefit from. Over the past weeks I was able to look into these suggestions and work on a few packages for reading and analyzing text. Below is an update on new and improved rOpenSci tools for text processsing in R!</p>
<h2>Google language detector 2 and 3</h2>
<p>New packages <code>cld2</code> and <code>cld3</code> are wrappers C++ libraries by Google for language identification. <a href="https://github.com/cld2owners/cld2#internals">CLD2</a> is a Naïve Bayesian classifier, whereas <a href="https://github.com/google/cld3#model">CLD3</a> uses a neural network model. I found <code>cld2</code> to give better results for short text.</p>
<div><pre><code><span></span><span># Get the latest versions</span>
install.packages<span>(</span><span>c</span><span>(</span><span>&quot;cld2&quot;</span><span>,</span> <span>&quot;cld3&quot;</span><span>))</span>

<span># Vectorized function</span>
text <span>&lt;-</span> <span>c</span><span>(</span><span>&quot;À chaque fou plaît sa marotte.&quot;</span><span>,</span> <span>&quot;猿も木から落ちる&quot;</span><span>,</span>
    <span>&quot;Алты́нного во́ра ве́шают, а полти́нного че́ствуют.&quot;</span><span>,</span> <span>&quot;Nou breekt mijn klomp!&quot;</span><span>)</span>

cld2<span>::</span>detect_language<span>(</span>text<span>)</span>
<span># [1] &quot;fr&quot; &quot;ja&quot; &quot;ru&quot; &quot;nl&quot;</span>
</code></pre></div>
<p><a href="https://maelle.github.io/">Maëlle</a> has written a <a href="http://www.masalmon.eu/2017/06/10/rolandgarros/">cool post</a> comparing language classification methods using 18000 <code>&quot;#RolandGarros2017&quot;</code> tweets and <a href="https://www.stat.auckland.ac.nz/people/tlum005">Thomas</a> <a href="http://notstatschat.tumblr.com/post/161449071226/stupid-word-games">reminds us</a> that algorithms can easily be fooled. Still I found the accuracy on real text quite astonishing given the relatively small size of these libraries.</p>
<p>Note that the algorithm for CLD3 is still under development and the engineers at Google have recently <a href="https://github.com/google/cld3/issues">opened</a> their Github issues page for feedback.</p>
<h2>(anti) word and (un)rtf</h2>
<p>Many archived documents are only available in legacy formats such as <code>.doc</code> and <code>.rtf</code>. The only tools available for extracting text from these documents were difficult to install and could not be imported from packages and scripts.</p>
<p>To make this a little easier we have packaged up utilities <a href="http://www.winfield.demon.nl/">antiword</a> and <a href="https://www.gnu.org/software/unrtf/">UnRTF</a> to read MS <code>doc</code> and <code>rtf</code> files respectively.</p>
<div><pre><code><span></span><span># Get the latest versions</span>
install.packages<span>(</span><span>c</span><span>(</span><span>&quot;antiword&quot;</span><span>,</span> <span>&quot;unrtf&quot;</span><span>))</span>

<span># Extract text from 'rtf' file</span>
text <span>&lt;-</span> unrtf<span>::</span>unrtf<span>(</span><span>&quot;https://jeroen.github.io/files/sample.rtf&quot;</span><span>,</span> format <span>=</span> <span>&quot;text&quot;</span><span>)</span>
<span>cat</span><span>(</span>text<span>)</span>
<span>### Lots of text...</span>

<span># Extract text from 'doc' file</span>
text <span>&lt;-</span> antiword<span>::</span>antiword<span>(</span><span>&quot;https://jeroen.github.io/files/UDHR-english.doc&quot;</span><span>)</span>
<span>cat</span><span>(</span>text<span>)</span>
<span>### Lots of text...</span>
</code></pre></div>
<p>Also have a look at meta packages <code>readtext</code> or <code>textreadr</code> which wrap these and other packages for automatically reading text in many different formats.</p>
<h2>pdf utilities</h2>
<p>Our <a href="https://cran.r-project.org/web/packages/pdftools/index.html">pdftools</a> package now supports reading pdf (extracting text or metadata) and rendering pdf to png, jpeg, tiff, or raw vectors on all platforms (incl. Windows).</p>
<div><pre><code><span></span><span># Read some text</span>
text <span>&lt;-</span> pdftools<span>::</span>pdf_text<span>(</span><span>'https://cran.r-project.org/doc/manuals/r-release/R-intro.pdf'</span><span>)</span>
<span>cat</span><span>(</span>text<span>[</span><span>1</span><span>])</span>
<span># An Introduction to R</span>
<span># Notes on R: A Programming Environment for Data Analysis and Graphics</span>
<span># Version 3.4.0 (2017-04-21)</span>
<span># W. N. Venables, D. M. Smith</span>
<span># and the R Core Team</span>

<span># Read meta data</span>
pdftools<span>::</span>pdf_info<span>(</span><span>'https://cran.r-project.org/doc/manuals/r-release/R-intro.pdf'</span><span>)</span>
<span># $version</span>
<span># [1] &quot;1.5&quot;</span>
<span>#</span>
<span># $pages</span>
<span># [1] 105</span>
<span>#</span>
<span># .... much more :)</span>
</code></pre></div>
<p>You can use either render a page into a raw bitmap array, or directly to an image format such as png, jpeg or tiff.</p>
<div><pre><code><span></span>files <span>&lt;-</span> pdftools<span>::</span>pdf_convert<span>(</span><span>'https://cran.r-project.org/doc/manuals/r-release/R-intro.pdf'</span><span>,</span> format <span>=</span> <span>&quot;png&quot;</span> pages <span>=</span> <span>1</span><span>:</span><span>5</span><span>)</span>
<span># Converting page 1 to R-intro_1.png... done!</span>
<span># Converting page 2 to R-intro_2.png... done!</span>
<span># Converting page 3 to R-intro_3.png... done!</span>
<span># Converting page 4 to R-intro_4.png... done!</span>
<span># Converting page 5 to R-intro_5.png... done!</span>
</code></pre></div>
<p>To extract text from scanned images, also check out our <a href="https://cran.r-project.org/web/packages/tesseract/index.html">tesseract</a> package which wraps Google's powerful OCR engine.</p>
<h2>Stemming, tokenizing and spell checking</h2>
<p>Our <a href="https://cran.r-project.org/web/packages/hunspell/index.html">hunspell</a> package has had a few updates recently as well. The package is a wrapper for <a href="http://hunspell.github.io/">libhunspell</a> which is a popular library for spell checking:</p>
<div><pre><code><span></span><span># Extract incorrect from a piece of text</span>
bad <span>&lt;-</span> hunspell<span>(</span><span>&quot;spell checkers are not neccessairy for langauge ninja's&quot;</span><span>)</span>
<span>print</span><span>(</span>bad<span>[[</span><span>1</span><span>]])</span>
<span># [1] &quot;neccessairy&quot; &quot;langauge&quot;</span>
hunspell_suggest<span>(</span>bad<span>[[</span><span>1</span><span>]])</span>
<span># [[1]]</span>
<span># [1] &quot;necessary&quot; &quot;necessarily&quot; &quot;necessaries&quot; &quot;recessionary&quot; &quot;accessory&quot; &quot;incarcerate&quot;</span>
<span>#</span>
<span># [[2]]</span>
<span># [1] &quot;language&quot; &quot;Langeland&quot; &quot;Lagrange&quot; &quot;Lange&quot; &quot;gaugeable&quot; &quot;linkage&quot; &quot;Langland&quot;</span>
</code></pre></div>
<p>The package is also used by <code>devtools</code> to spell-check manual pages in R packages:</p>
<div><pre><code><span></span>devtools<span>::</span>spell_check<span>()</span>
<span># WORD FOUND IN</span>
<span># alltogether pdftools.Rd:36</span>
<span># cairo pdf_render_page.Rd:42</span>
<span># jpeg pdf_render_page.Rd:40</span>
<span># libpoppler pdf_render_page.Rd:42, pdftools.Rd:30, description:1</span>
<span># png pdf_render_page.Rd:40</span>
<span># Poppler pdftools.Rd:34</span>
</code></pre></div>
<p>Finally hunspell also exposes the underlying methods needed for spell checking such as stemming words:</p>
<div><pre><code><span></span><span># Find possible stems for each word</span>
words <span>&lt;-</span> <span>c</span><span>(</span><span>&quot;loving&quot;</span><span>,</span> <span>&quot;loved&quot;</span><span>,</span> <span>&quot;lover&quot;</span><span>,</span> <span>&quot;lovely&quot;</span><span>,</span> <span>&quot;love&quot;</span><span>)</span>
hunspell_analyze<span>(</span>words<span>)</span>
<span># [[1]]</span>
<span># [1] &quot; st:loving&quot; &quot; st:love fl:G&quot;</span>
<span>#</span>
<span># [[2]]</span>
<span># [1] &quot; st:loved&quot; &quot; st:love fl:D&quot;</span>
<span>#</span>
<span># [[3]]</span>
<span># [1] &quot; st:lover&quot; &quot; st:love fl:R&quot;</span>
<span>#</span>
<span># [[4]]</span>
<span># [1] &quot; st:lovely&quot; &quot; st:love fl:Y&quot;</span>
<span>#</span>
<span># [[5]]</span>
<span># [1] &quot; st:love&quot;</span>
</code></pre></div>
<p>Hunspell also suppors tokenizing words from html, latex, man, or plain text. For more advanced word extraction, check out the rOpenSci <a href="https://github.com/ropensci/tokenizers#readme">tokenizers</a> package.</p>