<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/lessons-learned-from-rtika-a-digital-babel-fish/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://ropensci.org/blog/2018/04/25/rtika-introduction/"> rOpenSci - open tools for open science</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>The Apache Tika parser is like the Babel fish in Douglas Adam’s book, “The Hitchhikers’ Guide to the Galaxy” <sup><a rel="nofollow" href="https://ropensci.org/blog/2018/04/25/rtika-introduction/#fn:1" target="_blank">1</a></sup>. The Babel fish translates any natural language to any other. Although Tika does not yet translate natural language, it starts to tame the tower of babel of digital document formats. As the Babel fish allowed a person to understand Vogon poetry, Tika allows an analyst to extract text and objects from Microsoft Word.</p>
<pre><code>                        .----.      
               ____    __\\\\\\__                 
               \___'--&quot;          .-.          
               /___>------rtika  '0'          
              /____,--.        \----B      
                      &quot;&quot;.____  ___-&quot;    
                      //    / /                   
                            ]/               
</code></pre>
<p>Parsing files is a common concern for many communities, including journalism <sup><a rel="nofollow" href="https://ropensci.org/blog/2018/04/25/rtika-introduction/#fn:2" target="_blank">2</a></sup>, government, business, and academia. The complexity of parsing can vary a lot. Apache Tika is a common library to lower that complexity. The Tika auto-detect parser finds the content type of a file and processes it with an appropriate parser. It currently handles text or metadata extraction from <strong>over one thousand digital formats</strong>:</p>
<ul>
<li>Portable Document Format (<code>.pdf</code>)</li>
<li>Microsoft Office document formats (Word, PowerPoint, Excel, etc.)</li>
<li>Rich Text Format (<code>.rtf</code>)</li>
<li>Electronic Publication Format (<code>.epub</code>)</li>
<li>Image formats (<code>.jpeg</code>, <code>.png</code>, etc.)</li>
<li>Mail formats (<code>.mbox</code>, Outlook)</li>
<li>HyperText Markup Language (<code>.html</code>)</li>
<li>XML and derived formats (<code>.xml</code>, etc.)</li>
<li>Compression and packaging formats (<code>.gzip</code>, <code>.rar</code>, etc.)</li>
<li>OpenDocument Format</li>
<li>iWorks document formats</li>
<li>WordPerfect document formats</li>
<li>Text formats</li>
<li>Feed and Syndication formats</li>
<li>Help formats</li>
<li>Audio formats</li>
<li>Video formats</li>
<li>Java class files and archives</li>
<li>Source code</li>
<li>CAD formats</li>
<li>Font formats</li>
<li>Scientific formats</li>
<li>Executable programs and libraries</li>
<li>Crypto formats</li>
</ul>
<p>I am blown away by the thousands of hours spent on the included parsers, such as Apache <code>PDFBox</code>, Apache <code>Poi</code>, and others <sup><a rel="nofollow" href="https://ropensci.org/blog/2018/04/25/rtika-introduction/#fn:3" target="_blank">3</a></sup>. Tika began as a common back-end for search engines and to reduce duplicated time and effort. Automatically producing information from semi-structured documents is a deceptively complex process that involves tacit knowledge of how document formats have changed over time, the gray areas of their specifications, and dealing with inconsistencies in metadata. Tika began as part of Apache Nutch in 2005, and then became its own project in 2007 and a shared module in Lucene, Jackrabbit, Mahout, and Solr <sup><a rel="nofollow" href="https://ropensci.org/blog/2018/04/25/rtika-introduction/#fn:1" target="_blank">1</a></sup>. Now, Tika is the back-end of the <code>rtika</code> package.</p>
<h4>Motivation: the Dreaded <code>.lob</code> File Extension</h4>
<p>This package came together when parsing Word documents in a governmental archive. The files did not have a helpful file extension. They had been stored as ‘large object data’ in a database, and given the generic <code>.lob</code> extension. Some documents parsed with the <code>antiword</code> package:</p>
<pre><code>library('antiword')

timing &lt;- system.time(
  
  text &lt;- 
    batch[1:2000] %>%
    lapply(antiword)
  
)

# average time elapsed *per document* for antiword:
timing[3]/2000
#>   elapsed 
#> 0.0098275
</code></pre>
<p>However, the files farther into the batch were in the new Word format, and <code>antiword</code> did not parse them. The government had switched to the new version but left no obvious clues. Typically, modern Word documents use the <code>.docx</code> file extension and the ancient ones use <code>.doc</code>. I just had <code>.lob</code>:</p>
<pre><code>last_file &lt;- length(batch)
tryCatch(antiword(batch[last_file]), error = function(x){x})
#> </code></pre>