<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/pdftools-2-0-powerful-pdf-text-extraction-tools/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://ropensci.org/technotes/2018/12/14/pdftools-20/"> rOpenSci - open tools for open science</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p> <img alt="cover image" src="https://i0.wp.com/cdn.flipsnack.com/blog/wp-content/uploads/2017/10/09164009/page-turning-pdf.jpg?w=456&ssl=1" data-recalc-dims="1"></p>
<p>A new version of pdftools has been released to CRAN. Go get it while it’s hot:</p>
<pre><code>install.packages(&quot;pdftools&quot;)
</code></pre>
<p>This version has two major improvements: low level text extraction and encoding improvements.</p>
<h2>About PDF textboxes</h2>
<p>A pdf document may seem to contain paragraphs or tables in a viewer, but this is not actually true. PDF is a <em>printing format</em>: a page consists of a series of unrelated lines, bitmaps, and textboxes with a given size, position and content. Hence a table in a pdf file is really just a large unordered set of lines and words that are nicely visually positioned. This makes sense for printing, but makes extracting text or data from a pdf file extremely difficult.</p>
<p>Because the pdf format has little semantic structure, the <code>pdf_text()</code> function in pdftools has to render the PDF to a text canvas, in order to create the sentences or paragraphs. It does so pretty well, but some users have asked for something more low level.</p>
<blockquote class="twitter-tweet">
<p lang="en" dir="ltr"><a rel="nofollow" href="https://twitter.com/opencpu?ref_src=twsrc%5Etfw" target="_blank">@opencpu</a> any chance pdftext could return a data frame with one row for each text box with columns giving page, positions, and contents?</p>
<p>— Hadley Wickham (@hadleywickham) <a rel="nofollow" href="https://twitter.com/hadleywickham/status/945729495956369409?ref_src=twsrc%5Etfw" target="_blank">December 26, 2017</a></p></blockquote>
<p></p>
<p>Unfortunately this was not trivial because it required some work in the underlying <a rel="nofollow" href="https://poppler.freedesktop.org/" target="_blank">poppler</a> library. One year later, and this functionality is now finally available in the upcoming poppler version 0.73. The pdftools <a rel="nofollow" href="https://cran.r-project.org/package=pdftools" target="_blank">CRAN binary packages</a> for Windows and MacOS already contain a suitable libpoppler, however Linux users probably have to wait for the latest version of poppler to become available in their system package manager (or compile from source).</p>
<h2>Low-level text extraction</h2>
<p>We use an <a rel="nofollow" href="https://github.com/ropensci/tabulizer/raw/master/inst/examples/data.pdf" target="_blank">example pdf file</a> from the rOpenSci tabulizer package. This file contains a few standard datasets which have been printed as a pdf table. First let’s try the <code>pdf_text()</code> function, which returns a character vector of length equal to the number of pages in the file.</p>
<pre><code>library(pdftools)
pdf_file &lt;- &quot;https://github.com/ropensci/tabulizer/raw/master/inst/examples/data.pdf&quot;
txt &lt;- pdf_text(pdf_file)
cat(txt[1])
</code></pre>
<pre><code>                    mpg  cyl    disp  hp drat    wt  qsec vs am gear carb
Mazda RX4           21.0   6   160.0 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag       21.0   6   160.0 110 3.90 2.875 17.02  0  1    4    4
Datsun 710          22.8   4   108.0  93 3.85 2.320 18.61  1  1    4    1
Hornet 4 Drive      21.4   6   258.0 110 3.08 3.215 19.44  1  0    3    1
Hornet Sportabout   18.7   8   360.0 175 3.15 3.440 17.02  0  0    3    2
Valiant             18.1   6   225.0 105 2.76 3.460 20.22  1  0    3    1
Duster 360          14.3   8   360.0 245 3.21 3.570 15.84  0  0    3    4
Merc 240D           24.4   4   146.7  62 3.69 3.190 20.00  1  0    4    2
...
</code></pre>
<p>Hence <code>pdf_text()</code> converts all text on a page to a large string, which works pretty well. However if you would want to parse this text into a data frame (using e.g. <code>read.table</code>) you run into a problem: the first column contains spaces within the values. Therefore we can’t use the whitespace as the column delimiter (as is the default in <code>read.table</code>).</p>
<p>Hence to write a proper pdf table extractor, we have to infer the column from the <em>physical position</em> of the textbox, rather than rely on delimiting characters. The new <code>pdf_data()</code> provides exactly this. It returns a data frame with all textboxes in a page, including their width, height, and (x,y) position:</p>
<pre><code># All textboxes on page 1
pdf_data(pdf_file)[[1]]
</code></pre>
<pre><code># A tibble: 430 x 6
   width height     x     y space text  
   </code></pre>