<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/taming-exam-results-in-pdf-with-pdftools/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://talesofr.wordpress.com/2017/11/03/taming-exam-results-in-pdf-with-pdftools/"> R – Tales of R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<div><img data-orig-file="https://talesofr.files.wordpress.com/2017/11/edaic.png?w=456" data-orig-size="425,300" data-medium-file="https://talesofr.files.wordpress.com/2017/11/edaic.png?w=456?w=300" data-attachment-id="914" data-image-meta="{" data-image-description="" src="https://talesofr.files.wordpress.com/2017/11/edaic.png?w=456" data-permalink="https://talesofr.wordpress.com/2017/11/03/taming-exam-results-in-pdf-with-pdftools/edaic/" data-large-file="https://talesofr.files.wordpress.com/2017/11/edaic.png?w=456?w=425" alt="edaic" data-image-title="edaic" sizes="(max-width: 425px) 100vw, 425px" data-comments-opened="1" class="wp-image-914" data-recalc-dims="1"></div>
</div>
<div>
<div>You can also check this post, written in <a rel="nofollow" href="https://twitter.com/hashtag/blogdown" target="_blank">#blogdown</a>, here: <a rel="nofollow" href="https://aurora-mareviv.github.io/2017/11/taming-exam-results-with-pdftools/" target="_blank">taming-exam-results-with-pdf</a>. 
<div>
<h1>Introduction</h1>
<p>There are several ways to mine tables and other content from a pdf, using R. After a lot of trial &amp; error, here’s how I managed to extract global exam results from an international, massive, yearly examination, the <a rel="nofollow" href="https://www.esahq.org/education/edaic/exam-types/part-i-examination" target="_blank">EDAIC</a>.</p>
<p>This is my first use case of <em>“pdf mining”</em> with R, and also a fairly simple one. However, more complex and very fine examples of this can be found elsewhere, using both <a rel="nofollow" href="https://ropensci.org/blog/2016/03/01/pdftools-and-jeroen/" target="_blank"><span>pdftools</span></a> and <a rel="nofollow" href="https://github.com/ropensci/tabulizer" target="_blank"><span>tabulizer</span></a> packages.</p>
<p>As can be seen from the original <a rel="nofollow" href="http://www.esahq.org/~/media/ESA/Files/EDUCATION/Part%20I/Part%20I%202017%20Results/EDAIC%20Part%20I%202017%20-%20Anonymous%20Results.ashx" target="_blank">pdf</a>, exam results are anonymous. They consist on a numeric, 6-digit <strong>code</strong> and a binary result: “<strong>FAIL / PASS</strong>”. I was particularly interested into seeing how many of them passed the exam, as some indirect measure of how <em>“hard”</em> it can be.</p>
</div>
<div>
<h1>Mining the table</h1>
<p>In this case I preferred <span>pdftools</span> as it allowed me to extract the whole content from the pdf:</p>
<pre><code><span>install</span>.packages(<span>&quot;pdftools&quot;</span>)</code></pre>
<pre><code><span>library</span>(<span>pdftools</span>) 
<span>txt</span> <span>-</span> <span>pdf_text</span>(&quot;<span>EDAIC</span><span>.pdf</span>&quot;) 
<span>txt</span><span>[1]</span> 
<span>class</span>(<span>txt</span><span>[1]</span>) </code></pre>
<pre><code>  [<span>1</span>] <span>&quot;EDAIC Part I 2017 Overall Results\n Candidate N° Result\n 107131 FAIL\n 119233 PASS\n 123744 FAIL\n 127988 FAIL\n 133842 PASS\n 135692 PASS\n 140341 FAIL\n 142595 FAIL\n 151479 PASS\n 151632 PASS\n 152787 PASS\n 157691 PASS\n 158867 PASS\n 160211 PASS\n 161970 FAIL\n 162536 PASS\n 163331 PASS\n 164442 FAIL\n 164835 PASS\n 165734 PASS\n 165900 PASS\n 166469 PASS\n 167241 FAIL\n 167740 PASS\n 168151 FAIL\n 168331 PASS\n 168371 FAIL\n 168711 FAIL\n 169786 PASS\n 170721 FAIL\n 170734 FAIL\n 170754 PASS\n 170980 PASS\n 171894 PASS\n 171911 PASS\n 172047 FAIL\n 172128 PASS\n 172255 FAIL\n 172310 PASS\n 172706 PASS\n 173136 FAIL\n 173229 FAIL\n 174336 PASS\n 174360 PASS\n 175177 FAIL\n 175180 FAIL\n 175184 FAIL\nYour candidate number is indicated on your admission document Page 1 of 52\n&quot;</span></code></pre>
<pre><code>  [<span>1</span>] <span>&quot;character&quot;</span></code></pre>
<p>These commands return a lenghty <em>blob</em> of text. Fortunately, there are some <code>\n</code> symbols that signal the new lines in the original document.</p>
<p>We will use these to split the <em>blob</em> into something more approachable, using <a rel="nofollow" href="https://www.tidyverse.org/" target="_blank"><code>tidyversal</code></a> methods…</p>
<ul>
<li>Split the <em>blob</em>.</li>
<li>Transform the resulting <code>list</code> into a <code>character vector</code> with <code>unlist</code>.</li>
<li>Trim leading white spaces with <code>stringr::str_trim</code>.</li>
</ul>
<pre><code>library(tidyverse) 
library(stringr) 
tx2 &lt;- strsplit(txt, <span>&quot;\n&quot;</span>) %>% <span># divide by carriage returns</span>
  unlist() %>% 
  str_trim(side = <span>&quot;both&quot;</span>) <span># trim white spaces</span>
tx2[1:10]</code></pre>
<pre><code>   [<span>1</span>] <span>&quot;EDAIC Part I 2017 Overall Results&quot;</span>
   [<span>2</span>] <span>&quot;Candidate N° Result&quot;</span>                                                         
   [<span>3</span>] <span>&quot;107131 FAIL&quot;</span>                                                            
   [<span>4</span>] <span>&quot;119233 PASS&quot;</span>                                                            
   [<span>5</span>] <span>&quot;123744 FAIL&quot;</span>                                                            
   [<span>6</span>] <span>&quot;127988 FAIL&quot;</span>                                                            
   [<span>7</span>] <span>&quot;133842 PASS&quot;</span>                                                            
   [<span>8</span>] <span>&quot;135692 PASS&quot;</span>                                                            
   [<span>9</span>] <span>&quot;140341 FAIL&quot;</span>                                                            
  [<span>10</span>] <span>&quot;142595 FAIL&quot;</span></code></pre>
<ul>
<li>Remove the very first row.</li>
<li>Transform into a <code>tibble</code>.</li>
</ul>
<pre><code><span>tx3</span> <span>-</span> <span>tx2</span><span>[-1]</span> %>% 
  <span>data_frame</span>() 
<span>tx3</span></code></pre>
<pre><code>  <span># A tibble: 2,579 x 1</span>
                             .
                         <span>chr</span>>
   <span>1</span> Candidate N°       Result
   <span>2</span>    <span>107131</span>            FAIL
   <span>3</span>    <span>119233</span>            PASS
   <span>4</span>    <span>123744</span>            FAIL
   <span>5</span>    <span>127988</span>            FAIL
   <span>6</span>    <span>133842</span>            PASS
   <span>7</span>    <span>135692</span>            PASS
   <span>8</span>    <span>140341</span>            FAIL
   <span>9</span>    <span>142595</span>            FAIL
  <span>10</span>    <span>151479</span>            PASS
  <span># ... with 2,569 more rows</span></code></pre>
<ul>
<li>Use <code>tidyr::separate</code> to split each row into two columns.</li>
<li>Remove all spaces.</li>
</ul>
<pre><code>tx4 &lt;- separate(tx3, ., c(<span>&quot;key&quot;</span>, <span>&quot;value&quot;</span>), <span>&quot; &quot;</span>, extra = <span>&quot;merge&quot;</span>) %>%  
  mutate(key = gsub(<span>'\\s+'</span>, <span>''</span>, key)) %>%
  mutate(<span>value</span> = gsub(<span>'\\s+'</span>, <span>''</span>, <span>value</span>)) 
tx4</code></pre>
<pre><code>  <span># A tibble: 2,579 x 2</span>
           <span>key</span>    value
         <span></span></code></pre></div></div></div>