<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/whats-this-bird-classify-old-natural-history-drawings-with-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://ropensci.org/blog/2018/08/28/birds-ocr/"> rOpenSci - open tools for open science</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>In this new post, we're taking a break from modern birding data in <a rel="nofollow" href="https://ropensci.org/tags/birder" target="_blank">our<br>
birder's series</a>â€¦ let's explore<br>
gorgeous drawings from a natural history collection! Armed with<br>
rOpenSci's packages binding powerful C++ libraries and open taxonomy<br>
data, how much information can we automatically extract from images?<br>
Maybe not much, but we'll at least have explored image manipulation,<br>
optical character recognition (OCR), language detection, taxonomic name<br>
resolution with rOpenSci's packages.</p>
<h3>Free natural history images and appropriate R tooling!</h3>
<p>A long time ago I had bookmarked the <a rel="nofollow" href="https://www.flickr.com/people/biodivlibrary/" target="_blank">Flickr account of the Biodiversity<br>
Heritage Library (BHL)</a>.<br>
So many beautiful images of biodiversity, moreover free to use! In<br>
particular, I <a rel="nofollow" href="https://www.flickr.com/photos/biodivlibrary/sets/72157694191194992" target="_blank">downloaded all pictures from one of the Birds of<br>
Australia<br>
albums</a>.</p>
<p>I wanted to try to extract the bird names from images using packages of<br>
Jeroen Ooms', rOpenSci's post-doc hacker &amp; C(++)-bindings wizard. For<br>
that I worked with <a rel="nofollow" href="https://github.com/ropensci/magick" target="_blank"><code>magick</code></a> for<br>
image manipulation, <a rel="nofollow" href="https://github.com/ropensci/tesseract" target="_blank"><code>tesseract</code></a><br>
for optical character recognition (OCR),<br>
<a rel="nofollow" href="https://github.com/ropensci/cld2" target="_blank"><code>cld2</code></a>/<a rel="nofollow" href="https://github.com/ropensci/cld3" target="_blank"><code>cld3</code></a><br>
for language detectionâ€¦ Quite the armory! We'll also sprinkle some<br>
taxonomy magic by Scott Chamberlain, one of rOpenSci's co-founders, to<br>
resolve the names extracted.</p>
<h3>OCR bird naming workflow, piece by piece</h3>
<p>In this section, we explain the different elements of our R workflow:<br>
preparing images, extracting text, resolving taxonomic names.</p>
<h4>Image preparation</h4>
<p>I saved the pictures locally in a &quot;birdsâ€� folder. Yes, I click-buttoned<br>
instead of using <a rel="nofollow" href="https://www.flickr.com/services/api/" target="_blank">the Flickr API</a><br>
for which e.g.Â Jim Hester wrote a <a rel="nofollow" href="https://github.com/jimhester/flickrr" target="_blank">minimal R<br>
wrapper</a>â€¦ I don't do <em>everything</em><br>
with R scripts (yet).</p>
<pre><code>library(&quot;magrittr&quot;)
filenames &lt;- fs::dir_ls(&quot;birds&quot;)
</code></pre>
<p>Each image shows a bird and its name. Images are either landscape- or<br>
portrait-oriented, but for the sake of simplicity, I'll act as if they<br>
were all portrait-oriented. <em>A possible easy and lazy fix here would be<br>
to duplicate images rotated (<code>magick::image_rotate</code>) in all possible<br>
directions and then to apply the workflow to all 4 versions of each<br>
image, hoping to get good data from one of the rotated versions.</em></p>
<pre><code>magick::image_read(filenames[1])
</code></pre>
<p><img src="https://i0.wp.com/ropensci.org/img/blog-images/2018-08-28-birds-ocr/unnamed-chunk-2-1.png?w=450&ssl=1" data-recalc-dims="1"></p>
<p>From that image, I wanted to extract the name indicated below the bird.<br>
To maximize the efficiency of OCR, I shall first prepare the image,<br>
since the <a rel="nofollow" href="https://cran.r-project.org/web/packages/tesseract/vignettes/intro.html#preprocessing_with_magick" target="_blank">accuracy of OCR depends on the quality of the input<br>
image</a><br>
which can be influenced a bit. This part could be tweaked even more, and<br>
in real life examples it'll be worth spending time trying different<br>
<code>magick</code> functions and parameter values. Since I have in mind the case<br>
of a bunch of images to be batch-processed, nothing is done by hand.</p>
<pre><code>crop_bird &lt;- function(filename){
  image &lt;- magick::image_read(filename)
  
  height &lt;- magick::image_info(image)$height
  
  # crop the top of the image
  image &lt;- magick::image_crop(image, 
                     paste0(&quot;+0+&quot;,round(0.75*height))) %>%
    # convert the image to black and white
    magick::image_convert(type = &quot;grayscale&quot;) %>%
    # increase brightness
    magick::image_modulate(brightness = 120) %>%
    magick::image_enhance() %>%
    magick::image_median() %>%
    magick::image_contrast() 
  
  # we'll need the filename later
  attr(image, &quot;filename&quot;) &lt;- filename
  
  return(image)
}

crop_bird(filenames[1])
</code></pre>
<p><img src="https://i2.wp.com/ropensci.org/img/blog-images/2018-08-28-birds-ocr/unnamed-chunk-3-1.png?w=450&ssl=1" data-recalc-dims="1"></p>
<p> It does look cleaner now! </p>
<h4>Text extraction</h4>
<p>Now is the actual OCR step! The <code>tesseract</code> package provides bindings to<br>
the <a rel="nofollow" href="https://en.wikipedia.org/wiki/Tesseract_(software)" target="_blank">Tesseract OCR<br>
engine</a>, free<br>
software currently sponsored by Google. It is a powerful engine, with a<br>
ton of parameters. Here again, tweaking a lot is warranted. Particularly<br>
useful reads are <a rel="nofollow" href="https://cran.r-project.org/web/packages/tesseract/vignettes/intro.html" target="_blank"><code>tesseract</code><br>
vignette</a><br>
and this <a rel="nofollow" href="https://github.com/tesseract-ocr/tesseract/wiki/ImproveQuality" target="_blank">Wiki page of Tesseract about improving the quality of the<br>
output</a>.<br>
The <a rel="nofollow" href="https://github.com/dmi3kno/hocr" target="_blank"><code>hocr</code> package</a> might be of<br>
interest for post-processing of OCR results.</p>
<p>Below, the only option changed from default is the <a rel="nofollow" href="https://github.com/tesseract-ocr/tesseract/wiki/ImproveQuality#page-segmentation-method" target="_blank">page segmentation<br>
mode</a><br>
choosing 1 for &quot;Automatic page segmentation with Orientation and script<br>
detection (OSD)â€�. When using Latin instead of English training data the<br>
results were not as good.</p>
<p>One can use either <code>tesseract::ocr</code> for a file path, url, or raw vector<br>
to image, or <code>magick::image_ocr</code> for a <code>magick</code> object which is quite<br>
handy in our pipeline.</p>
<p>The function below also filters results of the OCR using language<br>
detection. By only keeping text recognized as either Latin or English by<br>
one of the two language detection packages <code>cld2</code> and <code>cld3</code> that are<br>
interfaces to Google compact language detectors 2 and 3, one gets a<br>
first quality filter. If not doing that, the output would contain more<br>
unusable text.</p>
<pre><code>get_names &lt;- function(image){
  filename &lt;- attr(image, &quot;filename&quot;)
  ocr_options &lt;- list(tessedit_pageseg_mode = 1)
   
  text &lt;- magick::image_ocr(image, options = ocr_options)
  text &lt;- stringr::str_split(text, &quot;\n&quot;, simplify = TRUE)
  text &lt;- stringr::str_remove_all(text, &quot;[0-9]&quot;)
  text &lt;- stringr::str_remove_all(text, &quot;[:punct:]&quot;)
  text &lt;- trimws(text)
  text &lt;- stringr::str_remove_all(text, &quot;~&quot;)
  text &lt;- text[text != &quot;&quot;]
  text &lt;- tolower(text)
  
  # remove one letter words
  # https://stackoverflow.com/questions/31203843/r-find-and-remove-all-one-to-two-letter-words
  text &lt;- stringr::str_remove_all(text, &quot; *\\b[[:alpha:]]{1,2}\\b *&quot;)
  text &lt;- text[text != &quot;&quot;]
  
  # keep only the words that are recognized as either Latin
  # or English by cld2 or cld3
  if(length(text) > 0){
    results &lt;- tibble::tibble(text = text,
                 cld2 = cld2::detect_language(text),
                 cld3 = cld3::detect_language(text),
                 filename = filename)
  
  results[results $cld2 %in% c(&quot;la&quot;, &quot;en&quot;) |
          results$cld3 %in% c(&quot;la&quot;, &quot;en&quot;),]
  }else{
    return(NULL)
  }
  
  
}

(results1 &lt;- filenames[1] %>%
  magick::image_read() %>%
  get_names())
</code></pre>
<pre><code>## NULL
</code></pre>
<p>Nothing! Now if we replace <code>magick::image_read</code> with the previously<br>
defined <code>crop_bird</code> function that crops and cleans the imageâ€¦</p>
<pre><code>(results2 &lt;- filenames[1] %>%
  crop_bird() %>%
  get_names())
</code></pre>
<pre><code>## # A tibble: 2 x 4
##   text                 cld2  cld3  filename                          
##   </code></pre>