<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/extracting-and-processing-ebird-data/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://ropensci.org/blog/2018/08/07/auk/"> rOpenSci - open tools for open science</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><center>
<p><img alt="auk hex sticker" src="https://i0.wp.com/ropensci.org/img/blog-images/2018-08-07-auk/auk.png?w=300&ssl=1" data-recalc-dims="1"></p>
<p></p>
<p><a rel="nofollow" href="http://www.ebird.org/" target="_blank">eBird</a> is an online tool for recording bird<br>
observations. The eBird database currently contains over 500 million<br>
records of bird sightings, spanning every country and nearly every bird<br>
species, making it an extremely valuable resource for bird research and<br>
conservation. These data can be used to map the distribution and<br>
abundance of species, and assess how species’ ranges are changing over<br>
time. This dataset is available for download as a text file; however,<br>
this file is huge (over 180 GB!) and, therefore, poses some unique<br>
challenges. In particular, it isn’t possible to import and manipulate<br>
the full dataset in R. Working with these data typically requires<br>
filtering them to a smaller subset of desired observations before<br>
reading into R. This filtering is most efficiently done using AWK, a<br>
Unix utility and programming language for processing column formatted<br>
text data. The <code>auk</code> package acts as a front end for AWK, allowing users<br>
to filter eBird data before import into R, and provides tools to perform<br>
some important pre-processing of the data. Them name of this package<br>
comes from the happy coincidence that the command line tool<br>
<a rel="nofollow" href="https://www.gnu.org/software/gawk/manual/gawk.html" target="_blank">AWK</a>, upon which<br>
the package is based, is pronounced the same as<br>
<a rel="nofollow" href="https://en.wikipedia.org/wiki/Auk" target="_blank">auk</a>, the family of sea birds also<br>
known as Alcids.</p>
<p>This post will demonstrate how to use <code>auk</code> to extract eBird data for<br>
two bird species (Swainson’s and Bicknell’s Thrush) in New Hampshire,<br>
then use these data to produce some maps comparing the distributions of<br>
these species.</p>
<h3>Installation</h3>
<p>This package is available on<br>
<a rel="nofollow" href="https://cran.r-project.org/package=auk" target="_blank">CRAN</a> and can be installed<br>
with:</p>
<pre><code>install.packages(&quot;auk&quot;)
</code></pre>
<p>The development version can be installed <a rel="nofollow" href="https://github.com/CornellLabofOrnithology/auk/" target="_blank">from GitHub</a> using <code>devtools</code>. Windows users should use this development version since it fixes a bug present in the CRAN version:</p>
<pre><code>devtools::install_github(&quot;CornellLabofOrnithology/auk&quot;)
</code></pre>
<p><code>auk</code> requires the Unix utility AWK, which is available on most Linux<br>
and Mac OS X machines. Windows users will first need to install<br>
<a rel="nofollow" href="https://www.cygwin.com/" target="_blank">Cygwin</a> before using this package. Note that<br>
<strong>Cygwin should be installed in the default location</strong><br>
(<code>C:/cygwin/bin/gawk.exe</code> or <code>C:/cygwin64/bin/gawk.exe</code>) in order for<br>
<code>auk</code> to work.</p>
<h3>Accessing the data</h3>
<p>Access to the full eBird dataset is provided via two large,<br>
tab-separated text files. The eBird Basic Dataset (EBD) contains the<br>
bird observation information and consists of one line for each<br>
observation of a species on a checklist. The Sampling Event Data<br>
contains just the checklist-level information (e.g. time and date,<br>
location, and search effort information) and consists of one line for<br>
each checklist.</p>
<p>To obtain the eBird data, begin by <a rel="nofollow" href="https://secure.birds.cornell.edu/cassso/login" target="_blank">creating an eBird account and<br>
signing in</a>. eBird data<br>
are freely available; however, you will need to <a rel="nofollow" href="http://ebird.org/ebird/data/request" target="_blank">request<br>
access</a> in order to obtain the EBD.<br>
Filling out the data request form allows eBird to keep track of the<br>
number of people using the data and obtain information on the<br>
applications for which the data are used.</p>
<p>Once you have access to the data, proceed to the <a rel="nofollow" href="http://ebird.org/ebird/data/download/ebd" target="_blank">download<br>
page</a>. Download and uncompress<br>
(twice!) both the EBD and the corresponding Sampling Event Data. Put<br>
these two large text files somewhere sensible on either your computer’s<br>
hard drive or an external drive, and remember the location of containing<br>
folder.</p>
<h3>Using <code>auk</code></h3>
<p>There are essentially five main tasks that <code>auk</code> is designed to<br>
accomplish, and I’ll use a simple example to demonstrate each in turn:</p>
<ol>
<li>Cleaning</li>
<li>Filtering</li>
<li>Importing</li>
<li>Pre-processing</li>
<li>Zero-filling</li>
</ol>
<p>Let’s start by loading the required packages and defining a variable for<br>
the path to the folder that contains both the EBD sand sampling event<br>
data.</p>
<pre><code>library(auk)
library(raster)
library(tidyverse)
library(sf)
library(rnaturalearth)
library(tigris)
library(viridisLite)
# path to ebird data
# change this to the location where you saved the data
ebd_dir &lt;- &quot;/Users/mes335/data/ebird/ebd_relMay-2018&quot; 
</code></pre>
<h4>1. Cleaning</h4>
<p>Some rows in the EBD may have special characters in the comments fields<br>
(for example, tabs) that will cause import errors, and the dataset has<br>
an extra blank column at the end. The function <code>auk_clean()</code> drops these<br>
problematic records and removes the blank column. In addition, you can<br>
use <code>remove_text = TRUE</code> to remove free text entry columns, including<br>
the comments and location and observation names. These fields are<br>
typically not required and removing them can significantly decrease file<br>
size.</p>
<p>This process should be run on both the EBD and sampling event data. It<br>
typically takes several hours for the full EBD; however, it only needs<br>
to be run once because the output from the process is saved out to a new<br>
tab-separated text file for subsequent use. After running <code>auk_clean()</code>,<br>
you can delete the original, uncleaned data files to save space</p>
<pre><code># ebd
f &lt;- file.path(ebd_dir, &quot;ebd_relMay-2018.txt&quot;)
f_clean &lt;- file.path(ebd_dir, &quot;ebd_relMay-2018_clean.txt&quot;)
auk_clean(f, f_out = f_clean, remove_text = TRUE)
# sampling
f_sampling &lt;- file.path(ebd_dir, &quot;ebd_sampling_relMay-2018.txt&quot;)
f_sampling_clean &lt;- file.path(ebd_dir, &quot;ebd_sampling_relMay-2018_clean.txt&quot;)
auk_clean(f, f_out = f_sampling_clean, remove_text = TRUE)
</code></pre>
<h4>2. Filtering</h4>
<p>The EBD is huge! If we’re going to work with it, we need to extract a<br>
manageable subset of the data. With this in mind, the main purpose of<br>
<code>auk</code> is to provide a variety of functions to define taxonomic, spatial,<br>
temporal, or effort-based filters. To get started, we’ll use <code>auk_ebd()</code><br>
to set up a reference to the EBD. We’ll also provide a reference to the<br>
sampling event data. This step is optional, but it will allow us to<br>
apply exactly the same set of filters (except for taxonomic filters) to<br>
the sampling event data and the EBD. We’ll see why this is valuable<br>
later.</p>
<pre><code># define the paths to ebd and sampling event files
f_in_ebd &lt;- file.path(ebd_dir, &quot;ebd_relMay-2018_clean.txt&quot;)
f_in_sampling &lt;- file.path(ebd_dir, &quot;ebd_sampling_relMay-2018_clean.txt&quot;)
# create an object referencing these files
auk_ebd(file = f_in_ebd, file_sampling = f_in_sampling)
#> Input 
#>   EBD: /Users/mes335/data/ebird/ebd_relMay-2018/ebd_relMay-2018_clean.txt 
#>   Sampling events: /Users/mes335/data/ebird/ebd_relMay-2018/ebd_sampling_relMay-2018_clean.txt 
#> 
#> Output 
#>   Filters not executed
#> 
#> Filters 
#>   Species: all
#>   Countries: all
#>   States: all
#>   Spatial extent: full extent
#>   Date: all
#>   Start time: all
#>   Last edited date: all
#>   Protocol: all
#>   Project code: all
#>   Duration: all
#>   Distance travelled: all
#>   Records with breeding codes only: no
#>   Complete checklists only: no
</code></pre>
<p>As an example, let’s extract records from New Hampshire for Bicknell’s<br>
Thrush and Swainson’s Thrush from<br>
<a rel="nofollow" href="http://help.ebird.org/customer/portal/articles/1006361-are-you-reporting-all-species" target="_blank">complete</a><br>
checklists in June of any year. Bicknell’s Thrush breeds at high<br>
elevations in New England, such as the White Mountains of New Hampshire,<br>
one of my favourite places in the Northeast. Swainson’s Thrush is a<br>
related, and much more widespread species, that breeds at lower<br>
elevations in the same area. While hiking in the White Mountains in the<br>
early summer, I love listening for the turnover from Swainson’s to<br>
Bicknell’s Thrush as I move up in elevation. Let’s see if this pattern<br>
is reflected in the data.</p>
<p>Note that each filtering criterion corresponds to a different function,<br>
all functions begin with <code>auk_</code> for easy auto-completion, and the<br>
functions can be chained together to define multiple filters. Also,<br>
printing the <code>auk_ebd</code> object will show which filters have been defined.</p>
<pre><code>ebd_filters &lt;- auk_ebd(f_in_ebd, f_in_sampling) %>% 
  auk_species(c(&quot;Bicknell's Thrush&quot;, &quot;Swainson's Thrush&quot;)) %>% 
  auk_state(&quot;US-NH&quot;) %>% 
  auk_date(c(&quot;*-06-01&quot;, &quot;*-06-30&quot;)) %>% 
  auk_complete()
ebd_filters
#> Input 
#>   EBD: /Users/mes335/data/ebird/ebd_relMay-2018/ebd_relMay-2018_clean.txt 
#>   Sampling events: /Users/mes335/data/ebird/ebd_relMay-2018/ebd_sampling_relMay-2018_clean.txt 
#> 
#> Output 
#>   Filters not executed
#> 
#> Filters 
#>   Species: Catharus bicknelli, Catharus ustulatus
#>   Countries: all
#>   States: US-NH
#>   Spatial extent: full extent
#>   Date: *-06-01 - *-06-30
#>   Start time: all
#>   Last edited date: all
#>   Protocol: all
#>   Project code: all
#>   Duration: all
#>   Distance travelled: all
#>   Records with breeding codes only: no
#>   Complete checklists only: yes
</code></pre>
<p>This is just a small set of the potential filters, for a full list of<br>
filters consult the<br>
<a rel="nofollow" href="https://cornelllabofornithology.github.io/auk/reference/index.html#section-filter" target="_blank">documentation</a>.</p>
<p>The above code only <em>defines</em> the filters, no data has actually been<br>
extracted yet. To compile the filters into an AWK script and run it, use<br>
<code>auk_filter()</code>. Since I provided an EBD and sampling event file to<br>
<code>auk_ebd()</code>, both will be filtered and I will need to provide two output<br>
filenames.</p>
<pre><code>f_out_ebd &lt;- &quot;data/ebd_thrush_nh.txt&quot;
f_out_sampling &lt;- &quot;data/ebd_thrush_nh_sampling.txt&quot;
ebd_filtered &lt;- auk_filter(ebd_filters, file = f_out_ebd, 
                           file_sampling = f_out_sampling)
</code></pre>
<p>Running <code>auk_filter()</code> takes a long time, typically a few hours, so<br>
you’ll need to be patient. To download the pre-filtered data used in<br>
the remainder of this post, visit the <a rel="nofollow" href="https://github.com/mstrimas/auk-blog-post/tree/master/data" target="_blank">GitHub<br>
repository</a>.</p>
<h4>3. Importing</h4>
<p>Now that we have an EBD extract of a reasonable size, we can read it<br>
into an R data frame. The files output from <code>auk_filter()</code> are just<br>
tab-separated text files, so we could read them using any of our usual R<br>
tools, e.g. <code>read.delim()</code>. However, <code>auk</code> contains functions<br>
specifically designed for reading in EBD data. These functions choose<br>
sensible variable names, set the data types of columns correctly, and<br>
perform two important post-processing steps: taxonomic roll-up and<br>
de-duplicating group checklists.</p>
<p>We’ll put the sampling event data aside for now, and just read in the<br>
EBD:</p>
<pre><code>ebd &lt;- read_ebd(f_out_ebd)
glimpse(ebd)
#> Observations: 2,253
#> Variables: 42
#> $ checklist_id                 </code></pre></center>