eBird is an online tool for recording bird
observations. The eBird database currently contains over 500 million
records of bird sightings, spanning every country and nearly every bird
species, making it an extremely valuable resource for bird research and
conservation. These data can be used to map the distribution and
abundance of species, and assess how species’ ranges are changing over
time. This dataset is available for download as a text file; however,
this file is huge (over 180 GB!) and, therefore, poses some unique
challenges. In particular, it isn’t possible to import and manipulate
the full dataset in R. Working with these data typically requires
filtering them to a smaller subset of desired observations before
reading into R. This filtering is most efficiently done using AWK, a
Unix utility and programming language for processing column formatted
text data. The auk package acts as a front end for AWK, allowing users
to filter eBird data before import into R, and provides tools to perform
some important pre-processing of the data. Them name of this package
comes from the happy coincidence that the command line tool
AWK, upon which
the package is based, is pronounced the same as
auk, the family of sea birds also
known as Alcids.
This post will demonstrate how to use auk to extract eBird data for
two bird species (Swainson’s and Bicknell’s Thrush) in New Hampshire,
then use these data to produce some maps comparing the distributions of
these species.
Installation
This package is available on
CRAN and can be installed
with:
install.packages(&quot;auk&quot;)
The development version can be installed from GitHub using devtools. Windows users should use this development version since it fixes a bug present in the CRAN version:
devtools::install_github(&quot;CornellLabofOrnithology/auk&quot;)
auk requires the Unix utility AWK, which is available on most Linux
and Mac OS X machines. Windows users will first need to install
Cygwin before using this package. Note that
Cygwin should be installed in the default location
(C:/cygwin/bin/gawk.exe or C:/cygwin64/bin/gawk.exe) in order for
auk to work.
Accessing the data
Access to the full eBird dataset is provided via two large,
tab-separated text files. The eBird Basic Dataset (EBD) contains the
bird observation information and consists of one line for each
observation of a species on a checklist. The Sampling Event Data
contains just the checklist-level information (e.g. time and date,
location, and search effort information) and consists of one line for
each checklist.
To obtain the eBird data, begin by creating an eBird account and
signing in. eBird data
are freely available; however, you will need to request
access in order to obtain the EBD.
Filling out the data request form allows eBird to keep track of the
number of people using the data and obtain information on the
applications for which the data are used.
Once you have access to the data, proceed to the download
page. Download and uncompress
(twice!) both the EBD and the corresponding Sampling Event Data. Put
these two large text files somewhere sensible on either your computer’s
hard drive or an external drive, and remember the location of containing
folder.
Using auk
There are essentially five main tasks that auk is designed to
accomplish, and I’ll use a simple example to demonstrate each in turn:
Cleaning
Filtering
Importing
Pre-processing
Zero-filling
Let’s start by loading the required packages and defining a variable for
the path to the folder that contains both the EBD sand sampling event
data.
library(auk)
library(raster)
library(tidyverse)
library(sf)
library(rnaturalearth)
library(tigris)
library(viridisLite)
# path to ebird data
# change this to the location where you saved the data
ebd_dir #__ Filters #__ Species: all
#__ Countries: all
#__ States: all
#__ Spatial extent: full extent
#__ Date: all
#__ Start time: all
#__ Last edited date: all
#__ Protocol: all
#__ Project code: all
#__ Duration: all
#__ Distance travelled: all
#__ Records with breeding codes only: no
#__ Complete checklists only: no
As an example, let’s extract records from New Hampshire for Bicknell’s
Thrush and Swainson’s Thrush from
complete
checklists in June of any year. Bicknell’s Thrush breeds at high
elevations in New England, such as the White Mountains of New Hampshire,
one of my favourite places in the Northeast. Swainson’s Thrush is a
related, and much more widespread species, that breeds at lower
elevations in the same area. While hiking in the White Mountains in the
early summer, I love listening for the turnover from Swainson’s to
Bicknell’s Thrush as I move up in elevation. Let’s see if this pattern
is reflected in the data.
Note that each filtering criterion corresponds to a different function,
all functions begin with auk_ for easy auto-completion, and the
functions can be chained together to define multiple filters. Also,
printing the auk_ebd object will show which filters have been defined.
ebd_filters % auk_species(c(&quot;Bicknell's Thrush&quot;, &quot;Swainson's Thrush&quot;)) %__% auk_state(&quot;US-NH&quot;) %__% auk_date(c(&quot;*-06-01&quot;, &quot;*-06-30&quot;)) %__% auk_complete()
ebd_filters
#__ Input #__ EBD: /Users/mes335/data/ebird/ebd_relMay-2018/ebd_relMay-2018_clean.txt #__ Sampling events: /Users/mes335/data/ebird/ebd_relMay-2018/ebd_sampling_relMay-2018_clean.txt #__ #__ Output #__ Filters not executed
#__ #__ Filters #__ Species: Catharus bicknelli, Catharus ustulatus
#__ Countries: all
#__ States: US-NH
#__ Spatial extent: full extent
#__ Date: *-06-01 - *-06-30
#__ Start time: all
#__ Last edited date: all
#__ Protocol: all
#__ Project code: all
#__ Duration: all
#__ Distance travelled: all
#__ Records with breeding codes only: no
#__ Complete checklists only: yes
This is just a small set of the potential filters, for a full list of
filters consult the
documentation.
The above code only defines the filters, no data has actually been
extracted yet. To compile the filters into an AWK script and run it, use
auk_filter(). Since I provided an EBD and sampling event file to
auk_ebd(), both will be filtered and I will need to provide two output
filenames.
f_out_ebd $ last_edited_date &quot;2014-05-07 19:23:51&quot;, &quot;2014-05-0...
#__ $ taxonomic_order 24602, 24600, 24600, 24602, 24602...
#__ $ category &quot;species&quot;, &quot;species&quot;, &quot;species&quot;, ...
#__ $ common_name &quot;Swainson's Thrush&quot;, &quot;Bicknell's ...
#__ $ scientific_name &quot;Catharus ustulatus&quot;, &quot;Catharus b...
#__ $ observation_count &quot;X&quot;, &quot;1&quot;, &quot;2&quot;, &quot;1&quot;, &quot;1&quot;, &quot;2&quot;, &quot;5&quot;...
#__ $ breeding_bird_atlas_code NA, NA, NA, NA, NA, NA, NA, NA, N...
#__ $ breeding_bird_atlas_category NA, NA, NA, NA, NA, NA, NA, NA, N...
#__ $ age_sex NA, NA, NA, NA, NA, NA, NA, NA, N...
#__ $ country &quot;United States&quot;, &quot;United States&quot;,...
#__ $ country_code &quot;US&quot;, &quot;US&quot;, &quot;US&quot;, &quot;US&quot;, &quot;US&quot;, &quot;US...
#__ $ state &quot;New Hampshire&quot;, &quot;New Hampshire&quot;,...
#__ $ state_code &quot;US-NH&quot;, &quot;US-NH&quot;, &quot;US-NH&quot;, &quot;US-NH...
#__ $ county &quot;Grafton&quot;, &quot;Grafton&quot;, &quot;Coos&quot;, &quot;Co...
#__ $ county_code &quot;US-NH-009&quot;, &quot;US-NH-009&quot;, &quot;US-NH-...
#__ $ iba_code NA, NA, &quot;US-NH_2408&quot;, &quot;US-NH_2419...
#__ $ bcr_code 14, 14, 14, 14, 14, 14, 14, 14, 1...
#__ $ usfws_code NA, NA, NA, NA, NA, NA, NA, NA, N...
#__ $ atlas_block NA, NA, NA, NA, NA, NA, NA, NA, N...
#__ $ locality_id &quot;L722150&quot;, &quot;L291562&quot;, &quot;L295828&quot;, ...
#__ $ locality_type &quot;H&quot;, &quot;H&quot;, &quot;H&quot;, &quot;H&quot;, &quot;H&quot;, &quot;H&quot;, &quot;H&quot;...
#__ $ latitude 43.94204, 44.16991, 44.27132, 45....
#__ $ longitude -71.72562, -71.68741, -71.30302, ...
#__ $ observation_date 2008-06-01, 2004-06-30, 2008-06-...
#__ $ time_observations_started NA, NA, NA, &quot;09:00:00&quot;, &quot;08:00:00...
#__ $ observer_id &quot;obsr50351&quot;, &quot;obsr131204&quot;, &quot;obsr1...
#__ $ sampling_event_identifier &quot;S3910100&quot;, &quot;S4263131&quot;, &quot;S3936987...
#__ $ protocol_type &quot;Incidental&quot;, &quot;Incidental&quot;, &quot;Inci...
#__ $ protocol_code &quot;P20&quot;, &quot;P20&quot;, &quot;P20&quot;, &quot;P22&quot;, &quot;P22&quot;...
#__ $ project_code &quot;EBIRD&quot;, &quot;EBIRD&quot;, &quot;EBIRD_BCN&quot;, &quot;E...
#__ $ duration_minutes NA, NA, NA, 120, 180, 180, 180, 1...
#__ $ effort_distance_km NA, NA, NA, 7.242, 11.265, 11.265...
#__ $ effort_area_ha NA, NA, NA, NA, NA, NA, NA, NA, N...
#__ $ number_observers NA, 1, NA, 2, 2, 2, 2, 2, 2, 16, ...
#__ $ all_species_reported TRUE, TRUE, TRUE, TRUE, TRUE, TRU...
#__ $ group_identifier NA, NA, NA, NA, NA, NA, NA, NA, N...
#__ $ has_media FALSE, FALSE, FALSE, FALSE, FALSE...
#__ $ approved TRUE, TRUE, TRUE, TRUE, TRUE, TRU...
#__ $ reviewed FALSE, FALSE, FALSE, FALSE, FALSE...
#__ $ reason NA, NA, NA, NA, NA, NA, NA, NA, N...
4. Pre-processing
By default, two important pre-processing steps are performed to handle
taxonomy and group checklists. In most cases, you’ll want this to be
done; however, these can be turned off to get the raw data.
ebd_raw Observations: 15,966
#__ Variables: 8
#__ $ taxon_order 3.0, 5.0, 6.0, 7.0, 13.0, 14.0, 17.0, 32.0, 33...
#__ $ category &quot;species&quot;, &quot;species&quot;, &quot;slash&quot;, &quot;species&quot;, &quot;spe...
#__ $ species_code &quot;ostric2&quot;, &quot;ostric3&quot;, &quot;y00934&quot;, &quot;grerhe1&quot;, &quot;le...
#__ $ common_name &quot;Common Ostrich&quot;, &quot;Somali Ostrich&quot;, &quot;Common/So...
#__ $ scientific_name &quot;Struthio camelus&quot;, &quot;Struthio molybdophanes&quot;, ...
#__ $ order &quot;Struthioniformes&quot;, &quot;Struthioniformes&quot;, &quot;Strut...
#__ $ family &quot;Struthionidae (Ostriches)&quot;, &quot;Struthionidae (O...
#__ $ report_as NA, NA, NA, NA, NA, &quot;lesrhe2&quot;, &quot;lesrhe2&quot;, NA, ...
We can use one of the example datasets in the package to explore what
auk_rollup() does.
# get the path to the example data included in the package
f [1] &quot;species&quot;
# yellow-rump warbler subspecies rollup
# without rollup, there are three observations
ebd_wo_ru %__%
filter(common_name == &quot;Yellow-rumped Warbler&quot;) %__% select(checklist_id, category, common_name, subspecies_common_name, observation_count)
#__ # A tibble: 5 x 5
#__ checklist_id category common_name subspecies_common_… observation_cou…
#__ #__ 1 S45216549 intergra… Yellow-rump… Yellow-rumped Warb… 3 #__ 2 S45115209 intergra… Yellow-rump… Yellow-rumped Warb… 1 #__ 3 S33053360 issf Yellow-rump… Yellow-rumped Warb… 2 #__ 4 S33053360 issf Yellow-rump… Yellow-rumped Warb… 1 #__ 5 S33053360 species Yellow-rump… 2
# with rollup, they have been combined
ebd_w_ru %__%
filter(common_name == &quot;Yellow-rumped Warbler&quot;) %__% select(checklist_id, category, common_name, observation_count)
#__ # A tibble: 3 x 4
#__ checklist_id category common_name observation_count
#__ #__ 1 S45216549 species Yellow-rumped Warbler 3 #__ 2 S45115209 species Yellow-rumped Warbler 1 #__ 3 S33053360 species Yellow-rumped Warbler 5
Group checklists
eBird observers birding together can share checklists resulting in group
checklists. In the simplest case, all observers will have seen the same
set of species; however, observers can also add or remove species from
their checklist. In the EBD, group checklists result in duplicate
records, one for each observer. auk_unique() (called by default by
read_ebd()) de-duplicates the EBD, resulting in one record for each
species on each group checklist.
5. Zero-filling
So far we’ve been working with presence-only data; however, many
applications of the eBird data require presence-absence information.
Although observers only explicitly record presence, they have the option
of designating their checklists as “complete”, meaning they are
reporting all the species they saw or heard. With complete checklists,
any species not reported can be taken to have an implicit count of zero.
Therefore, by focusing on complete checklists, we can use the sampling
event data to “zero-fill” the EBD producing presence-absence data. This
is why it’s important to filter both the EBD and the sampling event data
at the same time; we need to ensure that the EBD observations are drawn
from the population of checklists defined by the sampling event data.
Given an EBD file or data frame, and corresponding sampling event data,
the function auk_zerofill() produces zero-filled, presence-absence
data.
zf Zero-filled EBD: 12,938 unique checklists, for 2 species.
zf$observations
#__ # A tibble: 25,876 x 4
#__ checklist_id scientific_name observation_count species_observed
#__ #__ 1 G1003224 Catharus bicknelli 0 FALSE #__ 2 G1003224 Catharus ustulatus X TRUE #__ 3 G1022603 Catharus bicknelli 0 FALSE #__ 4 G1022603 Catharus ustulatus 0 FALSE #__ 5 G1054429 Catharus bicknelli 3 TRUE #__ 6 G1054429 Catharus ustulatus X TRUE #__ 7 G1054430 Catharus bicknelli 0 FALSE #__ 8 G1054430 Catharus ustulatus X TRUE #__ 9 G1054431 Catharus bicknelli 1 TRUE #__ 10 G1054431 Catharus ustulatus X TRUE #__ # ... with 25,866 more rows
zf$sampling_events
#__ # A tibble: 12,938 x 29
#__ checklist_id last_edited_date country country_code state state_code
#__ #__ 1 S10836735 2014-05-07 19:1… United… US New … US-NH #__ 2 S10936397 2014-05-07 19:0… United… US New … US-NH #__ 3 S10995603 2014-05-07 19:0… United… US New … US-NH #__ 4 S11005948 2012-06-19 12:4… United… US New … US-NH #__ 5 S11013341 2012-06-20 18:2… United… US New … US-NH #__ 6 S15136517 2013-09-10 08:1… United… US New … US-NH #__ 7 S11012701 2017-06-12 06:3… United… US New … US-NH #__ 8 S10995655 2012-06-17 19:5… United… US New … US-NH #__ 9 S15295339 2014-05-07 18:5… United… US New … US-NH #__ 10 S10932312 2014-05-07 18:4… United… US New … US-NH #__ # ... with 12,928 more rows, and 23 more variables: county ,
#__ # county_code , iba_code , bcr_code , usfws_code ,
#__ # atlas_block , locality_id , locality_type ,
#__ # latitude , longitude , observation_date ,
#__ # time_observations_started , observer_id ,
#__ # sampling_event_identifier , protocol_type ,
#__ # protocol_code , project_code , duration_minutes ,
#__ # effort_distance_km , effort_area_ha ,
#__ # number_observers , all_species_reported ,
#__ # group_identifier The resulting auk_zerofill object is a list of two data frames:
observations stores the species’ counts for each checklist and
sampling_events stores the checklists. The checklist_id field can be
used to combine the files together manually, or you can use the
collapse_zerofill() function.
collapse_zerofill(zf)
#__ # A tibble: 25,876 x 32
#__ checklist_id last_edited_date country country_code state state_code
#__ #__ 1 S10836735 2014-05-07 19:1… United… US New … US-NH #__ 2 S10836735 2014-05-07 19:1… United… US New … US-NH #__ 3 S10936397 2014-05-07 19:0… United… US New … US-NH #__ 4 S10936397 2014-05-07 19:0… United… US New … US-NH #__ 5 S10995603 2014-05-07 19:0… United… US New … US-NH #__ 6 S10995603 2014-05-07 19:0… United… US New … US-NH #__ 7 S11005948 2012-06-19 12:4… United… US New … US-NH #__ 8 S11005948 2012-06-19 12:4… United… US New … US-NH #__ 9 S11013341 2012-06-20 18:2… United… US New … US-NH #__ 10 S11013341 2012-06-20 18:2… United… US New … US-NH #__ # ... with 25,866 more rows, and 26 more variables: county ,
#__ # county_code , iba_code , bcr_code , usfws_code ,
#__ # atlas_block , locality_id , locality_type ,
#__ # latitude , longitude , observation_date ,
#__ # time_observations_started , observer_id ,
#__ # sampling_event_identifier , protocol_type ,
#__ # protocol_code , project_code , duration_minutes ,
#__ # effort_distance_km , effort_area_ha ,
#__ # number_observers , all_species_reported ,
#__ # group_identifier , scientific_name ,
#__ # observation_count , species_observed This zero-filled dataset is now suitable for applications such as
species distribution modeling.
Applications of eBird data
Now that we’ve imported some eBird data into R, it’s time for the fun
stuff! Whether we’re making maps or fitting models, most of the
application of eBird data will occur outside of auk; however, I’ll
include a couple simple examples here just to give you a sense for
what’s possible.
Making an occurrence map
One of the most obvious things to do with the species occurrence data is
to make a map! Here I’ll compare the distributions of the two thrush
species.
# convert ebd to spatial object
ebd_sf % select(common_name, latitude, longitude) %__% st_as_sf(coords = c(&quot;longitude&quot;, &quot;latitude&quot;), crs = 4326)
# get state boundaries using rnaturalearth package
states % st_geometry(), col = &quot;grey40&quot;, border = &quot;white&quot;, add = TRUE)
plot(nh, col = &quot;grey20&quot;, border = &quot;white&quot;, add = TRUE)
# ebird data
plot(ebd_sf %__% filter(common_name == &quot;Swainson's Thrush&quot;), col = &quot;#4daf4a99&quot;, pch = 19, cex = 0.75, add = TRUE)
plot(ebd_sf %__% filter(common_name == &quot;Bicknell's Thrush&quot;) %__% st_geometry(), col = &quot;#377eb899&quot;, pch = 19, cex = 0.75, add = TRUE)
Bicknell’s Thrush is a high elevation breeder and, as expected,
observations of this species are restricted to the White Mountains in
the North-central part of the state. In constrast, Swainson’s Thrush
occurs more widely throughout the state.
Mapping frequency of occurrence
The above map doesn’t account for effort. For example, is a higher
density of observations indicative of a biological pattern or of more
birders visiting an area? We can address this by using the zero-filled
data to produce maps of frequency of observation on checklists. To do
this, I’ll start by converting the eBird data into a spatial object in
sf format and projecting to an equal area coordinate reference system.
# pres-abs data by species
checklists_sf % st_as_sf(coords = c(&quot;longitude&quot;, &quot;latitude&quot;), crs = 4326) %__% # project to equal area
st_transform(crs =&quot;+proj=laea +lat_0=44 +lon_0=-71.70&quot;) %__% inner_join(ebird_taxonomy %__% select(species_code, scientific_name, common_name),
by = &quot;scientific_name&quot;) %__% select(species_code, common_name, scientific_name, species_observed)
Next, I’ll create a raster template from this spatial object consisting
of a regular 10km grid of cells. Then I use rasterize() to calculate
the observation frequency within each cell.
r_template % st_transform(crs = projection(r_freq)), main = common)
# land
s % st_transform(crs = projection(r_freq)) %__% st_geometry()
plot(s, col = &quot;grey60&quot;, border = NA, add = TRUE)
# frequency data
plot(r_freq[[sp]], breaks = seq(0, 1, length.out = 101), col = viridis(100),
legend = FALSE, add = TRUE)
# state lines
plot(s, col = NA, border = &quot;white&quot;, add = TRUE)
}
# legend
par(mfrow = c(1, 1), mar = c(0, 0, 0, 0), new = TRUE)
fields::image.plot(zlim = c(0, 1), legend.only = TRUE, col = viridis(100),
smallplot = c(0.25, 0.75, 0.05, 0.08),
horizontal = TRUE,
axis.args = list(at = seq(0, 1, by = 0.25), labels = c(0, 0.25, 0.5, 0.75, 1),
fg = NA, col.axis = &quot;black&quot;,
lwd.ticks = 0, line = -1),
legend.args = list(text = &quot;Observation Frequency&quot;, side = 3, col = &quot;black&quot;))
Swainson’s Thrush are more widepsread and more frequently seen, while
Bicknell’s Thrush is constrained to the White Mountains, and both
species are uncommon in southern New Hampshire.
auk vs. rebird
Those interested in eBird data may also want to consider
rebird, an R package that
provides an interface to the eBird
APIs. The
functions in rebird are mostly limited to accessing recent (
i.e. within the last 30 days) observations, although ebirdfreq() does
provide historical frequency of observation data. In contrast, auk
gives access to the full set of ~ 500 million eBird observations. For
most ecological applications, users will require auk; however, for
some use cases, e.g. building tools for birders, rebird provides a
quick and easy way to access data. rebird is part of the species
occurrence data (spocc) suite.
Users working with species occurrence data may be interested in
scrubr, for cleaning biological
occurrence records, and the rOpenSci taxonomy
suite.
Acknowledgements
This package is based on AWK scripts provided as part of the eBird Data
Workshop given by Wesley Hochachka, Daniel Fink, Tom Auer, and Frank La
Sorte at the 2016 NAOC on August 15, 2016.
auk benefited significantly from the rOpenSci
review process, including helpful suggestions from Auriel
Fournier and Edmund
Hart.