<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/finch-parse-darwin-core-files/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://ropensci.org/blog/technotes/2016/12/23/finch-release"> rOpenSci Blog - R</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><code>finch</code> has just been released to CRAN (binaries should be up soon). </p>
<p><code>finch</code> is a package to parse Darwin Core files. <a target="_blank" rel="nofollow" href="http://rs.tdwg.org/dwc/">Darwin Core</a> is:</p>
<blockquote>
<p>a body of standards. It includes a glossary of terms (in other contexts these might be called properties, elements, fields, columns, attributes, or concepts) intended to facilitate the sharing of information about biological diversity by providing reference definitions, examples, and commentaries. The Darwin Core is primarily based on taxa, their occurrence in nature as documented by observations, specimens, samples, and related information. … The Simple Darwin Core [SIMPLEDWC] is a specification for one particular way to use the terms – to share data about taxa and their occurrences in a simply structured way – and is probably what is meant if someone suggests to &quot;format your data according to the Darwin Core&quot;.</p>
</blockquote>
<p>Let's use <code>DwC</code> for short going forward. </p>
<p>GBIF (Global Biodiversity Information Facility) is the biggest holder of biodiversity data. When you request<br>
data in bulk format from GBIF they call give it to you in what's called a Darwin Core Archive, or<br>
<code>DwC-A</code>. GBIF has a validator for DwC-A files as well: <a target="_blank" rel="nofollow" href="http://tools.gbif.org/dwca-validator/">http://tools.gbif.org/dwca-validator/</a></p>
<p>One of our most used packages is probably <code>rgbif</code>, a client to interact with GBIF's web services.<br>
There's a series of functions in <code>rgbif</code> to request data in bulk format (see functions starting<br>
with <code>occ_download</code>), and from this you get a DwC-A file. This is where <code>finch</code> comes in:<br>
it can parse these DwC-A files into something useable inside R. </p>
<h2>Installation</h2>
<div>
<pre><code data-lang="r">install.packages<span>(</span><span>&quot;finch&quot;</span><span>)</span>
<span># or from source if binary not available yet</span>
install.packages<span>(</span><span>&quot;finch&quot;</span><span>,</span> type <span>=</span> <span>&quot;source&quot;</span><span>)</span>
</code></pre>
</div>
<div>
<pre><code data-lang="r"><span>library</span><span>(</span><span>&quot;finch&quot;</span><span>)</span>
</code></pre>
</div>
<p>To parse a simple darwin core file like</p>
<div>
<pre><code data-lang="text">&lt;?xml version=&quot;1.0&quot; encoding=&quot;UTF-8&quot;?>
&lt;SimpleDarwinRecordSet
 xmlns=&quot;http://rs.tdwg.org/dwc/xsd/simpledarwincore/&quot;
 xmlns:dc=&quot;http://purl.org/dc/terms/&quot;
 xmlns:dwc=&quot;http://rs.tdwg.org/dwc/terms/&quot;
 xmlns:xsi=&quot;http://www.w3.org/2001/XMLSchema-instance&quot;
 xsi:schemaLocation=&quot;http://rs.tdwg.org/dwc/xsd/simpledarwincore/ ../../xsd/tdwg_dwc_simple.xsd&quot;>
 &lt;SimpleDarwinRecord>
  &lt;dwc:occurrenceID>urn:catalog:YPM:VP.057488&lt;/dwc:occurrenceID>
  &lt;dc:type>PhysicalObject&lt;/dc:type>
  &lt;dc:modified>2009-02-12T12:43:31&lt;/dc:modified>
  &lt;dc:language>en&lt;/dc:language>
  &lt;dwc:basisOfRecord>FossilSpecimen&lt;/dwc:basisOfRecord>
  &lt;dwc:institutionCode>YPM&lt;/dwc:institutionCode>
  &lt;dwc:collectionCode>VP&lt;/dwc:collectionCode>
  &lt;dwc:catalogNumber>VP.057488&lt;/dwc:catalogNumber>
  &lt;dwc:individualCount>1&lt;/dwc:individualCount>
  &lt;dwc:locationID xsi:nil=&quot;true&quot;/>
  &lt;dwc:continent>North America&lt;/dwc:continent>
  &lt;dwc:country>United States&lt;/dwc:country>
  &lt;dwc:countryCode>US&lt;/dwc:countryCode>
  &lt;dwc:stateProvince>Montana&lt;/dwc:stateProvince>
  &lt;dwc:county>Garfield&lt;/dwc:county>
  &lt;dwc:scientificName>Tyrannosourus rex&lt;/dwc:scientificName>
  &lt;dwc:genus>Tyrannosourus&lt;/dwc:genus>
  &lt;dwc:specificEpithet>rex&lt;/dwc:specificEpithet>
  &lt;dwc:earliestPeriodOrHighestSystem>Creataceous&lt;/dwc:earliestPeriodOrHighestSystem>
  &lt;dwc:latestPeriodOrHighestSystem>Creataceous&lt;/dwc:latestPeriodOrHighestSystem>
  &lt;dwc:earliestEonOrHighestEonothem>Late Cretaceous&lt;/dwc:earliestEonOrHighestEonothem>
  &lt;dwc:latestEonOrHighestEonothem>Late Cretaceous&lt;/dwc:latestEonOrHighestEonothem>
 &lt;/SimpleDarwinRecord>
&lt;/SimpleDarwinRecordSet>
</code></pre>
</div>
<p>This file is in this package as an example file, get the file, then <code>simple()</code></p>
<div>
<pre><code data-lang="r">file <span>&lt;-</span> <span>system.file</span><span>(</span><span>&quot;examples&quot;</span><span>,</span> <span>&quot;example_simple_fossil.xml&quot;</span><span>,</span> package <span>=</span> <span>&quot;finch&quot;</span><span>)</span>
out <span>&lt;-</span> simple_read<span>(</span><span>file</span><span>)</span>
</code></pre>
</div>
<p>Index to <code>meta</code>, <code>dc</code> or <code>dwc</code></p>
<div>
<pre><code data-lang="r">out<span>$</span>dc
<span>#> [[1]]</span>
<span>#> [[1]]$type</span>
<span>#> [1] &quot;PhysicalObject&quot;</span>
<span>#> </span>
<span>#> </span>
<span>#> [[2]]</span>
<span>#> [[2]]$modified</span>
<span>#> [1] &quot;2009-02-12T12:43:31&quot;</span>
<span>#> </span>
<span>#> </span>
<span>#> [[3]]</span>
<span>#> [[3]]$language</span>
<span>#> [1] &quot;en&quot;</span>
</code></pre>
</div>
<h2>Parse Darwin Core Archive</h2>
<p>To parse a Darwin Core Archive like can be gotten from GBIF use <code>dwca_read()</code></p>
<p><code>dwca_read()</code> can parse a DwC-A file as a directory, zipped file, or from a URL.</p>
<p>There's an example Darwin Core Archive:</p>
<div>
<pre><code data-lang="r">file <span>&lt;-</span> <span>system.file</span><span>(</span><span>&quot;examples&quot;</span><span>,</span> <span>&quot;0000154-150116162929234.zip&quot;</span><span>,</span> package <span>=</span> <span>&quot;finch&quot;</span><span>)</span>
<span>(</span>out <span>&lt;-</span> dwca_read<span>(</span><span>file</span><span>,</span> read <span>=</span> <span>TRUE</span><span>))</span>
<span>#> &lt;gbif dwca></span>
<span>#> Package ID: 6cfaaf9c-d518-4ca3-8dc5-f5aadddc0390</span>
<span>#> No. data sources: 10</span>
<span>#> No. datasets: 3</span>
<span>#> Dataset occurrence.txt: [225 X 443]</span>
<span>#> Dataset multimedia.txt: [15 X 1]</span>
<span>#> Dataset verbatim.txt: [209 X 443]</span>
</code></pre>
</div>
<p>List files in the archive</p>
<div>
<pre><code data-lang="r">out<span>$</span>files
<span>#> $xml_files</span>
<span>#> [1] &quot;/Library/Frameworks/R.framework/Versions/3.3/Resources/library/finch/examples/0000154-150116162929234/meta.xml&quot; </span>
<span>#> [2] &quot;/Library/Frameworks/R.framework/Versions/3.3/Resources/library/finch/examples/0000154-150116162929234/metadata.xml&quot;</span>
<span>#> </span>
<span>#> $txt_files</span>
<span>#> [1] &quot;/Library/Frameworks/R.framework/Versions/3.3/Resources/library/finch/examples/0000154-150116162929234/citations.txt&quot; </span>
<span>#> [2] &quot;/Library/Frameworks/R.framework/Versions/3.3/Resources/library/finch/examples/0000154-150116162929234/multimedia.txt&quot;</span>
<span>#> [3] &quot;/Library/Frameworks/R.framework/Versions/3.3/Resources/library/finch/examples/0000154-150116162929234/occurrence.txt&quot;</span>
<span>#> [4] &quot;/Library/Frameworks/R.framework/Versions/3.3/Resources/library/finch/examples/0000154-150116162929234/rights.txt&quot; </span>
<span>#> [5] &quot;/Library/Frameworks/R.framework/Versions/3.3/Resources/library/finch/examples/0000154-150116162929234/verbatim.txt&quot; </span>
<span>...</span>
</code></pre>
</div>
<p>High level metadata for the whole archive (printing a subset for brevity)</p>
<div>
<pre><code data-lang="r">out<span>$</span>emlmeta
<span>#> &lt;eml packageId=&quot;6cfaaf9c-d518-4ca3-8dc5-f5aadddc0390&quot; system=&quot;http://gbif.org&quot; scope=&quot;system&quot; xml:lang=&quot;en&quot; xsi:schemaLocation=&quot;eml://ecoinformatics.org/eml-2.1.1 http://rs.gbif.org/schema/eml-gbif-profile/1.0.2/eml.xsd&quot;></span>
<span>#> &lt;dataset></span>
<span>#> &lt;title>GBIF Occurrence Download 0000154-150116162929234&lt;/title></span>
<span>#> &lt;creator></span>
<span>#> &lt;individualName></span>
<span>#> &lt;surName>GBIF Download Service&lt;/surName></span>
<span>#> &lt;/individualName></span>
<span>#> &lt;/creator></span>
<span>#> &lt;metadataProvider></span>
<span>#> &lt;individualName></span>
<span>#> &lt;surName>GBIF Download Service&lt;/surName></span>
<span>#> &lt;/individualName></span>
<span>#> &lt;/metadataProvider></span>
<span>#> &lt;associatedParty></span>
<span>#> &lt;organizationName>OZCAM (Online Zoological Collections of Australian Museums) Provider&lt;/organizationName></span>
<span>#> &lt;onlineUrl>http://www.ozcam.org.au/&lt;/onlineUrl></span>
<span>#> &lt;role>CONTENT_PROVIDER&lt;/role></span>
<span>#> &lt;/associatedParty></span>
<span>#> &lt;associatedParty></span>
<span>#> &lt;individualName></span>
<span>...</span>
</code></pre>
</div>
<p>High level metadata for each data file, there's many files, but we'll just look at one</p>
<div>
<pre><code data-lang="r">hm <span>&lt;-</span> out<span>$</span>highmeta
<span>head</span><span>(</span> hm<span>$</span>occurrence.txt <span>)</span>
<span>#> index term delimitedBy</span>
<span>#> 1 0 http://rs.gbif.org/terms/1.0/gbifID &lt;NA></span>
<span>#> 2 1 http://purl.org/dc/terms/abstract &lt;NA></span>
<span>#> 3 2 http://purl.org/dc/terms/accessRights &lt;NA></span>
<span>#> 4 3 http://purl.org/dc/terms/accrualMethod &lt;NA></span>
<span>#> 5 4 http://purl.org/dc/terms/accrualPeriodicity &lt;NA></span>
<span>#> 6 5 http://purl.org/dc/terms/accrualPolicy &lt;NA></span>
</code></pre>
</div>
<p>You can get the same metadata as above for each dataset that went into the tabular dataset downloaded</p>
<div>
<pre><code data-lang="r">out<span>$</span>dataset_meta<span>[[</span><span>1</span><span>]]</span>
</code></pre>
</div>
<p>View one of the datasets, brief overview.</p>
<div>
<pre><code data-lang="r"><span>head</span><span>(</span>out<span>$</span>data<span>[[</span><span>1</span><span>]][,</span><span>c</span><span>(</span><span>1</span><span>:</span><span>5</span><span>)])</span>
<span>#> gbifID abstract accessRights accrualMethod accrualPeriodicity</span>
<span>#> 1 50280003 NA NA NA</span>
<span>#> 2 477550574 NA NA NA</span>
<span>#> 3 239703844 NA NA NA</span>
<span>#> 4 239703843 NA NA NA</span>
<span>#> 5 239703833 NA NA NA</span>
<span>#> 6 477550692 NA NA NA</span>
</code></pre>
</div>
<div>
<pre><code data-lang="r"><span>names</span><span>(</span>out<span>$</span>data<span>[[</span><span>1</span><span>]])[</span><span>1</span><span>:</span><span>20</span><span>]</span>
<span>#> [1] &quot;gbifID&quot; &quot;abstract&quot; </span>
<span>#> [3] &quot;accessRights&quot; &quot;accrualMethod&quot; </span>
<span>#> [5] &quot;accrualPeriodicity&quot; &quot;accrualPolicy&quot; </span>
<span>#> [7] &quot;alternative&quot; &quot;audience&quot; </span>
<span>#> [9] &quot;available&quot; &quot;bibliographicCitation&quot;</span>
<span>#> [11] &quot;conformsTo&quot; &quot;contributor&quot; </span>
<span>#> [13] &quot;coverage&quot; &quot;created&quot; </span>
<span>#> [15] &quot;creator&quot; &quot;date&quot; </span>
<span>#> [17] &quot;dateAccepted&quot; &quot;dateCopyrighted&quot; </span>
<span>#> [19] &quot;dateSubmitted&quot; &quot;description&quot;</span>
</code></pre>
</div>
<h2>Future work</h2>
<p>DwC-A files can be very large – This is for sure going to be a pain point for some.<br>
We'll continue to test and refine on big data files.</p>
<h2>Feedback?</h2>
<p>We'd love to know what people think about this package.</p>
<p>Documentation can be better, e.g., there's no vignette yet (but adding<br>
that soon).</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://ropensci.org/blog/technotes/2016/12/23/finch-release"> rOpenSci Blog - R</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/l0cXw-hzIzY"></a></div>