<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/darwinazing-biodiversity-data-in-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://vijaybarve.wordpress.com/2018/03/14/darwinazing-biodiversity-data-in-r/"> Vijay Barve</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>“Darwin Core (DwC) is a standard maintained by the Darwin Core maintenance group. It includes a glossary of terms (in other contexts these might be called properties, elements, fields, columns, attributes, or concepts) intended to facilitate the sharing of information about biological diversity by providing identifiers, labels, and definitions. Darwin Core is primarily based on taxa, their occurrence in nature as documented by observations, specimens, samples, and related information.” <a rel="nofollow" href="https://github.com/tdwg/dwc" target="_blank">Darwin Core website</a></p>
<p>DwC is an evolving community-developed biodiversity data standard (<a rel="nofollow" href="http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0029715" target="_blank">Wieczorek et al. 2012</a>). In simple words, it’s a list and definitions of common biodiversity data terms, ~200 of them, for more details see <a rel="nofollow" href="http://rs.tdwg.org/dwc/terms/" target="_blank">DwC quick reference guide</a>. Today, hundreds of millions of biodiversity records from around the world are published in DwC format and aggregated into various portals (e.g. GBIF, vertNet, iDigBio). Nonetheless, data publishers still struggle with the essential step of mapping fields in their data to the terms in Darwin Core (<a rel="nofollow" href="https://biss.pensoft.net/article/20486/instance/3707784/" target="_blank">Wieczorek et al. 2017a</a>). Doing so requires a good understanding of both the data set and Darwin Core (<a rel="nofollow" href="https://biss.pensoft.net/article/20280/element/4/3719095/" target="_blank">Wieczorek et al. 2017b</a>).</p>
<h2><a rel="nofollow" href="https://github.com/rstats-gsoc/gsoc2018/wiki/Darwinazing-biodiversity-data-in-R#related-work" target="_blank"></a>Related work</h2>
<p>The remarkable <a rel="nofollow" href="http://wiki.datakurator.org/wiki/Kurator" target="_blank">Kurator</a> project creates biodiversity data quality workflows, and via it web interface (<a rel="nofollow" href="http://wiki.datakurator.org/wiki/Kurator-Web_User_Documentation" target="_blank">Kurator-Web</a>)– data quality-control is highly accessible. Thanks to this invaluable project, we have easy access to different lookup tables, that aggregates rare and highly valuable data regarding DwC vocabularies. For example in the <a rel="nofollow" href="https://github.com/kurator-org/kurator-validation/blob/master/packages/kurator_dwca/data/vocabularies/darwin_cloud.txt" target="_blank">Darwin Cloud</a> table, knowledge is being accumulated about variations in DwC field names. Fully utilizing these precious data in the R environment can significantly enhance our ability to address more biodiversity data quality issues.</p>
<h2><a rel="nofollow" href="https://github.com/rstats-gsoc/gsoc2018/wiki/Darwinazing-biodiversity-data-in-R#details-of-your-coding-project" target="_blank"></a>Details of project</h2>
<h3><a rel="nofollow" href="https://github.com/rstats-gsoc/gsoc2018/wiki/Darwinazing-biodiversity-data-in-R#darwinizer-workflow-in-r" target="_blank"></a>Darwinizer workflow in R:</h3>
<p>While DwC was adopted by most biodiversity data publishers, it implementation is somewhat incomplete. Imposing controlled vocabulary on millions of records is a complex and daunting task. For example, there are inconsistencies regarding field names between different data publishers. The <a rel="nofollow" href="https://github.com/kurator-org/kurator-validation/wiki/CSV-File-Darwinizer" target="_blank">CSV File Darwinizer</a> Kurator workflow standardizes field names to the DwC standard name, thanks to the <a rel="nofollow" href="https://github.com/kurator-org/kurator-validation/blob/master/packages/kurator_dwca/data/vocabularies/darwin_cloud.txt" target="_blank">Darwin Cloud</a> lookup file. By generating this workflow in R, we can easily input a wider range of data from different publishers. This module needs to work on various data files downloaded from different biodiversity portals, and handle all of them.</p>
<h3><a rel="nofollow" href="https://github.com/rstats-gsoc/gsoc2018/wiki/Darwinazing-biodiversity-data-in-R#data-checks" target="_blank"></a>Data checks:</h3>
<p>Data checks must be specifically tailored around the structure of the data, in our case- the DwC standard. Under this module we will address three major data checks collections:</p>
<ul>
<li><a rel="nofollow" href="https://github.com/tdwg/bdq/tree/master/tg2" target="_blank">TDWG core suite of tests and assertions</a>: Tests and rules generating assertions at the record-level are more fundamental than the tools or workflows that will be based on them. Ideally, this core suite of data quality checks need to be embrace by all data publishers (as a standard), and hopefully in the long term, this will be the case. However, in the short term, since constructing many of them in R is rather feasible we plan to achieve that. Furthermore, embracing this standard will improve our ability to properly manage data checks. In <a rel="nofollow" href="https://github.com/AshwinAgrawal16/BiodiversitydatacleaningGSoC17" target="_blank">Ashwin</a>and <a rel="nofollow" href="https://github.com/thiloshon/Biodiversity" target="_blank">Thiloshon</a> last year GSoC projects various data checks have been developed, while some adjustment and further development is still required.</li>
<li>Imposing controlled vocabulary on key data fields: Using <a rel="nofollow" href="https://github.com/kurator-org/kurator-validation/tree/master/packages/kurator_dwca/data/vocabularies" target="_blank">Kurator’s vocabulary data</a>, different DwC standardization procedures can be addressed. The challenge will be to assess and prioritize the development of these procedures.</li>
<li>New frontiers: Enriching DwC data (i.e. accurately joining external data) can greatly boost data checks capacity and diversity. For example, joining species trait data, or retrieving climatic data for each record opens variety of check capabilities. Here we need to screen for robust data enrichment procedures in R and design exciting data checks around them.</li>
</ul>
<h3><a rel="nofollow" href="https://github.com/rstats-gsoc/gsoc2018/wiki/Darwinazing-biodiversity-data-in-R#r-based-kurator-actors" target="_blank"></a>R based Kurator actors:</h3>
<p>Following a communication with the Kurator development team, we will explore the development of R actors (functions), that hopefully, will be seemingly integrated into the kurator infrastructure. Handling some Java and Python code will be required.</p>
<h2><a rel="nofollow" href="https://github.com/rstats-gsoc/gsoc2018/wiki/Darwinazing-biodiversity-data-in-R#expected-impact" target="_blank"></a>Expected impact</h2>
<p>Improving the quality of biodiversity research, in some measure, is based on improving user-level data cleaning tools and skills. Adopting a more comprehensive approach for incorporating data cleaning as part of data analysis will not only improve the quality of biodiversity data, but will impose a more appropriate usage of such data.</p>
<p>If you are a full time University student and would like to participate in <a rel="nofollow" href="https://vijaybarve.wordpress.com/2018/03/13/google-summer-of-code-2018/" target="_blank">Google Summer of Code 2018</a> helping us build this, the project idea is listed <a rel="nofollow" href="https://github.com/rstats-gsoc/gsoc2018/wiki/Darwinazing-biodiversity-data-in-R" target="_blank">here</a> which has further details on how to start.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://vijaybarve.wordpress.com/2018/03/14/darwinazing-biodiversity-data-in-r/"> Vijay Barve</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/-H-Dgh787iw" height="1"></a></div>