<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/querying-zenodo-org-repository-with-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://behrica.github.com//blog//2017/10/querying-zenodo-org-repository-with-r.html"> Carsten's blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<h1>Zenodo</h1>
<p><a rel="nofollow" href="http://zenodo.org/" target="_blank">Zenodo</a> is a repository which allows everybody to deposit free of charge any type of research output, in all disciplines of science.</p>
<p>EFSA is piloting it’s use for creating a knowledge base on all types of food safety related evidence(data, documents, models).</p>
<p>Zenodo has an API and can be queried using the standard OAI-PMH protocol, which allows to harvest the metadata and all deposits.</p>
<h1>‘oai’ package</h1>
<p>R has a package available to query any OAI-PMH repository, including Zenodo.<br>
It can be installed from CRAN like this:</p>
<figure>
<pre><code data-lang="r"><span>install.packages</span><span>(</span><span>&quot;oai&quot;</span><span>)</span></code></pre>
</figure>
<p>The development version is available on Github at<br>
<a rel="nofollow" href="https://github.com/ropensci/oai" target="_blank">https://github.com/ropensci/oai</a></p>
<p>The libraries I use in this tutorial are:</p>
<figure>
<pre><code data-lang="r"><span>library</span><span>(</span><span>knitr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>tidyverse</span><span>)</span><span>
</span><span>library</span><span>(</span><span>httr</span><span>)</span><span>
</span><span>library</span><span>(</span><span>oai</span><span>)</span><span>
</span><span>library</span><span>(</span><span>xml2</span><span>)</span><span>
</span><span>opts_chunk</span><span>$</span><span>set</span><span>(</span><span>echo</span><span>=</span><span>T</span><span>)</span></code></pre>
</figure>
<h1>Retreive records from Zenodo</h1>
<p>The oai package allows to retrieve all records of a given Zenodo community, in this case the EFSA pilot community.<br>
The following code shows all records of a community with their digital object identifier and the title.</p>
<figure>
<pre><code data-lang="r"><span>record_list</span><span>&lt;-</span><span> </span><span>list_records</span><span>(</span><span>&quot;https://zenodo.org/oai2d&quot;</span><span>,</span><span>metadataPrefix</span><span>=</span><span>&quot;oai_datacite&quot;</span><span>,</span><span>set</span><span>=</span><span>&quot;user-efsa-pilot&quot;</span><span>)</span><span>
</span><span>kable</span><span>(</span><span>record_list</span><span> </span><span>%>%</span><span> </span><span>select</span><span>(</span><span>identifier.3</span><span>,</span><span>title</span><span>))</span></code></pre>
</figure>
<table>
<thead>
<tr>
<th>identifier.3</th>
<th>title</th>
</tr>
</thead>
<tbody>
<tr>
<td>10.5281/zenodo.57132</td>
<td>EFSA Source Attribution Model (EFSA_SAM)</td>
</tr>
<tr>
<td>10.5281/zenodo.57017</td>
<td>PRIMo rev.1 – Pesticide Residue Intake Model</td>
</tr>
<tr>
<td>10.5281/zenodo.56662</td>
<td>Bee-Tool V.1</td>
</tr>
<tr>
<td>10.5281/zenodo.56668</td>
<td>Bee-Tool V.2</td>
</tr>
<tr>
<td>10.5281/zenodo.154720</td>
<td>Egg Pooling Module</td>
</tr>
<tr>
<td>10.5281/zenodo.161300</td>
<td>GMOANALYSIS VERSION 2.1.0 – 10 JULY 2014</td>
</tr>
<tr>
<td>10.5281/zenodo.159163</td>
<td>Pesticide Residues Overview File: PROFile (3.0)</td>
</tr>
<tr>
<td>10.5281/zenodo.154725</td>
<td>Food Additives Intake Model (FAIM) – Version 1.1 – July 2013</td>
</tr>
<tr>
<td>10.5281/zenodo.163080</td>
<td>Modelling continental-scale spread of Schmallenberg virus in Europe</td>
</tr>
<tr>
<td>10.5281/zenodo.57079</td>
<td>C-TSEMM – Cattle TSE Monitoring Model</td>
</tr>
<tr>
<td>10.5281/zenodo.57505</td>
<td>TSEi – TSE Infectivity Model</td>
</tr>
<tr>
<td>10.5281/zenodo.159414</td>
<td>Dietary Exposure Calculator Smoke Flavouring</td>
</tr>
<tr>
<td>10.5281/zenodo.159890</td>
<td>CHIP: Commodity based Hazard Identification Tool</td>
</tr>
<tr>
<td>10.5281/zenodo.56287</td>
<td>PRIMo rev.2 – Pesticide Residue Intake Model</td>
</tr>
<tr>
<td>10.5281/zenodo.56669</td>
<td>Bee-Tool V.3</td>
</tr>
<tr>
<td>10.5281/zenodo.161298</td>
<td>Exposure of operators, workers, residents and bystanders in risk assessment for plant protection products calculator (Version 30MAR2015)</td>
</tr>
<tr>
<td>10.5281/zenodo.163026</td>
<td>Within farms transmission model for Schmallenberg Virus</td>
</tr>
<tr>
<td>10.5281/zenodo.154724</td>
<td>User-friendly interface version of the QMRA model for Salmonella in pigs</td>
</tr>
</tbody>
</table>
<p>Currently there are 18 records available.</p>
<h2>Statistics on keywords</h2>
<h3>Query records from Zenodo</h3>
<p>I was further on interested in the current distribution of keywords each record was tagged with. Zenodo supports two types of keywords. Simple free text keywords and ‘subjects’.<br>
Subjects need to come from a controlled vocabulary, in which each topic has an URI.</p>
<p>EFSA uses the <a rel="nofollow" href="http://browser.agrisemantics.org/gacs/en/" target="_blank">GACS</a> vocabulary, and so a certain topic ‘salmonella’ is represented as URI ‘http://browser.agrisemantics.org/gacs/en/page/C2225’.</p>
<p>The API returns therefore for the subjects only the URI, which is nicely unique and clear but not user friendly as a label. On the URI of each ‘subject’, additional information is available.</p>
<p>The following code retrieves all records and extract all their subjects (which have a Xpath of //d3:subject). The current oai package has some problems with some Zenodo specific metadata,<br>
so I parse the raw XML by hand.</p>
<p>The OIA-PMH standard and the oai::get_records function, allow the client to select, in which metadata format he wants to receive the metadata.<br>
Here I have selected ‘oai-datacite’, because it is recommended from the<br>
Zenodo API <a rel="nofollow" href="https://zenodo.org/dev#harvest-metadata" target="_blank">documenation</a> and should contain <em>all</em> metadata Zenodo supports, while other metadata formats might only support a smaller subset.</p>
<figure>
<pre><code data-lang="r"><span>record_data_xml</span><span> </span><span>&lt;-</span><span> </span><span>get_records</span><span>(</span><span>record_list</span><span>$</span><span>identifier</span><span>,</span><span>url</span><span>=</span><span>&quot;https://zenodo.org/oai2d&quot;</span><span>,</span><span>prefix</span><span>=</span><span>&quot;oai_datacite&quot;</span><span>,</span><span>as</span><span>=</span><span>&quot;raw&quot;</span><span>)</span><span> </span><span>keyword_counts</span><span> </span><span>&lt;-</span><span> </span><span>record_data_xml</span><span> </span><span>%>%</span><span>
</span><span>map</span><span>(</span><span>read_xml</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>map</span><span>(</span><span>xml_find_all</span><span>,</span><span>&quot;//d3:subject&quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>map</span><span>(</span><span>xml_text</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>reduce</span><span>(</span><span>c</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>table</span><span>()</span><span> </span><span>%>%</span><span>
</span><span>tbl_df</span><span>()</span><span>
</span><span>kable</span><span>(</span><span>keyword_counts</span><span> </span><span>%>%</span><span> </span><span>filter</span><span>(</span><span>grepl</span><span>(</span><span>&quot;.*C22.*|^food&quot;</span><span>,</span><span>`.`</span><span>)))</span></code></pre>
</figure>
<table>
<thead>
<tr>
<th>.</th>
<th>n</th>
</tr>
</thead>
<tbody>
<tr>
<td>food additives</td>
<td>1</td>
</tr>
<tr>
<td>food additives intake model</td>
<td>1</td>
</tr>
<tr>
<td>food composition difference testing</td>
<td>1</td>
</tr>
<tr>
<td>http://id.agrisemantics.org/gacs/C22070</td>
<td>2</td>
</tr>
<tr>
<td>http://id.agrisemantics.org/gacs/C22092</td>
<td>1</td>
</tr>
<tr>
<td>http://id.agrisemantics.org/gacs/C2225</td>
<td>3</td>
</tr>
</tbody>
</table>
<p>I use the ‘map’ function from the ‘purrr’ package to apply to every vector in the result (which is first an xml string) a number of transformations:</p>
<ol>
<li>read_xml() – to convert from string to class xml_document</li>
<li>xml_find_all() – to find all xml nodes given by xpath expression</li>
<li>xml_text() – get the text from the xml node</li>
</ol>
<p>Then I combine all this via c() and the reduce() function to obtain a single list of all subjects.</p>
<p>The API returns both types of subjects, the generic keywords and the terms referring to a controlled vocabulary.</p>
<p>The table() command produces then a frequency table for them, of which I show here a subset.<br>
We have in this table entries with an English label, and some with the GACS URI.</p>
<h3>Add human readable label to GACS topics</h3>
<p>To add a human readable label to each GACS URI, I use the GACS API which allows to query information on each topic.<br>
So I call the API for each URI and make a table where each row contains a list of (URI,label). This gets the converted into a table with bind_rows()</p>
<p>I use again the ‘map’ function with an anonymous function, which does the call to the GACS API. GACS uses the (Skomsos)[https://github.com/NatLibFi/Skosmos] software, so has an (API)[http://api.finto.fi/doc/] to query the vocabulary.</p>
<figure>
<pre><code data-lang="r"><span>gacs</span><span> </span><span>&lt;-</span><span> </span><span>keyword_counts</span><span> </span><span>%>%</span><span> </span><span>filter</span><span>(</span><span>grepl</span><span>(</span><span>&quot;*gacs*&quot;</span><span>,</span><span>.</span><span>))</span><span>
</span><span>gacs_label_en</span><span> </span><span>&lt;-</span><span> </span><span>map</span><span>(</span><span>gacs</span><span>$</span><span>`.`</span><span>,</span><span>function</span><span>(</span><span>uri</span><span>)</span><span> </span><span>{</span><span>
</span><span>r</span><span>=</span><span>GET</span><span>(</span><span>&quot;http://browser.agrisemantics.org/rest/v1/gacs/label&quot;</span><span>,</span><span>query</span><span>=</span><span>list</span><span>(</span><span>uri</span><span>=</span><span>uri</span><span>,</span><span>lang</span><span>=</span><span>&quot;en&quot;</span><span>))</span><span>
</span><span>list</span><span>(</span><span>uri</span><span>=</span><span>uri</span><span>,</span><span>label</span><span>=</span><span>content</span><span>(</span><span>r</span><span>)</span><span>$</span><span>prefLabel</span><span>)</span><span>
</span><span>})</span><span> </span><span>%>%</span><span>
</span><span>bind_rows</span><span>()</span><span>
</span><span>kable</span><span>(</span><span>gacs_label_en</span><span>[</span><span>1</span><span>:</span><span>5</span><span>,])</span></code></pre>
</figure>
<table>
<thead>
<tr>
<th>uri</th>
<th>label</th>
</tr>
</thead>
<tbody>
<tr>
<td>http://id.agrisemantics.org/gacs/C10152</td>
<td>Bayesian theory</td>
</tr>
<tr>
<td>http://id.agrisemantics.org/gacs/C10826</td>
<td>commodities</td>
</tr>
<tr>
<td>http://id.agrisemantics.org/gacs/C12237</td>
<td>flavourings</td>
</tr>
<tr>
<td>http://id.agrisemantics.org/gacs/C1263</td>
<td>screening</td>
</tr>
<tr>
<td>http://id.agrisemantics.org/gacs/C14046</td>
<td>emerging infectious diseases</td>
</tr>
</tbody>
</table>
<h2>Distributions of labels in efsa-pilot community</h2>
<p>To get the final table, I join the label-GACS pairs with the former table and do some clean-up with the<br>
functions from tidyr package.</p>
<p>The table is then sorted by frequency and shown on the screen.</p>
<p>As we can see, the most frequent words are ‘risk assessment’ and ‘exposure assessment’, which is no surprise as these is the core of EFSA’s<br>
scientific work.</p>
<figure>
<pre><code data-lang="r"><span>table</span><span> </span><span>&lt;-</span><span> </span><span>left_join</span><span>(</span><span>keyword_counts</span><span>,</span><span>gacs_label_en</span><span>,</span><span>by</span><span>=</span><span>c</span><span>(</span><span>&quot;.&quot;</span><span>=</span><span>&quot;uri&quot;</span><span>))</span><span> </span><span>%>%</span><span> </span><span>replace_na</span><span>(</span><span>list</span><span>(</span><span>label</span><span>=</span><span>&quot;&quot;</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>unite</span><span>(</span><span>&quot;label&quot;</span><span>,</span><span>c</span><span>(</span><span>label</span><span>,</span><span>`.`</span><span>),</span><span>sep</span><span>=</span><span>&quot; - &quot;</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>mutate</span><span>(</span><span>label</span><span> </span><span>=</span><span> </span><span>gsub</span><span>(</span><span>&quot;^ - &quot;</span><span>,</span><span>&quot;&quot;</span><span>,</span><span>label</span><span>))</span><span> </span><span>%>%</span><span>
</span><span>rename</span><span>(</span><span>count</span><span>=</span><span>n</span><span>)</span><span> </span><span>%>%</span><span>
</span><span>arrange</span><span>(</span><span>-</span><span>count</span><span>)</span><span>
</span><span>write.csv</span><span>(</span><span>table</span><span>,</span><span>&quot;keywords.csv&quot;</span><span>,</span><span>row.names</span><span> </span><span>=</span><span> </span><span>F</span><span>)</span><span>
</span><span>knitr</span><span>::</span><span>kable</span><span>(</span><span>table</span><span> </span><span>%>%</span><span> </span><span>slice</span><span>(</span><span>1</span><span>:</span><span>20</span><span>))</span></code></pre>
</figure>
<table>
<thead>
<tr>
<th>label</th>
<th>count</th>
</tr>
</thead>
<tbody>
<tr>
<td>risk assessment – http://id.agrisemantics.org/gacs/C1470</td>
<td>8</td>
</tr>
<tr>
<td>quantitative analysis – http://id.agrisemantics.org/gacs/C603</td>
<td>7</td>
</tr>
<tr>
<td>exposure assessment – http://id.agrisemantics.org/gacs/C29232</td>
<td>6</td>
</tr>
<tr>
<td>population – http://id.agrisemantics.org/gacs/C2955</td>
<td>5</td>
</tr>
<tr>
<td>prion diseases – http://id.agrisemantics.org/gacs/C18728</td>
<td>4</td>
</tr>
<tr>
<td>pesticides – http://id.agrisemantics.org/gacs/C284</td>
<td>4</td>
</tr>
<tr>
<td>Apoidea – http://id.agrisemantics.org/gacs/C1932</td>
<td>3</td>
</tr>
<tr>
<td>Salmonella – http://id.agrisemantics.org/gacs/C2225</td>
<td>3</td>
</tr>
<tr>
<td>pesticide residues – http://id.agrisemantics.org/gacs/C3009</td>
<td>3</td>
</tr>
<tr>
<td>linear models – http://id.agrisemantics.org/gacs/C3504</td>
<td>3</td>
</tr>
<tr>
<td>model validation – http://id.agrisemantics.org/gacs/C4332</td>
<td>3</td>
</tr>
<tr>
<td>time – http://id.agrisemantics.org/gacs/C4525</td>
<td>3</td>
</tr>
<tr>
<td>pollinators – http://id.agrisemantics.org/gacs/C5325</td>
<td>3</td>
</tr>
<tr>
<td>decision support systems – http://id.agrisemantics.org/gacs/C8154</td>
<td>3</td>
</tr>
<tr>
<td>acute risk assesment</td>
<td>2</td>
</tr>
<tr>
<td>chronic risk assesment</td>
<td>2</td>
</tr>
<tr>
<td>Epidemiology</td>
<td>2</td>
</tr>
<tr>
<td>exposure assessment</td>
<td>2</td>
</tr>
<tr>
<td>bovine spongiform encephalopathy – http://id.agrisemantics.org/gacs/C14182</td>
<td>2</td>
</tr>
<tr>
<td>calculation – http://id.agrisemantics.org/gacs/C15337</td>
<td>2</td>
</tr>
</tbody>
</table>
<p>To monitor regularly this distribution can help in keeping the list of all keywords clean and eventually propose additional subjects to the GACS vocabulary.</p>
<h1>Session info</h1>
<figure>
<pre><code data-lang="r"><span>sessionInfo</span><span>()</span></code></pre>
</figure>
<figure>
<pre><code data-lang="text">## R version 3.4.1 (2017-06-30)
## Platform: x86_64-pc-linux-gnu (64-bit)
## Running under: Arch Linux
## 
## Matrix products: default
## BLAS: /usr/lib/libblas_nehalemp-r0.2.19.so
## LAPACK: /usr/lib/liblapack.so.3.7.1
## 
## locale:
##  [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C              
##  [3] LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
##  [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8   
##  [7] LC_PAPER=en_US.UTF-8       LC_NAME=C                 
##  [9] LC_ADDRESS=C               LC_TELEPHONE=C            
## [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       
## 
## attached base packages:
## [1] stats     graphics  grDevices utils     datasets  base     
## 
## other attached packages:
##  [1] bindrcpp_0.2      xml2_1.1.1        oai_0.2.2        
##  [4] httr_1.3.1        dplyr_0.7.3.9000  purrr_0.2.3      
##  [7] readr_1.1.1       tidyr_0.7.1       tibble_1.3.4.9001
## [10] ggplot2_2.2.1     tidyverse_1.1.1   knitr_1.17       
## 
## loaded via a namespace (and not attached):
##  [1] Rcpp_0.12.12      highr_0.6         cellranger_1.1.0 
##  [4] pillar_0.0.0.9000 compiler_3.4.1    plyr_1.8.4       
##  [7] bindr_0.1         methods_3.4.1     forcats_0.2.0    
## [10] tools_3.4.1       digest_0.6.12     lubridate_1.6.0  
## [13] jsonlite_1.5      evaluate_0.10.1   nlme_3.1-131     
## [16] gtable_0.2.0      lattice_0.20-35   pkgconfig_2.0.1  
## [19] rlang_0.1.2       psych_1.7.8       parallel_3.4.1   
## [22] haven_1.1.0       stringr_1.2.0     hms_0.3          
## [25] tidyselect_0.2.0  grid_3.4.1        glue_1.1.1       
## [28] R6_2.2.2          readxl_1.0.0      foreign_0.8-69   
## [31] modelr_0.1.1      reshape2_1.4.2    magrittr_1.5     
## [34] servr_0.7         scales_0.5.0      rvest_0.3.2      
## [37] assertthat_0.2.0  mnormt_1.5-5      colorspace_1.3-2 
## [40] httpuv_1.3.5      stringi_1.1.5     lazyeval_0.2.0   
## [43] munsell_0.4.3     broom_0.4.2</code></pre>
</figure>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://behrica.github.com//blog//2017/10/querying-zenodo-org-repository-with-r.html"> Carsten's blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/hh3K7uMIw8g" height="1"></a></div>