<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/exploring-the-diversity-of-life-using-rvest-and-the-catalog-of-life/"></div></div><div><a data-url="http://www.r-bloggers.com/exploring-the-diversity-of-life-using-rvest-and-the-catalog-of-life/" data-counturl="http://www.r-bloggers.com/exploring-the-diversity-of-life-using-rvest-and-the-catalog-of-life/" data-count="horizontal" data-text="Exploring the diversity of Life using Rvest and the Catalog of Life" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://biologyforfun.wordpress.com/2016/07/18/exploring-the-diversity-of-life-using-rvest-and-the-catalog-of-life/"> R – biologyforfun</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>I am writing the general introduction for my thesis and wanted to have a nice illustration of the diversity of Arthropods compared to other phyla (my work focus on Arthropods so this is a nice motivation). As the literature I have had access so far use pie charts to graphically represent these diversities and knowing that <a target="_blank" href="https://en.wikipedia.org/wiki/Misleading_graph#Pie_chart">pie chart are bad</a>, I decided to create my own illustration.</p>
<p>Fortunately I came across the <a target="_blank" href="http://www.catalogueoflife.org/">Catalogue of Life</a> website which provide (among other things) an overview of the number of species in each phylum. So I decided to try and have a go at directly importing the data from the website into R using the <a target="_blank" href="https://cran.r-project.org/web/packages/rvest/index.html">rvest</a> package.</p>
<p>Let’s go:</p>
<p> </p>
<pre>#load the packages
library(rvest)
library(ggplot2)
library(scales)#for comma separator in ggplot2 axis

#read the data
col&lt;-read_html(&quot;http://www.catalogueoflife.org/col/info/totals&quot;)
col%>%
  html_table(header=TRUE)->sp_list  
sp_list&lt;-sp_list[[1]]
#some minor data re-formatting
#re-format the data frame keeping only animals, plants and
#fungi
sp_list&lt;-sp_list[c(3:37,90:94,98:105),-4]
#add a kingdom column
sp_list$kingdom&lt;-rep(c(&quot;Animalia&quot;,&quot;Fungi&quot;,&quot;Plantae&quot;),times=c(35,5,8))
#remove the nasty commas and turn into numeric
sp_list[,2]&lt;-as.numeric(gsub(&quot;,&quot;,&quot;&quot;,sp_list[,2]))
sp_list[,3]&lt;-as.numeric(gsub(&quot;,&quot;,&quot;&quot;,sp_list[,3]))
names(sp_list)[2:3]&lt;-c(&quot;Nb_Species_Col&quot;,&quot;Nb_Species&quot;)</pre>
<p>Now we are read to make the first plot</p>
<pre>ggplot(sp_list,aes(x=Taxon,y=Nb_Species,fill=kingdom))+
 geom_bar(stat=&quot;identity&quot;)+
 coord_flip()+
 scale_fill_discrete(name=&quot;Kingdom&quot;)+
 labs(y=&quot;Number of Species&quot;,x=&quot;&quot;,title=&quot;The diversity of life&quot;)</pre>
<p> </p>
<p><img alt="Div1" class="wp-image-695" src="http://i1.wp.com/biologyforfun.files.wordpress.com/2016/07/div1.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>This is a bit overwhelming, half of the phyla have less than 1000 species so let’s make a second graph only with the phyla comprising more than 1000 species. And just to make things nicer I sort the phyla by the number of species:</p>
<pre>subs&lt;-subset(sp_list,Nb_Species>1000)
subs$Taxon&lt;-factor(subs$Taxon,levels=subs$Taxon[order(subs$Nb_Species)])
ggplot(subs,aes(x=Taxon,y=Nb_Species,fill=kingdom))+
 geom_bar(stat=&quot;identity&quot;)+
 theme(panel.border=element_rect(linetype=&quot;dashed&quot;,color=&quot;black&quot;,fill=NA),
 panel.background=element_rect(fill=&quot;white&quot;),
 panel.grid.major.x=element_line(linetype=&quot;dotted&quot;,color=&quot;black&quot;))+
 coord_flip()+
 scale_fill_discrete(name=&quot;Kingdom&quot;)+
 labs(y=&quot;Number of Species&quot;,x=&quot;&quot;,
 title=&quot;The diversity of multicellular organisms from the Catalog of Life&quot;)+
 scale_y_continuous(expand=c(0,0),limits=c(0,1250000),labels=comma)</pre>
<p> </p>
<p><img alt="Div2" class="wp-image-694" src="http://i1.wp.com/biologyforfun.files.wordpress.com/2016/07/div2.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>That’s it for a first exploration of the powers of rvest, this was actually super easy I expected to have to spend much more time trying to decipher xml code, but rvest seems to know its way around …</p>
<p>This graph is also a nice reminder that most of the described multicellular species out there are small crawling beetles and that we still know alarmingly very little about their ecology and status of threat. As a comparison all the vertebrates (all birds, mammals, reptiles, fishes and some other taxa) are within the Chordata and having a total of 50,000 described species. An even more sobering thought is the fact that the total number of described species is <a target="_blank" href="http://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.1001127">only a fraction</a> of what is left undescribed.</p>
<p>Filed under: <a target="_blank" href="https://biologyforfun.wordpress.com/category/biological-stuff/">Biological Stuff</a>, <a target="_blank" href="https://biologyforfun.wordpress.com/category/r-and-stat/">R and Stat</a> Tagged: <a target="_blank" href="https://biologyforfun.wordpress.com/tag/biodiversity/">Biodiversity</a>, <a target="_blank" href="https://biologyforfun.wordpress.com/tag/ecology/">ecology</a>, <a target="_blank" href="https://biologyforfun.wordpress.com/tag/r/">R</a>, <a target="_blank" href="https://biologyforfun.wordpress.com/tag/rvest/">rvest</a> <a target="_blank" rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/biologyforfun.wordpress.com/652/"><img alt="" src="http://feeds.wordpress.com/1.0/comments/biologyforfun.wordpress.com/652/" border="0"></a> <a target="_blank" rel="nofollow" href="http://feeds.wordpress.com/1.0/godelicious/biologyforfun.wordpress.com/652/"><img alt="" src="http://feeds.wordpress.com/1.0/delicious/biologyforfun.wordpress.com/652/" border="0"></a> <a target="_blank" rel="nofollow" href="http://feeds.wordpress.com/1.0/gofacebook/biologyforfun.wordpress.com/652/"><img alt="" src="http://feeds.wordpress.com/1.0/facebook/biologyforfun.wordpress.com/652/" border="0"></a> <a target="_blank" rel="nofollow" href="http://feeds.wordpress.com/1.0/gotwitter/biologyforfun.wordpress.com/652/"><img alt="" src="http://feeds.wordpress.com/1.0/twitter/biologyforfun.wordpress.com/652/" border="0"></a> <a target="_blank" rel="nofollow" href="http://feeds.wordpress.com/1.0/gostumble/biologyforfun.wordpress.com/652/"><img alt="" src="http://feeds.wordpress.com/1.0/stumble/biologyforfun.wordpress.com/652/" border="0"></a> <a target="_blank" rel="nofollow" href="http://feeds.wordpress.com/1.0/godigg/biologyforfun.wordpress.com/652/"><img alt="" src="http://feeds.wordpress.com/1.0/digg/biologyforfun.wordpress.com/652/" border="0"></a> <a target="_blank" rel="nofollow" href="http://feeds.wordpress.com/1.0/goreddit/biologyforfun.wordpress.com/652/"><img alt="" src="http://feeds.wordpress.com/1.0/reddit/biologyforfun.wordpress.com/652/" border="0"></a> <img alt="" src="http://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" border="0" data-recalc-dims="1"></p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/exploring-the-diversity-of-life-using-rvest-and-the-catalog-of-life/"></div></div><div><a data-url="http://www.r-bloggers.com/exploring-the-diversity-of-life-using-rvest-and-the-catalog-of-life/" data-counturl="http://www.r-bloggers.com/exploring-the-diversity-of-life-using-rvest-and-the-catalog-of-life/" data-count="vertical" data-text="Exploring the diversity of Life using Rvest and the Catalog of Life" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://biologyforfun.wordpress.com/2016/07/18/exploring-the-diversity-of-life-using-rvest-and-the-catalog-of-life/"> R – biologyforfun</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/QJEGgNRV4k8"></a></div>