<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/data-acquisition-in-r-24/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://ikashnitsky.github.io/2017/data-acquisition-two/"> Ilya Kashnitsky</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>R is an incredible tool for reproducible research. In the present series of blog posts I want to show how one can easily acquire data within an R session, documenting every step in a fully reproducible way. There are numerous data acquisition options for R users. Of course, I do not attempt to show all the data possibilities and tend to focus mostly on demographic data. If your prime interest lies outside human population statistics, it’s worth checking the amazing <a rel="nofollow" href="https://github.com/ropensci/opendata" target="_blank">Open Data Task View</a>.</p>
<p>The series consists of four posts:</p>
<ul>
<li><a rel="nofollow" href="https://ikashnitsky.github.io/2017/data-acquisition-one/" target="_blank">Loading prepared datasets</a></li>
<li><strong>Accessing popular statistical databases</strong></li>
<li>Demographic data sources</li>
<li>Getting spatial data</li>
</ul>
<p>For each of the data acquisition options I provide a small visualization use case.</p>
<h1>Eurostat</h1>
<p>The package <code>eurostat</code> has a function <code>search_eurostat</code> to search for the relevant datasets. Though, sadly enough, this function does not provide the codes of all the datasets that has the expression of interest in the title. For example, the search on the expression <code>life expectancy</code> produces an output with just 2 results, which does not make any sense. Thus, the best strategy is to go to <a rel="nofollow" href="http://ec.europa.eu/eurostat/data/database" target="_blank">Eurostat website</a>, find the needed dataset code, and fetch the desired dataset by its code. Note that there is a <a rel="nofollow" href="http://ec.europa.eu/eurostat/web/regions/data/database" target="_blank">separate database for subregional level indicators</a>.</p>
<p>I am going to download life expectancy estimates for European countries; the dataset code is <code>demo_mlexpec</code>.</p>
<div>
<div>
<pre><code>library(tidyverse) 
library(lubridate)
library(eurostat) 

# download the selected dataset
e0 &lt;- get_eurostat(&quot;demo_mlexpec&quot;)
</code></pre>
</div>
</div>
<p>It can take a while because the dataset is quite big (0.4m obs). If the automated procedure does not work, one can download the data manually via the <a rel="nofollow" href="http://ec.europa.eu/eurostat/estat-navtree-portlet-prod/BulkDownloadListing" target="_blank">Bulk Download Service</a> of Eurostat.</p>
<p>Let’s have a look at the remaining life expectancy at age 65, the most common conventional age at retirement, in some European countries, separately for males, females, and total population. Some data preparation steps are needed. First, we only need the life expectancy estimates for those aged 65. Next, we don’t need total population, only males and females separately. Finally, let’s select just a bunch of countries: Germany, France, Italy, Russia, Spain, the UK.</p>
<div>
<div>
<pre><code>e0 %>% 
        filter(! sex == &quot;T&quot;,
               age == &quot;Y65&quot;, 
               geo %in% c(&quot;DE&quot;, &quot;FR&quot;, &quot;IT&quot;, &quot;RU&quot;, &quot;ES&quot;, &quot;UK&quot;)) %>% 
        ggplot(aes(x = time %>% year(), y = values, color = sex))+
        geom_path()+
        facet_wrap(~ geo, ncol = 3)+
        labs(y = &quot;Life expectancy at age 65&quot;, x = NULL)+
        theme_minimal()
</code></pre>
</div>
</div>
<p><a rel="nofollow" href="https://i1.wp.com/ikashnitsky.github.io/images/171107/eurostat.png?ssl=1" target="_blank"><img alt="fig1" src="https://i1.wp.com/ikashnitsky.github.io/images/171107/eurostat.png?w=456&ssl=1" data-recalc-dims="1"></a></p>
<h1>World Bank</h1>
<p>There are several packages that provide an API to World Bank data. Probably, the most elaborated one is a fairly recent <a rel="nofollow" href="https://cran.r-project.org/web/packages/wbstats/README.html" target="_blank"><code>wbstats</code></a>. Its <code>wbsearch</code> function really does great job searching through the database for the relevant datasets. For example, <code>wbsearch(&quot;fertility&quot;)</code> produces a dataframe of 339 entries with the codes and names of the relevant indicators.</p>
<div>
<div>
<pre><code>library(tidyverse) 
library(wbstats)

# search for a dataset of interest
wbsearch(&quot;fertility&quot;) %>% head
</code></pre>
</div>
</div>
<table>
<thead>
<tr>
<th> </th>
<th>indicatorID</th>
<th>indicator</th>
</tr>
</thead>
<tbody>
<tr>
<td>2479</td>
<td>SP.DYN.WFRT.Q5</td>
<td>Total wanted fertility rate (births per woman): Q5 (highest)</td>
</tr>
<tr>
<td>2480</td>
<td>SP.DYN.WFRT.Q4</td>
<td>Total wanted fertility rate (births per woman): Q4</td>
</tr>
<tr>
<td>2481</td>
<td>SP.DYN.WFRT.Q3</td>
<td>Total wanted fertility rate (births per woman): Q3</td>
</tr>
<tr>
<td>2482</td>
<td>SP.DYN.WFRT.Q2</td>
<td>Total wanted fertility rate (births per woman): Q2</td>
</tr>
<tr>
<td>2483</td>
<td>SP.DYN.WFRT.Q1</td>
<td>Total wanted fertility rate (births per woman): Q1 (lowest)</td>
</tr>
<tr>
<td>2484</td>
<td>SP.DYN.WFRT</td>
<td>Wanted fertility rate (births per woman)</td>
</tr>
</tbody>
</table>
<p>Let’s have a look at the indicator <code>Lifetime risk of maternal death (%)</code> (code <code>SH.MMR.RISK.ZS</code>). World Bank provides a variety of country groupings. One of the curious groupings divides countries based on the advance over the Demographic Transition path. Below I plot our selected indicator for (1) the countries that have passed the Demographic Transition, (2) the countries that haven’t yet experienced demographic dividend, and (3) the whole World.</p>
<div>
<div>
<pre><code># fetch the selected dataset
df_wb &lt;- wb(indicator = &quot;SH.MMR.RISK.ZS&quot;, startdate = 2000, enddate = 2015)

# have look at the data for one year
df_wb %>% filter(date == 2015) %>% View

df_wb %>% 
        filter(iso2c %in% c(&quot;V4&quot;, &quot;V1&quot;, &quot;1W&quot;)) %>% 
        ggplot(aes(x = date %>% as.numeric(), y = value, color = country))+
        geom_path(size = 1)+
        scale_color_brewer(NULL, palette = &quot;Dark2&quot;)+
        labs(x = NULL, y = NULL, title = &quot;Lifetime risk of maternal death (%)&quot;)+
        theme_minimal()+
        theme(panel.grid.minor = element_blank(),
              legend.position = c(.8, .9))

</code></pre>
</div>
</div>
<p><a rel="nofollow" href="https://i1.wp.com/ikashnitsky.github.io/images/171107/worldbank.png?ssl=1" target="_blank"><img alt="fig2" src="https://i1.wp.com/ikashnitsky.github.io/images/171107/worldbank.png?w=456&ssl=1" data-recalc-dims="1"></a></p>
<h1>OECD</h1>
<p>Organization for Economic Cooperation and Development provides detailed economic and demographic data on the member countries. There is an R package <code>OECD</code> that streamlines the use of their data in R. The function <code>search_dataset</code> works nicely to browse the available datasets by keywords. Then <code>get_dataset</code> would fetch the chosen dataset. In the example below I grab the data on the duration of unemployment and then plot the data for the male population of EU16, EU28 and the US as heatmaps.</p>
<div>
<div>
<pre><code>library(tidyverse) 
library(viridis)
library(OECD)

# search by keyword
search_dataset(&quot;unemployment&quot;) %>% View

# download the selected dataset
df_oecd &lt;- get_dataset(&quot;AVD_DUR&quot;)

# turn variable names to lowercase
names(df_oecd) &lt;- names(df_oecd) %>% tolower()

df_oecd %>% 
        filter(country %in% c(&quot;EU16&quot;, &quot;EU28&quot;, &quot;USA&quot;), sex == &quot;MEN&quot;, ! age == &quot;1524&quot;) %>% 
        ggplot(aes(obstime, age, fill = obsvalue))+
        geom_tile()+
        scale_fill_viridis(&quot;Months&quot;, option = &quot;B&quot;)+
        scale_x_discrete(breaks = seq(1970, 2015, 5) %>% paste)+
        facet_wrap(~ country, ncol = 1)+
        labs(x = NULL, y = &quot;Age groups&quot;, 
             title = &quot;Average duration of unemployment in months, males&quot;)+
        theme_minimal()
</code></pre>
</div>
</div>
<p><a rel="nofollow" href="https://i1.wp.com/ikashnitsky.github.io/images/171107/oecd.png?ssl=1" target="_blank"><img alt="fig3" src="https://i1.wp.com/ikashnitsky.github.io/images/171107/oecd.png?w=456&ssl=1" data-recalc-dims="1"></a></p>
<h1>WID</h1>
<p>[World Wealth and Income Database][wdi] is a harmonized dataset on income and wealth inequality. The developers of the database provide an R package to get their data, which is only <a rel="nofollow" href="https://github.com/WIDworld/wid-r-tool" target="_blank">available from github</a> so far.</p>
<div>
<div>
<pre><code>library(tidyverse) 

#install.packages(&quot;devtools&quot;)
devtools::install_github(&quot;WIDworld/wid-r-tool&quot;)
library(wid)
</code></pre>
</div>
</div>
<p>The function to acquire data is <code>download_wid()</code>. To specify the arguments, one would have to consult help pages of the package and select desired datasets.</p>
<div>
<div>
<pre><code>?wid_series_type
?wid_concepts
</code></pre>
</div>
</div>
<p>The following nice example is adapted from <a rel="nofollow" href="https://github.com/WIDworld/wid-r-tool/raw/master/inst/doc/wid-demo.pdf" target="_blank">the package vignette</a>. It shows the share of wealth that was owned by the richest 1% and 10% of population in France and Great Britain.</p>
<div>
<div>
<pre><code>df_wid &lt;- download_wid(
        indicators = &quot;shweal&quot;, # Shares of personal wealth
        areas = c(&quot;FR&quot;, &quot;GB&quot;), # In France an Italy
        perc = c(&quot;p90p100&quot;, &quot;p99p100&quot;) # Top 1% and top 10%
)


df_wid %>% 
        ggplot(aes(x = year, y = value, color = country)) +
        geom_path()+
        labs(title = &quot;Top 1% and top 10% personal wealth shares in France and Great Britain&quot;,
             y = &quot;top share&quot;)+
        facet_wrap(~ percentile)+
        theme_minimal()

</code></pre>
</div>
</div>
<p><a rel="nofollow" href="https://i0.wp.com/ikashnitsky.github.io/images/171107/wid.png?ssl=1" target="_blank"><img alt="fig4" src="https://i0.wp.com/ikashnitsky.github.io/images/171107/wid.png?w=456&ssl=1" data-recalc-dims="1"></a></p>
<h1>Full reproducibility</h1>
<p>All the code chunks together can be found in <a rel="nofollow" href="https://gist.github.com/ikashnitsky/2362ce308f47ff38c3da556384acd20f" target="_blank">this gist</a>.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://ikashnitsky.github.io/2017/data-acquisition-two/"> Ilya Kashnitsky</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/vWNTTpQtvGw" height="1"></a></div>