<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/data-acquisition-in-r-14/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://ikashnitsky.github.io/2017/data-acquisition-one/"> Ilya Kashnitsky</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>R is an incredible tool for reproducible research. In the present series of blog posts I want to show how one can easily acquire data within an R session, documenting every step in a fully reproducible way. There are numerous data acquisition options for R users. Of course, I do not attempt to show all the data possibilities and tend to focus mostly on demographic data. If your prime interest lies outside human population statistics, it’s worth checking the amazing <a rel="nofollow" href="https://github.com/ropensci/opendata" target="_blank">Open Data Task View</a>.</p>
<p>The series consists of four posts:</p>
<ul>
<li><strong>Loading prepared datasets</strong></li>
<li>Accessing popular statistical databases</li>
<li>Demographic data sources</li>
<li>Getting spatial data</li>
</ul>
<p>For each of the data acquisition options I provide a small visualization use case.</p>
<h1>Biult-in datasets</h1>
<p>For illustration purposes, many R packages include data samples. Base R comes with a <code>datasets</code> package that offers a wide range of simple, sometimes very famous, datasets. Quite a detailed list of built-in datasets from various packages is <a rel="nofollow" href="https://vincentarelbundock.github.io/Rdatasets/datasets.html" target="_blank">maintained by Vincent Arel-Bundock</a>.</p>
<p>The nice feature of the datasets form <code>datasets</code> package is that they are “always there”. The unique names of the datasets may be referred as the objects from Global Environment. Let’s have a look at a beautiful small dataset calls <code>swiss</code> – Swiss Fertility and Socioeconomic Indicators (1888) Data. I am going to check visually the difference in fertility based of rurality and domination of Catholic population.</p>
<div>
<pre><code>library(tidyverse)

swiss %>% 
        ggplot(aes(x = Agriculture, y = Fertility, 
                   color = Catholic > 50))+
        geom_point()+
        stat_ellipse()+
        theme_minimal(base_family = &quot;mono&quot;)
</code></pre>
</div>
<p><a rel="nofollow" href="https://i0.wp.com/ikashnitsky.github.io/images/171017/swiss.png?ssl=1" target="_blank"><img alt="fig1" src="https://i0.wp.com/ikashnitsky.github.io/images/171017/swiss.png?w=456&ssl=1" data-recalc-dims="1"></a></p>
<h1>Gapminder</h1>
<p>Some packages are created specifically to disseminate datasets in a ready to use format. One of the nice examples is a package <code>gapminder</code> that contains a neat dataset widely used by Hans Rosling in his Gapminder project.</p>
<div>
<pre><code>library(tidyverse)
library(gapminder)

gapminder %>% 
        ggplot(aes(x = year, y = lifeExp, 
                   color = continent))+
        geom_jitter(size = 1, alpha = .2, width = .75)+
        stat_summary(geom = &quot;path&quot;, fun.y = mean, size = 1)+
        theme_minimal(base_family = &quot;mono&quot;)
</code></pre>
</div>
<p><a rel="nofollow" href="https://i2.wp.com/ikashnitsky.github.io/images/171017/gapminder.png?ssl=1" target="_blank"><img alt="fig2" src="https://i2.wp.com/ikashnitsky.github.io/images/171017/gapminder.png?w=456&ssl=1" data-recalc-dims="1"></a></p>
<h1>Grab a dataset by URL</h1>
<p>If a dataset is hosted online and has a direct link to the file, it can be easily imported into the R session just specifying the URL. For illustration, I will access <code>Galton</code> dataset from <code>HistData</code> package using a direct link from <a rel="nofollow" href="https://vincentarelbundock.github.io/Rdatasets/datasets.html" target="_blank">Vincent Arel-Bundock’s list</a>.</p>
<div>
<pre><code>library(tidyverse)

galton &lt;- read_csv(&quot;https://raw.githubusercontent.com/vincentarelbundock/Rdatasets/master/csv/HistData/Galton.csv&quot;)

galton %>% 
        ggplot(aes(x = father, y = height))+
        geom_point(alpha = .2)+
        stat_smooth(method = &quot;lm&quot;)+
        theme_minimal(base_family = &quot;mono&quot;)
</code></pre>
</div>
<p><a rel="nofollow" href="https://i1.wp.com/ikashnitsky.github.io/images/171017/galton.png?ssl=1" target="_blank"><img alt="fig3" src="https://i1.wp.com/ikashnitsky.github.io/images/171017/galton.png?w=456&ssl=1" data-recalc-dims="1"></a></p>
<h1>Download and unzip an archive</h1>
<p>Quite often datasets are stored in archived from. With R it is very simple to download and unzip the desired data archives. As an example, I will download <a rel="nofollow" href="https://catalog.data.gov/dataset/historical-new-york-city-crime-data-ad47e" target="_blank">Historical New York City Crime Data</a> provided by the Government of the Sate of New York and hosted at data.gov portal. The logic of the process is: first, we create a directory for the unzipped data; second, we download the archive; finally, unzip the archive and read the data.</p>
<div>
<pre><code>library(tidyverse)
library(readxl)

# create a directory for the unzipped data
ifelse(!dir.exists(&quot;unzipped&quot;), dir.create(&quot;unzipped&quot;), &quot;Directory already exists&quot;)

# specify the URL of the archive
url_zip &lt;- &quot;http://www.nyc.gov/html/nypd/downloads/zip/analysis_and_planning/citywide_historical_crime_data_archive.zip&quot;

# storing the archive in a temporary file
f &lt;- tempfile()
download.file(url_zip, destfile = f)
unzip(f, exdir = &quot;unzipped/.&quot;)
</code></pre>
</div>
<p>If the zipped file is rather big and we don’t want to download it again the next time we run the code, it might be useful to keep the archived data.</p>
<div>
<pre><code># if we want to keep the .zip file
path_unzip &lt;- &quot;unzipped/data_archive.zip&quot;
ifelse(!file.exists(path_unzip), 
       download.file(url_zip, path_unzip, mode=&quot;wb&quot;), 
       'file alredy exists')
unzip(path_unzip, exdir = &quot;unzipped/.&quot;)

</code></pre>
</div>
<p>Finally, let’s read and plot some of the downloaded data.</p>
<div>
<pre><code>murder &lt;- read_xls(&quot;unzipped/Web Data 2010-2011/Seven Major Felony Offenses 2000 - 2011.xls&quot;,
                   sheet = 1, range = &quot;A5:M13&quot;) %>% 
        filter(OFFENSE %>% substr(1, 6) == &quot;MURDER&quot;) %>% 
        gather(&quot;year&quot;, &quot;value&quot;, 2:13) %>% 
        mutate(year = year %>% as.numeric())

murder %>% 
        ggplot(aes(year, value))+
        geom_point()+
        stat_smooth(method = &quot;lm&quot;)+
        theme_minimal(base_family = &quot;mono&quot;)+
        labs(title = &quot;Murders in New York&quot;)
</code></pre>
</div>
<p><a rel="nofollow" href="https://i2.wp.com/ikashnitsky.github.io/images/171017/new-york.png?ssl=1" target="_blank"><img alt="fig4" src="https://i2.wp.com/ikashnitsky.github.io/images/171017/new-york.png?w=456&ssl=1" data-recalc-dims="1"></a></p>
<h1>Figshare</h1>
<p>In Academia it is becoming more and more popular to store the datasets accompanying papers in the specialized repositories. Figshare is one of the most popular free repositories. There is an R package <code>rfigshare</code> to access the datasets from this portal. As an example I will grab <a rel="nofollow" href="https://dx.doi.org/10.6084/m9.figshare.3394735.v2" target="_blank">the dataset on ice-hockey playes height</a> that I assembled manually for <a rel="nofollow" href="https://ikashnitsky.github.io/2017/ice-hockey-players-height/" target="_blank">my blog post</a>. Please note that at the first run the package will ask to enter your Figshare login details to access API – a web page will be opened in browser.</p>
<p>There is a search function <code>fs_search</code>, though my experience shows that it is easier to search for a dataset in a browser and then use the id of a file to download it. The function <code>fs_download</code> turns an id number into a direct URL to download the file.</p>
<div>
<pre><code>library(tidyverse)
library(rfigshare)

url &lt;- fs_download(article_id = &quot;3394735&quot;)

hockey &lt;- read_csv(url)

hockey %>% 
        ggplot(aes(x = year, y = height))+
        geom_jitter(size = 2, color = &quot;#35978f&quot;, alpha = .1, width = .25)+
        stat_smooth(method = &quot;lm&quot;, size = 1)+
        ylab(&quot;height, cm&quot;)+
        xlab(&quot;year of competition&quot;)+
        scale_x_continuous(breaks = seq(2005, 2015, 5), labels = seq(2005, 2015, 5))+
        theme_minimal(base_family = &quot;mono&quot;)
</code></pre>
</div>
<p><a rel="nofollow" href="https://i0.wp.com/ikashnitsky.github.io/images/171017/ice-hockey.png?ssl=1" target="_blank"><img alt="fig5" src="https://i0.wp.com/ikashnitsky.github.io/images/171017/ice-hockey.png?w=456&ssl=1" data-recalc-dims="1"></a></p>
<h1>Full reproducibility</h1>
<p>All the code chunks can be found in <a rel="nofollow" href="https://gist.github.com/ikashnitsky/e1d93a51fe5e2b5ba770096060bacd8a" target="_blank">this gist</a>.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://ikashnitsky.github.io/2017/data-acquisition-one/"> Ilya Kashnitsky</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/BpxkEEtIcTk" height="1"></a></div>