<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/merge-a-list-of-datasets-together/"></div></div><div><a data-url="https://www.r-bloggers.com/merge-a-list-of-datasets-together/" data-counturl="https://www.r-bloggers.com/merge-a-list-of-datasets-together/" data-count="horizontal" data-text="Merge a list of datasets together" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://b-rodrigues.github.com/2016/07/30/merge-a-list-of-datasets-together"> Econometrics and Free Software</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a target="_blank" href="http://www.brodrigues.co/2016/07/26/read-a-lot-of-datasets-at-once-with-r">Last week</a> I showed how to read a lot of datasets at once with R, and this week I’ll continue from there and show a very simple function that uses this list of read datasets and merges them all together.</p>
<p>First we’ll use <code>read_list()</code> to read all the datasets at once (for more details read <a target="_blank" href="http://www.brodrigues.co/2016/07/26/read-a-lot-of-datasets-at-once-with-r">last week’s post</a>):</p>
<pre><code>library(&quot;readr&quot;)
library(&quot;tibble&quot;)

data_files &lt;- list.files(pattern = &quot;.csv&quot;)

print(data_files)</code></pre>
<pre><code>## [1] &quot;data_1.csv&quot; &quot;data_2.csv&quot; &quot;data_3.csv&quot;</code></pre>
<pre><code>list_of_data_sets &lt;- read_list(data_files, read_csv)

glimpse(list_of_data_sets)</code></pre>
<pre><code>## List of 3
##  $ data_1:Classes 'tbl_df', 'tbl' and 'data.frame':  19 obs. of  3 variables:
##   ..$ col1: chr [1:19] &quot;0,018930679&quot; &quot;0,8748013128&quot; &quot;0,1025635934&quot; &quot;0,6246140983&quot; ...
##   ..$ col2: chr [1:19] &quot;0,0377725807&quot; &quot;0,5959457638&quot; &quot;0,4429121533&quot; &quot;0,558387159&quot; ...
##   ..$ col3: chr [1:19] &quot;0,6241767189&quot; &quot;0,031324594&quot; &quot;0,2238059868&quot; &quot;0,2773350732&quot; ...
##  $ data_2:Classes 'tbl_df', 'tbl' and 'data.frame':  19 obs. of  3 variables:
##   ..$ col1: chr [1:19] &quot;0,9098418493&quot; &quot;0,1127788509&quot; &quot;0,5818891392&quot; &quot;0,1011773532&quot; ...
##   ..$ col2: chr [1:19] &quot;0,7455905887&quot; &quot;0,4015039612&quot; &quot;0,6625796605&quot; &quot;0,029955339&quot; ...
##   ..$ col3: chr [1:19] &quot;0,327232932&quot; &quot;0,2784035673&quot; &quot;0,8092386735&quot; &quot;0,1216045306&quot; ...
##  $ data_3:Classes 'tbl_df', 'tbl' and 'data.frame':  19 obs. of  3 variables:
##   ..$ col1: chr [1:19] &quot;0,9236124896&quot; &quot;0,6303271761&quot; &quot;0,6413583054&quot; &quot;0,5573887416&quot; ...
##   ..$ col2: chr [1:19] &quot;0,2114708388&quot; &quot;0,6984538266&quot; &quot;0,0469865249&quot; &quot;0,9271510226&quot; ...
##   ..$ col3: chr [1:19] &quot;0,4941919971&quot; &quot;0,7391538511&quot; &quot;0,3876723797&quot; &quot;0,2815014394&quot; ...</code></pre>
<p>You see that all these datasets have the same column names. We can now merge them using this simple function:</p>
<pre><code>multi_join &lt;- function(list_of_loaded_data, join_func, ...){

    require(&quot;dplyr&quot;)

    output &lt;- Reduce(function(x, y) {join_func(x, y, ...)}, list_of_loaded_data)

    return(output)
}</code></pre>
<p>This function uses <code>Reduce()</code>. <code>Reduce()</code> is a very important function that can be found in all functional programming languages. What does <code>Reduce()</code> do? Let’s take a look at the following example:</p>
<pre><code>Reduce(`+`, c(1, 2, 3, 4, 5))</code></pre>
<pre><code>## [1] 15</code></pre>
<p><code>Reduce()</code> has several arguments, but you need to specify at least two: a function, here <code>+</code> and a list, here <code>c(1, 2, 3, 4, 5)</code>. The next code block shows what <code>Reduce()</code> basically does:</p>
<pre><code>0 + c(1, 2, 3, 4, 5)
0 + 1 + c(2, 3, 4, 5)
0 + 1 + 2 + c(3, 4, 5)
0 + 1 + 2 + 3 + c(4, 5)
0 + 1 + 2 + 3 + 4 + c(5)
0 + 1 + 2 + 3 + 4 + 5</code></pre>
<p><code>0</code> had to be added as in “init”. You can also specify this “init” to <code>Reduce()</code>:</p>
<pre><code>Reduce(`+`, c(1, 2, 3, 4, 5), init = 20)</code></pre>
<pre><code>## [1] 35</code></pre>
<p>So what <code>multi_join()</code> does, is the same operation as in the example above, but where the function is a user supplied join or merge function, and the list of datasets is the one read with <code>read_list()</code>.</p>
<p>Let’s see what happens when we use <code>multi_join()</code> on our list:</p>
<pre><code>merged_data &lt;- multi_join(list_of_data_sets, full_join)</code></pre>
<pre><code>class(merged_data)</code></pre>
<pre><code>## [1] &quot;tbl_df&quot;     &quot;tbl&quot;        &quot;data.frame&quot;</code></pre>
<pre><code>glimpse(merged_data)</code></pre>
<pre><code>## Observations: 57
## Variables: 3
## $ col1 &lt;chr> &quot;0,018930679&quot;, &quot;0,8748013128&quot;, &quot;0,1025635934&quot;, &quot;0,6246140...
## $ col2 &lt;chr> &quot;0,0377725807&quot;, &quot;0,5959457638&quot;, &quot;0,4429121533&quot;, &quot;0,558387...
## $ col3 &lt;chr> &quot;0,6241767189&quot;, &quot;0,031324594&quot;, &quot;0,2238059868&quot;, &quot;0,2773350...</code></pre>
<p>You should make sure that all the data frames have the same column names but you can also join data frames with different column names if you give the argument <code>by</code> to the join function. This is possible thanks to <code>...</code> that allows you to pass further argument to <code>join_func()</code>.</p>
<p>This function was inspired by the one found on the blog <a target="_blank" href="http://novicemetrics.blogspot.lu/2011/04/merging-multiple-data-files-into-one.html">Coffee and Econometrics in the Morning</a>.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/merge-a-list-of-datasets-together/"></div></div><div><a data-url="https://www.r-bloggers.com/merge-a-list-of-datasets-together/" data-counturl="https://www.r-bloggers.com/merge-a-list-of-datasets-together/" data-count="vertical" data-text="Merge a list of datasets together" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://b-rodrigues.github.com/2016/07/30/merge-a-list-of-datasets-together"> Econometrics and Free Software</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/wj-qU0WpE78"></a></div>