<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/how-to-reshape-data-in-r-tidyr-vs-reshape2/"></div></div><div><a data-url="http://www.r-bloggers.com/how-to-reshape-data-in-r-tidyr-vs-reshape2/" data-counturl="http://www.r-bloggers.com/how-to-reshape-data-in-r-tidyr-vs-reshape2/" data-count="horizontal" data-text="How to reshape data in R: tidyr vs reshape2" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.milanor.net/blog/reshape-data-r-tidyr-vs-reshape2/"> MilanoR</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p></p>
<p>We often find ourselves tidying and reshaping data. Here we consider the two packages <em>tidyr</em> and <em>reshape2</em>, our aim is to see where their purposes overlap and where they differ by comparing the functions <code>gather()</code>, <code>separate()</code> and <code>spread()</code>, from <em>tidyr</em>, with the functions <code>melt()</code>, <code>colsplit()</code> and <code>dcast()</code>, from <em>reshape2</em>.</p>
<div>
<h2>Data tidying</h2>
<p>Data tidying is the operation of transforming data into a clear and simple form that makes it easy to work with. “Tidy data” represent the information from a dataset as data frames where each row is an observation and each column contains the values of a variable (i.e. an attribute of what we are observing). Compare the two data frames below (cf.<span>Wickham (2014)</span>) to get an idea of the differences: <code>example.tidy</code> is the tidy version of <code>example.messy</code>, the same information is organized in two different ways.</p>
<pre>example.messy</pre>

<pre>##              treatmenta treatmentb
## John Smith           NA          2
## Jane Doe             16         11
## Mary Johnson          3          1</pre>

<pre>example.tidy</pre>

<pre>##           name        trt result
## 1   John Smith treatmenta     NA
## 2     Jane Doe treatmenta     16
## 3 Mary Johnson treatmenta      3
## 4   John Smith treatmentb      2
## 5     Jane Doe treatmentb     11
## 6 Mary Johnson treatmentb      1</pre>

<div>
<h3>From the wide to the long format: gather() vs melt()</h3>
<p>We now begin by seeing in action how we can bring data from the “wide” to the “long” format.</p>
<p>Let’s start loading the packages we need:</p>
<pre>library(tidyr)
library(reshape2)</pre>
<p>and some data (from <a target="_blank" href="http://blog.rstudio.org/2014/07/22/introducing-tidyr/">RStudio Blog – Introducing tidyr</a>): we have measurements of how much time people spend on their phones, measured at two locations (work and home), at two times. Each person has been randomly assigned to either treatment or control.</p>
<pre>set.seed(10)
messy &lt;- data.frame(id = 1:4,
                    trt = sample(rep(c('control', 'treatment'), each = 2)),
                    work.T1 = runif(4),
                    home.T1 = runif(4),
                    work.T2 = runif(4),
                    home.T2 = runif(4))
messy</pre>

<pre>##   id       trt    work.T1   home.T1   work.T2    home.T2
## 1  1 treatment 0.08513597 0.6158293 0.1135090 0.05190332
## 2  2   control 0.22543662 0.4296715 0.5959253 0.26417767
## 3  3 treatment 0.27453052 0.6516557 0.3580500 0.39879073
## 4  4   control 0.27230507 0.5677378 0.4288094 0.83613414</pre>
<p>Our first step is to put the data in the tidy format, to do that we use <em>tidyr</em>’s functions <code>gather()</code> and <code>separate()</code>. Following Wickham’s tidy data definition, this data frame is not tidy because some variable values are in the column names. We bring this messy data frame from the wide to the long format by using the <code>gather()</code> function (give a look at <a target="_blank" href="http://seananderson.ca/2013/10/19/reshape.html">Sean C. Anderson – An Introduction to reshape2</a> to get an idea of the wide/long format). We want to gather all the columns, except for the <em>id</em> and <em>trt</em> ones, in two columns <em>key</em> and <em>value</em>:</p>
<pre>gathered.messy &lt;- gather(messy, key, value, -id, -trt)
head(gathered.messy)</pre>

<pre>##   id       trt     key      value
## 1  1 treatment work.T1 0.08513597
## 2  2   control work.T1 0.22543662
## 3  3 treatment work.T1 0.27453052
## 4  4   control work.T1 0.27230507
## 5  1 treatment home.T1 0.61582931
## 6  2   control home.T1 0.42967153</pre>
<p>Note that in <code>gather()</code> we used bare variable names to specify the names of the <em>key</em>, <em>value</em>, <em>id</em> and <em>trt</em> columns.</p>
<p>We can get the same result with the <code>melt()</code> function from <em>reshape2</em>:</p>
<pre>molten.messy &lt;- melt(messy, 
                     variable.name = &quot;key&quot;,
                     value.names = &quot;value&quot;,
                     id.vars = c(&quot;id&quot;, &quot;trt&quot;))
head(molten.messy)</pre>

<pre>##   id       trt     key      value
## 1  1 treatment work.T1 0.08513597
## 2  2   control work.T1 0.22543662
## 3  3 treatment work.T1 0.27453052
## 4  4   control work.T1 0.27230507
## 5  1 treatment home.T1 0.61582931
## 6  2   control home.T1 0.42967153</pre>
<p>We now compare the two functions by running them over the data without any further parameter and see what happen:</p>
<pre>head(gather(messy))</pre>

<pre>## Warning: attributes are not identical across measure variables; they will
## be dropped</pre>

<pre>##   key     value
## 1  id         1
## 2  id         2
## 3  id         3
## 4  id         4
## 5 trt treatment
## 6 trt   control</pre>

<pre>head(melt(messy))</pre>

<pre>## Using trt as id variables</pre>

<pre>##         trt variable      value
## 1 treatment       id 1.00000000
## 2   control       id 2.00000000
## 3 treatment       id 3.00000000
## 4   control       id 4.00000000
## 5 treatment  work.T1 0.08513597
## 6   control  work.T1 0.22543662</pre>
<p>We see a different behaviour: <code>gather()</code> has brought <code>messy</code> into a long data format with a warning by treating all columns as variable, while <code>melt()</code> has treated <em>trt</em> as an “id variables”. Id columns are the columns that contain the identifier of the observation that is represented as a row in our data set. Indeed, if <code>melt()</code> does not receive any id.variables specification, then it will use the factor or character columns as id variables. <code>gather()</code> requires the columns that needs to be treated as ids, all the other columns are going to be used as key-value pairs.</p>
<p>Despite those last different results, we have seen that the two functions can be used to perform the exactly same operations on data frames, and only on data frames! Indeed, <code>gather()</code> cannot handle matrices or arrays, while <code>melt()</code> can as shown below.</p>
<pre>set.seed(3)
M &lt;- matrix(rnorm(6), ncol = 3)
dimnames(M) &lt;- list(letters[1:2], letters[1:3])

melt(M)</pre>

<pre>##   Var1 Var2       value
## 1    a    a -0.96193342
## 2    b    a -0.29252572
## 3    a    b  0.25878822
## 4    b    b -1.15213189
## 5    a    c  0.19578283
## 6    b    c  0.03012394</pre>

<pre>gather(M)</pre>

<pre>## Error in UseMethod(&quot;gather_&quot;): no applicable method for 'gather_' applied to an object of class &quot;c('matrix', 'double', 'numeric')&quot;</pre>
</div>
<div>
<h3>Split a column: separate() vs colsplit()</h3>
<p>Our next step is to split the column <em>key</em> into two different columns in order to separate the <em>location</em> and <em>time</em> variables and obtain a tidy data frame:</p>
<pre>tidy &lt;- separate(gathered.messy,
                 key, into = c(&quot;location&quot;, &quot;time&quot;), sep = &quot;\.&quot;) 
res.tidy &lt;- cbind(molten.messy[1:2], 
                  colsplit(molten.messy[, 3], &quot;\.&quot;, c(&quot;location&quot;, &quot;time&quot;)),
                  molten.messy[4])

head(tidy)</pre>

<pre>##   id       trt location time      value
## 1  1 treatment     work   T1 0.08513597
## 2  2   control     work   T1 0.22543662
## 3  3 treatment     work   T1 0.27453052
## 4  4   control     work   T1 0.27230507
## 5  1 treatment     home   T1 0.61582931
## 6  2   control     home   T1 0.42967153</pre>

<pre>head(res.tidy)</pre>

<pre>##   id       trt location time      value
## 1  1 treatment     work   T1 0.08513597
## 2  2   control     work   T1 0.22543662
## 3  3 treatment     work   T1 0.27453052
## 4  4   control     work   T1 0.27230507
## 5  1 treatment     home   T1 0.61582931
## 6  2   control     home   T1 0.42967153</pre>
<p>Again, the result is the same but we need a workaround: because <code>colsplit()</code> operates only on a single column we use<code>cbind()</code> to insert the new two columns in the data frame. <code>separate()</code> performs all the operation at once reducing the possibility of making mistakes.</p>
</div>
<div>
<h3>From the long to the wide format: spread() vs dcast()</h3>
<p>Finally, we compare <code>spread()</code> with <code>dcast()</code> using the data frame example for the <code>spread()</code> documentation itself. Briefly,<code>spread()</code> is complementary to <code>gather()</code> and brings data from the long to the wide format.</p>
<pre>set.seed(14)
stocks &lt;- data.frame(time = as.Date('2009-01-01') + 0:9,
                     X = rnorm(10, 0, 1),
                     Y = rnorm(10, 0, 2),
                     Z = rnorm(10, 0, 4))
stocksm &lt;- gather(stocks, stock, price, -time)
spread.stock &lt;- spread(stocksm, stock, price)
head(spread.stock)</pre>

<pre>##         time           X          Y          Z
## 1 2009-01-01 -0.66184983 -0.7656438 -5.0672590
## 2 2009-01-02  1.71895416  0.5988432 -0.7943331
## 3 2009-01-03  2.12166699  1.3484795  0.5554631
## 4 2009-01-04  1.49715368 -0.5856326 -1.1173440
## 5 2009-01-05 -0.03614058  0.9761067  2.8356777
## 6 2009-01-06  1.23194518  1.7656036 -3.0664418</pre>

<pre>cast.stock &lt;- dcast(stocksm, formula = time ~ stock, value.var = &quot;price&quot;)
head(cast.stock)</pre>

<pre>##         time           X          Y          Z
## 1 2009-01-01 -0.66184983 -0.7656438 -5.0672590
## 2 2009-01-02  1.71895416  0.5988432 -0.7943331
## 3 2009-01-03  2.12166699  1.3484795  0.5554631
## 4 2009-01-04  1.49715368 -0.5856326 -1.1173440
## 5 2009-01-05 -0.03614058  0.9761067  2.8356777
## 6 2009-01-06  1.23194518  1.7656036 -3.0664418</pre>
<p>Again, the same result produced by <code>spread()</code> can be obtained using <code>dcast()</code> by specifying the correct <code>formula</code>.</p>
<p>In the next session, we are going to modify the <code>formula</code> parameter in order to perform some data aggregation and compare further the two packages.</p>
</div>
</div>
<div>
<h2>Data aggregation</h2>
<p>Up to now we made <em>reshape2</em> following <em>tidyr</em>, showing that everything you can do with <em>tidyr</em> can be achieved by <em>reshape2</em>, too, at the price of a some workarounds. As we now go on with our simple example we will get out of the purposes of <em>tidyr</em> and have no more functions available for our needs. Now we have a tidy data set – one observation per row and one variable per column – to work with. We show some aggregations that are possible with <code>dcast()</code> using the <code>tips </code>data frame from <em>reshape2</em>. <code>Tips</code> contains the information one waiter recorded about each tip he received over a period of a few months working in one restaurant.</p>
<pre>head(tips)</pre>

<pre>##   total_bill  tip    sex smoker day   time size
## 1      16.99 1.01 Female     No Sun Dinner    2
## 2      10.34 1.66   Male     No Sun Dinner    3
## 3      21.01 3.50   Male     No Sun Dinner    3
## 4      23.68 3.31   Male     No Sun Dinner    2
## 5      24.59 3.61 Female     No Sun Dinner    4
## 6      25.29 4.71   Male     No Sun Dinner    4</pre>

<pre>m.tips &lt;- melt(tips)</pre>

<pre>## Using sex, smoker, day, time as id variables</pre>

<pre>head(m.tips)</pre>

<pre>##      sex smoker day   time   variable value
## 1 Female     No Sun Dinner total_bill 16.99
## 2   Male     No Sun Dinner total_bill 10.34
## 3   Male     No Sun Dinner total_bill 21.01
## 4   Male     No Sun Dinner total_bill 23.68
## 5 Female     No Sun Dinner total_bill 24.59
## 6   Male     No Sun Dinner total_bill 25.29</pre>
<p>We use <code>dcast()</code> to get information on the average total bill, tip and group size per day and time:</p>
<pre>dcast(m.tips, day+time ~ variable, mean)</pre>

<pre>##    day   time total_bill      tip     size
## 1  Fri Dinner   19.66333 2.940000 2.166667
## 2  Fri  Lunch   12.84571 2.382857 2.000000
## 3  Sat Dinner   20.44138 2.993103 2.517241
## 4  Sun Dinner   21.41000 3.255132 2.842105
## 5 Thur Dinner   18.78000 3.000000 2.000000
## 6 Thur  Lunch   17.66475 2.767705 2.459016</pre>
<p>Averages per smoker or not in the group.</p>
<pre>dcast(m.tips, smoker ~ variable, mean)</pre>

<pre>##   smoker total_bill      tip     size
## 1     No   19.18828 2.991854 2.668874
## 2    Yes   20.75634 3.008710 2.408602</pre>
<p>There is no function in the <em>tidyr</em> package that allows us to perform a similar operation, the reason is that <em>tidyr</em> is designed only for data tidying and not for data reshaping.</p>
</div>
<div>
<h2>Conclusions</h2>
<p>At the beginning we have seen <em>tidyr</em> and <em>reshape2</em> functions performing the same operations, therefore, suggesting that the two packages are similar, if not equal in what they do; lately, we have seen that <em>reshape2</em>’s functions can do data aggregation that is not possible with <em>tidyr</em>. Indeed, <em>tidyr</em>’s aim is data tidying while <em>reshape2</em> has the wider purpose of data reshaping and aggregating. It follows that <em>tidyr</em> syntax is easier to understand and to work with, but its functionalities are limited. Therefore, we use <em>tidyr</em> <code>gather()</code> and <code>separate()</code> functions to quickly tidy our data and <em>reshape2</em><code>dcast()</code> to aggregate them.</p>
<div>
<h3>Further readings:</h3>
<ul>
<li><a target="_blank" href="http://blog.rstudio.org/2014/07/22/introducing-tidyr/">RStudio Blog – Introducing tidyr</a></li>
<li><a target="_blank" href="http://www.r-bloggers.com/reshape-and-aggregate-data-with-the-r-package-reshape2/">R-blogger – Reshape and aggregate data with the R package reshape2</a>.</li>
<li>reshape2 – R Documentation</li>
<li>tidyr – R Documentation</li>
</ul>
<div>
<h3>References:</h3>
<p>Wickham, Hadley. 2014. “Tidy data.” <em>Journal of Statistical Software</em> 59 (10).</p>
</div>
</div>
</div>
<div></div>
</div>
<p>The post <a target="_blank" rel="nofollow" href="http://www.milanor.net/blog/reshape-data-r-tidyr-vs-reshape2/">How to reshape data in R: tidyr vs reshape2</a> appeared first on <a target="_blank" rel="nofollow" href="http://www.milanor.net/blog">MilanoR</a>.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/how-to-reshape-data-in-r-tidyr-vs-reshape2/"></div></div><div><a data-url="http://www.r-bloggers.com/how-to-reshape-data-in-r-tidyr-vs-reshape2/" data-counturl="http://www.r-bloggers.com/how-to-reshape-data-in-r-tidyr-vs-reshape2/" data-count="vertical" data-text="How to reshape data in R: tidyr vs reshape2" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.milanor.net/blog/reshape-data-r-tidyr-vs-reshape2/"> MilanoR</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/hMT_Pz_AgTs"></a></div>