<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/arbitrary-data-transforms-using-cdata/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2017/11/arbitrary-data-transforms-using-cdata/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>We have been writing a lot on higher-order data transforms lately:</p>
<ul>
<li><a rel="nofollow" href="http://winvector.github.io/FluidData/RowsAndColumns.html" target="_blank">Coordinatized Data: A Fluid Data Specification</a></li>
<li><a rel="nofollow" href="http://winvector.github.io/FluidData/DataWranglingAtScale.html" target="_blank">Data Wrangling at Scale</a></li>
<li><a rel="nofollow" href="http://winvector.github.io/FluidData/FluidData.html" target="_blank">Fluid Data</a></li>
<li><a rel="nofollow" href="http://www.win-vector.com/blog/2017/10/big-data-transforms/" target="_blank">Big Data Transforms</a>.</li>
</ul>
<p><a rel="nofollow" href="https://github.com/WinVector/cdata" target="_blank"><img border="0" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/11/cdata-1.png?resize=300%2C300" alt="Cdata" title="cdata.png" data-recalc-dims="1"></a></p>
<p>What I want to do now is &quot;write a bit more, so I finally feel I have been concise.&quot;</p>
<p><span></span></p>
<p>The <a rel="nofollow" href="https://winvector.github.io/cdata/" target="_blank"><code>cdata</code></a> <a rel="nofollow" href="https://www.r-project.org/" target="_blank"><code>R</code></a> package supplies general data transform operators.</p>
<ul>
<li>The whole system is based on two primitives or operators <a rel="nofollow" href="https://winvector.github.io/cdata/reference/moveValuesToRowsD.html" target="_blank"><code>cdata::moveValuesToRowsD()</code></a> and <a rel="nofollow" href="https://winvector.github.io/cdata/reference/moveValuesToColumnsD.html" target="_blank"><code>cdata::moveValuesToColumnsD()</code></a>.</li>
<li>These operators have pivot, un-pivot, one-hot encode, transpose, moving multiple rows and columns, and many other transforms as simple special cases.</li>
<li>It is easy to write many different operations in terms of the <code>cdata</code> primitives.</li>
<li>These operators can work-in memory or at big data scale (with databases and Apache Spark; for big data we use the <a rel="nofollow" href="https://winvector.github.io/cdata/reference/moveValuesToRowsN.html" target="_blank"><code>cdata::moveValuesToRowsN()</code></a> and <a rel="nofollow" href="https://winvector.github.io/cdata/reference/moveValuesToColumnsN.html" target="_blank"><code>cdata::moveValuesToColumnsN()</code></a> variants).</li>
<li>The transforms are controlled by a control table that itself is a diagram of or picture of the transform.</li>
</ul>
<p>We will end with a quick example, centered on pivoting/un-pivoting values to/from more than one column at the same time.</p>
<p>Suppose we had some sales data supplied as the following table:</p>
<table>
<thead>
<tr>
<th align="left">SalesPerson</th>
<th align="left">Period</th>
<th align="right">BookingsWest</th>
<th align="right">BookingsEast</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">a</td>
<td align="left">2017Q1</td>
<td align="right">100</td>
<td align="right">175</td>
</tr>
<tr>
<td align="left">a</td>
<td align="left">2017Q2</td>
<td align="right">110</td>
<td align="right">180</td>
</tr>
<tr>
<td align="left">b</td>
<td align="left">2017Q1</td>
<td align="right">250</td>
<td align="right">0</td>
</tr>
<tr>
<td align="left">b</td>
<td align="left">2017Q2</td>
<td align="right">245</td>
<td align="right">0</td>
</tr>
</tbody>
</table>
<p>Suppose we are interested in adding a derived column: which region the salesperson made most of their bookings in.</p>
<div>
<pre><code><span>library</span>(<span>&quot;cdata&quot;</span>)</code></pre>
</div>
<pre><code>## Loading required package: wrapr</code></pre>
<div>
<pre><code><span>library</span>(<span>&quot;seplyr&quot;</span>)</code></pre>
</div>
<div>
<pre><code>d &lt;-<span> </span>d  %.>%<span> </span>
<span> </span>dplyr::<span>mutate</span>(., <span>BestRegion =</span> <span>ifelse</span>(BookingsWest ><span> </span>BookingsEast, 
                                       <span>&quot;West&quot;</span>,
                                       <span>ifelse</span>(BookingsEast ><span> </span>BookingsWest, 
                                              <span>&quot;East&quot;</span>, 
                                              <span>&quot;Both&quot;</span>)))</code></pre>
</div>
<p>Our notional goal is (as part of a larger data processing plan) to reformat the data a thin/tall table or a <a rel="nofollow" href="https://en.wikipedia.org/wiki/Semantic_triple" target="_blank">RDF-triple</a> like form. Further suppose we wanted to copy the derived column into every row of the transformed table (perhaps to make some other step involving this value easy).</p>
<p>We can use <a rel="nofollow" href="https://winvector.github.io/cdata/reference/moveValuesToRowsD.html" target="_blank"><code>cdata::moveValuesToRowsD()</code></a> to do this quickly and easily.</p>
<p>First we design what is called a transform control table.</p>
<div>
<pre><code>cT1 &lt;-<span> </span><span>data.frame</span>(<span>Region =</span> <span>c</span>(<span>&quot;West&quot;</span>, <span>&quot;East&quot;</span>),
                  <span>Bookings =</span> <span>c</span>(<span>&quot;BookingsWest&quot;</span>, <span>&quot;BookingsEast&quot;</span>),
                  <span>BestRegion =</span> <span>c</span>(<span>&quot;BestRegion&quot;</span>, <span>&quot;BestRegion&quot;</span>),
                  <span>stringsAsFactors =</span> <span>FALSE</span>)
<span>print</span>(cT1)</code></pre>
</div>
<pre><code>##   Region     Bookings BestRegion
## 1   West BookingsWest BestRegion
## 2   East BookingsEast BestRegion</code></pre>
<p>In a control table:</p>
<ul>
<li>The column names specify new columns that will be formed by <code>cdata::moveValuesToRowsD()</code>.</li>
<li>The values specify where to take values from.</li>
</ul>
<p>This control table is called &quot;non trivial&quot; as it does not correspond to a simple pivot/un-pivot (those tables all have two columns). The control table is a picture of of the mapping we want to perform.</p>
<p>An interesting fact is <code>cdata::moveValuesToColumnsD(cT1, cT1, keyColumns = NULL)</code> is a picture of the control table as a one-row table (and this one row table can be mapped back to the original control table by <code>cdata::moveValuesToRowsD()</code>, these two operators work roughly as inverses of each other; though <code>cdata::moveValuesToRowsD()</code> operates on rows and <a rel="nofollow" href="https://winvector.github.io/cdata/reference/moveValuesToColumnsD.html" target="_blank"><code>cdata::moveValuesToColumnsD()</code></a> operates on groups of rows specified by the keying columns).</p>
<p>The mnemonic is:</p>
<ul>
<li><code>cdata::moveValuesToColumnsD()</code> converts arbitrary grouped blocks of rows that look like the control table into many columns.</li>
<li><code>cdata::moveValuesToRowsD()</code> converts each row into row blocks that have the same shape as the control table.</li>
</ul>
<p>Because pivot and un-pivot are fairly common needs <code>cdata</code> also supplies functions that pre-populate the controls tables for these operations (<a rel="nofollow" href="https://winvector.github.io/cdata/reference/buildPivotControlTableD.html" target="_blank"><code>buildPivotControlTableD()</code></a> and <a rel="nofollow" href="https://winvector.github.io/cdata/reference/buildUnPivotControlTable.html" target="_blank"><code>buildUnPivotControlTable()</code></a>).</p>
<p>To design any transform you draw out the control table and then apply one of these operators (you can pretty much move from any block structure to any block structure by chaining two or more of these steps).</p>
<p>We can now use the control table to supply the same transform for each row.</p>
<div>
<pre><code>d  %.>%<span> </span>
<span> </span>dplyr::<span>mutate</span>(., 
                <span>Quarter =</span> <span>substr</span>(Period,<span>5</span>,<span>6</span>),
                <span>Year =</span> <span>as.numeric</span>(<span>substr</span>(Period,<span>1</span>,<span>4</span>)))  %.>%<span> </span>
<span> </span>dplyr::<span>select</span>(., -Period)  %.>%<span> </span>
<span> </span><span>moveValuesToRowsD</span>(., 
                    <span>controlTable =</span> cT1, 
                    <span>columnsToCopy =</span> <span>c</span>(<span>'SalesPerson'</span>, 
                                      <span>'Year'</span>, 
                                      <span>'Quarter'</span>)) %.>%<span> </span>
<span> </span><span>arrange_se</span>(., <span>c</span>(<span>'SalesPerson'</span>, <span>'Year'</span>, <span>'Quarter'</span>, <span>'Region'</span>))  %.>%<span> </span>
<span> </span>knitr::<span>kable</span>(.)  </code></pre>
</div>
<table>
<thead>
<tr>
<th align="left">SalesPerson</th>
<th align="right">Year</th>
<th align="left">Quarter</th>
<th align="left">Region</th>
<th align="right">Bookings</th>
<th align="left">BestRegion</th>
</tr>
</thead>
<tbody>
<tr>
<td align="left">a</td>
<td align="right">2017</td>
<td align="left">Q1</td>
<td align="left">East</td>
<td align="right">175</td>
<td align="left">East</td>
</tr>
<tr>
<td align="left">a</td>
<td align="right">2017</td>
<td align="left">Q1</td>
<td align="left">West</td>
<td align="right">100</td>
<td align="left">East</td>
</tr>
<tr>
<td align="left">a</td>
<td align="right">2017</td>
<td align="left">Q2</td>
<td align="left">East</td>
<td align="right">180</td>
<td align="left">East</td>
</tr>
<tr>
<td align="left">a</td>
<td align="right">2017</td>
<td align="left">Q2</td>
<td align="left">West</td>
<td align="right">110</td>
<td align="left">East</td>
</tr>
<tr>
<td align="left">b</td>
<td align="right">2017</td>
<td align="left">Q1</td>
<td align="left">East</td>
<td align="right">0</td>
<td align="left">West</td>
</tr>
<tr>
<td align="left">b</td>
<td align="right">2017</td>
<td align="left">Q1</td>
<td align="left">West</td>
<td align="right">250</td>
<td align="left">West</td>
</tr>
<tr>
<td align="left">b</td>
<td align="right">2017</td>
<td align="left">Q2</td>
<td align="left">East</td>
<td align="right">0</td>
<td align="left">West</td>
</tr>
<tr>
<td align="left">b</td>
<td align="right">2017</td>
<td align="left">Q2</td>
<td align="left">West</td>
<td align="right">245</td>
<td align="left">West</td>
</tr>
</tbody>
</table>
<p>Notice we were able to easily copy the extra <code>BestRegion</code> values into all the correct rows.</p>
<p>It can be hard to figure out how to specify such a transformation in terms of pivots and un-pivots. However, as we have said: by drawing control tables one can easily design and manage fairly arbitrary data transform sequences (often stepping through either a denormalized intermediate where all values per-instance are in a single row, or a thin intermediate like the triple-like structure we just moved into).</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2017/11/arbitrary-data-transforms-using-cdata/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/DPMUbDOfIHw" height="1"></a></div>