<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/permutation-theory-in-action/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2017/09/permutation-theory-in-action/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>While working on a large client project using <a rel="nofollow" href="https://spark.rstudio.com/" target="_blank"><code>Sparklyr</code></a> and multinomial regression we recently ran into a problem: <a rel="nofollow" href="https://spark.apache.org/" target="_blank"><code>Apache Spark</code></a> chooses the order of multinomial regression outcome targets, whereas <a rel="nofollow" href="https://www.r-project.org/" target="_blank"><code>R</code></a> users are used to choosing the order of the targets (please see <a rel="nofollow" href="https://github.com/rstudio/sparklyr/issues/907" target="_blank">here</a> for some details). So to make things more like <code>R</code> users expect, we need a way to translate one order to another.</p>
<p>Providing good solutions to gaps like this is one of the thing <a rel="nofollow" href="http://www.win-vector.com/" target="_blank">Win-Vector LLC</a> does both in our <a rel="nofollow" href="http://www.win-vector.com/site/service-offerings/consulting/" target="_blank">consulting</a> and <a rel="nofollow" href="http://www.win-vector.com/site/service-offerings/training-overview/" target="_blank">training</a> practices.</p>
<p><span></span></p>
<p>Let’s take a look at an example. Suppose our two orderings are <code>o1</code> (the ordering <code>Spark ML</code> chooses) and <code>o2</code> (the order the <code>R</code> user chooses).</p>
<div>
<pre><code><span>set.seed</span>(<span>326346</span>)
symbols &lt;-<span> </span>letters[<span>1</span>:<span>7</span>]

o1 &lt;-<span> </span><span>sample</span>(symbols, <span>length</span>(symbols), <span>replace =</span> <span>FALSE</span>)
o1</code></pre>
</div>
<pre><code>## [1] &quot;e&quot; &quot;a&quot; &quot;b&quot; &quot;f&quot; &quot;d&quot; &quot;c&quot; &quot;g&quot;</code></pre>
<div>
<pre><code>o2 &lt;-<span> </span><span>sample</span>(symbols, <span>length</span>(symbols), <span>replace =</span> <span>FALSE</span>)
o2</code></pre>
</div>
<pre><code>## [1] &quot;d&quot; &quot;g&quot; &quot;f&quot; &quot;e&quot; &quot;b&quot; &quot;c&quot; &quot;a&quot;</code></pre>
<p>To translate <code>Spark</code> results into <code>R</code> results we need a <a rel="nofollow" href="https://en.wikipedia.org/wiki/Permutation" target="_blank">permutation</a> that takes <code>o1</code> to <code>o2</code>. The idea is: if we had a permeation that takes <code>o1</code> to <code>o2</code> we could use it to re-map predictions that are in <code>o1</code> order to be predictions in <code>o2</code> order.</p>
<p>To solve this we crack open <a rel="nofollow" href="http://www.win-vector.com/blog/2017/05/on-indexing-operators-and-composition/" target="_blank">our article on the algebra of permutations</a>.</p>
<p>We are going to use the fact that the <code>R</code> command <code>base::order(x)</code> builds a permutation <code>p</code> such that <code>x[p]</code> is in order.</p>
<p>Given this the solution is: we find permutations <code>p1</code> and <code>p2</code> such that <code>o1[p1]</code> is ordered and <code>o2[p2]</code> is ordered. Then build a permutation <code>perm</code> such that <code>o1[perm] = (o1[p1])[inverse_permutation(p2)]</code>. I.e., to get from <code>o1</code> to <code>o2</code> move <code>o1</code> to sorted order and then move from the sorted order to <code>o2</code>‘s order (by using the reverse of the process that sorts <code>o2</code>). Again, the tools to solve this are in <a rel="nofollow" href="http://www.win-vector.com/blog/2017/05/on-indexing-operators-and-composition/" target="_blank">our article on the relation between permutations and indexing</a>.</p>
<p>Below is the complete solution (including combining the two steps into a single permutation):</p>
<div>
<pre><code>p1 &lt;-<span> </span><span>order</span>(o1)
p2 &lt;-<span> </span><span>order</span>(o2)

<span># invert p2</span>
<span># see: http://www.win-vector.com/blog/2017/05/on-indexing-operators-and-composition/</span>
p2inv &lt;-<span> </span><span>seq_len</span>(<span>length</span>(p2))
p2inv[p2] &lt;-<span> </span><span>seq_len</span>(<span>length</span>(p2))

(o1[p1])[p2inv]</code></pre>
</div>
<pre><code>## [1] &quot;d&quot; &quot;g&quot; &quot;f&quot; &quot;e&quot; &quot;b&quot; &quot;c&quot; &quot;a&quot;</code></pre>
<div>
<pre><code><span># composition rule: (o1[p1])[p2inv] == o1[p1[p2inv]]</span>
<span># see: http://www.win-vector.com/blog/2017/05/on-indexing-operators-and-composition/</span>
perm &lt;-<span> </span>p1[p2inv]
o1[perm]</code></pre>
</div>
<pre><code>## [1] &quot;d&quot; &quot;g&quot; &quot;f&quot; &quot;e&quot; &quot;b&quot; &quot;c&quot; &quot;a&quot;</code></pre>
<p>The equivilence &quot;<code>(o1[p1])[p2inv] == o1[p1[p2inv]]</code>&quot; is frankly magic (though also quickly follows &quot;by definition&quot;), and studying it is the topic of <a rel="nofollow" href="http://www.win-vector.com/blog/2017/05/on-indexing-operators-and-composition/" target="_blank">our original article on permutations</a>.</p>
<p>The above application is a good example of why it is nice to have a little theory worked out, even before you think you need it.</p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2017/09/permutation-theory-in-action/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/KsgVkz6xLdM" height="1"></a></div>