<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/r-formulas-in-spark-and-un-nesting-data-in-sparklyr-nice-and-handy/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://longhowlam.wordpress.com/2017/02/15/r-formulas-in-spark-and-un-nesting-data-in-sparklyr-nice-and-handy/"> R – Longhow Lam's Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img sizes="(max-width: 517px) 100vw, 517px" data-image-description="" data-orig-file="https://i0.wp.com/longhowlam.files.wordpress.com/2017/02/sparkblog.png?resize=456%2C228&amp;ssl=1" data-orig-size="451,225" data-image-title="sparkblog" class="wp-image-2125" data-comments-opened="1" data-recalc-dims="1" data-attachment-id="2125" data-permalink="https://longhowlam.wordpress.com/2017/02/15/r-formulas-in-spark-and-un-nesting-data-in-sparklyr-nice-and-handy/sparkblog/" alt="sparkblog" data-medium-file="https://i0.wp.com/longhowlam.files.wordpress.com/2017/02/sparkblog.png?resize=456%2C228&amp;ssl=1?w=300" data-large-file="https://i0.wp.com/longhowlam.files.wordpress.com/2017/02/sparkblog.png?resize=456%2C228&amp;ssl=1?w=440" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" src="https://i0.wp.com/longhowlam.files.wordpress.com/2017/02/sparkblog.png?resize=456%2C228&ssl=1"></p>
<h2>Intro</h2>
<p>In an earlier post I talked about <a target="_blank" rel="nofollow" href="http://www.linkedin.com/pulse/een-wegwerp-spark-clustertje-om-medische-claims-te-analyseren-lam?">Spark and sparklyR</a> and did some experiments. At my work here at <strong>RTL Nederland</strong> we have a Spark cluster on Amazon EMR to do some serious heavy lifting on click and video-on-demand data. For an R user it makes perfectly sense to use Spark through the <a target="_blank" rel="nofollow" href="http://spark.rstudio.com/index.html">sparklyR </a>interface. However, using Spark through the <a target="_blank" rel="nofollow" href="http://spark.apache.org/docs/latest/api/python/pyspark.html">pySpark </a>interface certainly has its benefits. It exposes much more of the Spark functionality and I find the concept of <a target="_blank" rel="nofollow" href="http://spark.apache.org/docs/2.1.0/ml-pipeline.html">ML Pipelines</a> in Spark very elegant.</p>
<p>In using Spark I like to share two little <em>tricks</em> described below with you.</p>
<h2>The RFormula feature selector</h2>
<p>As an R user you have to get used to using Spark through pySpark, moreover, I had to brush up some of my rusty Python knowledge. For training machine learning models there is some help though by using an RFormula <img alt="🙂" class="wp-smiley" src="https://i0.wp.com/s0.wp.com/wp-content/mu-plugins/wpcom-smileys/twemoji/2/72x72/1f642.png?w=456&ssl=1" data-recalc-dims="1"></p>
<p>R users know the concept of <a target="_blank" rel="nofollow" href="http://stat.ethz.ch/R-manual/R-devel/library/stats/html/formula.html">model formulae</a> in R, it can be handy way to formulate predictive models in a concise way. In Spark you can also use this concept, only a limited set of R operators are available (<strong>+</strong>, <strong>–</strong> , <strong>.</strong> and <strong>:</strong>) , but it is enough to be useful. The two figures below show a simple example.<img sizes="(max-width: 738px) 100vw, 738px" data-image-description="" data-orig-file="https://i1.wp.com/longhowlam.files.wordpress.com/2017/02/rformula1.png?resize=456%2C182&amp;ssl=1" data-orig-size="1094,437" data-image-title="rformula1" data-comments-opened="1" class="wp-image-2131" data-recalc-dims="1" data-attachment-id="2131" data-permalink="https://longhowlam.wordpress.com/2017/02/15/r-formulas-in-spark-and-un-nesting-data-in-sparklyr-nice-and-handy/rformula1/" alt="rformula1" data-medium-file="https://i1.wp.com/longhowlam.files.wordpress.com/2017/02/rformula1.png?resize=456%2C182&amp;ssl=1?w=300" data-large-file="https://i1.wp.com/longhowlam.files.wordpress.com/2017/02/rformula1.png?resize=456%2C182&amp;ssl=1?w=440" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" src="https://i1.wp.com/longhowlam.files.wordpress.com/2017/02/rformula1.png?resize=456%2C182&ssl=1"></p>
<pre>

from pyspark.ml.feature import RFormula
f1 = &quot;Targetf ~ paidDuration + Gender &quot;
formula = RFormula(formula = f1)
train2 = formula.fit(train).transform(train)

</pre>
<p><img sizes="(max-width: 690px) 100vw, 690px" data-image-description="" data-orig-file="https://i1.wp.com/longhowlam.files.wordpress.com/2017/02/sparkrformula.png?resize=456%2C267&amp;ssl=1" data-orig-size="838,490" data-image-title="sparkrformula" data-comments-opened="1" class="wp-image-2169" data-recalc-dims="1" data-attachment-id="2169" data-permalink="https://longhowlam.wordpress.com/2017/02/15/r-formulas-in-spark-and-un-nesting-data-in-sparklyr-nice-and-handy/sparkrformula/" alt="sparkrformula" data-medium-file="https://i1.wp.com/longhowlam.files.wordpress.com/2017/02/sparkrformula.png?resize=456%2C267&amp;ssl=1?w=300" data-large-file="https://i1.wp.com/longhowlam.files.wordpress.com/2017/02/sparkrformula.png?resize=456%2C267&amp;ssl=1?w=440" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" src="https://i1.wp.com/longhowlam.files.wordpress.com/2017/02/sparkrformula.png?resize=456%2C267&ssl=1"></p>
<p>A handy thing about an RFormula in Spark is (just like using a formula in R in lm and some other modeling functions) that string features used in an RFormula will be automatically onehot encoded, so that they can be used directly in the Spark machine learning algorithms.</p>
<h2>Nested (hierarchical) data in sparklyR</h2>
<p>Sometimes you may find your self with nested hierarchical data. In pySpark you can flatten this hierarchy if needed. A simple example, suppose you read in a parquet file and it has the following structure:<img sizes="(max-width: 440px) 100vw, 440px" data-image-description="" data-orig-file="https://i0.wp.com/longhowlam.files.wordpress.com/2017/02/schema.png?w=456&amp;ssl=1" data-orig-size="589,393" data-image-title="schema" data-comments-opened="1" class="wp-image-2141" data-recalc-dims="1" data-attachment-id="2141" data-permalink="https://longhowlam.wordpress.com/2017/02/15/r-formulas-in-spark-and-un-nesting-data-in-sparklyr-nice-and-handy/schema/" alt="schema" data-medium-file="https://i0.wp.com/longhowlam.files.wordpress.com/2017/02/schema.png?w=456&amp;ssl=1?w=300" data-large-file="https://i0.wp.com/longhowlam.files.wordpress.com/2017/02/schema.png?w=456&amp;ssl=1?w=440" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" src="https://i0.wp.com/longhowlam.files.wordpress.com/2017/02/schema.png?w=456&ssl=1">Then to flatten the data you could use:<img sizes="(max-width: 440px) 100vw, 440px" data-image-description="" data-orig-file="https://i2.wp.com/longhowlam.files.wordpress.com/2017/02/sparkdf.png?w=456&amp;ssl=1" data-orig-size="762,415" data-image-title="sparkdf" data-comments-opened="1" class="wp-image-2146" data-recalc-dims="1" data-attachment-id="2146" data-permalink="https://longhowlam.wordpress.com/2017/02/15/r-formulas-in-spark-and-un-nesting-data-in-sparklyr-nice-and-handy/sparkdf/" alt="sparkdf" data-medium-file="https://i2.wp.com/longhowlam.files.wordpress.com/2017/02/sparkdf.png?w=456&amp;ssl=1?w=300" data-large-file="https://i2.wp.com/longhowlam.files.wordpress.com/2017/02/sparkdf.png?w=456&amp;ssl=1?w=440" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" src="https://i2.wp.com/longhowlam.files.wordpress.com/2017/02/sparkdf.png?w=456&ssl=1">In SparklyR however, reading the same parquet file results in something that isn’t useful to work with at first sight. If you open the table viewer to see the data, you will see rows with: &lt;environment>.<img sizes="(max-width: 440px) 100vw, 440px" data-image-description="" data-orig-file="https://i0.wp.com/longhowlam.files.wordpress.com/2017/02/nesteddata.png?w=456&amp;ssl=1" data-orig-size="688,232" data-image-title="nesteddata" data-comments-opened="1" class="wp-image-2148" data-recalc-dims="1" data-attachment-id="2148" data-permalink="https://longhowlam.wordpress.com/2017/02/15/r-formulas-in-spark-and-un-nesting-data-in-sparklyr-nice-and-handy/nesteddata/" alt="nesteddata" data-medium-file="https://i0.wp.com/longhowlam.files.wordpress.com/2017/02/nesteddata.png?w=456&amp;ssl=1?w=300" data-large-file="https://i0.wp.com/longhowlam.files.wordpress.com/2017/02/nesteddata.png?w=456&amp;ssl=1?w=440" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" src="https://i0.wp.com/longhowlam.files.wordpress.com/2017/02/nesteddata.png?w=456&ssl=1">Fortunately, the facilities used internally by sparklyR to call Spark are available to the end user. You can <strong>invoke </strong>more methods in Spark if needed. So we can invoke the <em>select </em>and <em>col </em>method our self to flatten the hierarchy.<img sizes="(max-width: 440px) 100vw, 440px" data-image-description="" data-orig-file="https://i2.wp.com/longhowlam.files.wordpress.com/2017/02/rparsed.png?w=456&amp;ssl=1" data-orig-size="779,267" data-image-title="rparsed" data-comments-opened="1" class="wp-image-2155" data-recalc-dims="1" data-attachment-id="2155" data-permalink="https://longhowlam.wordpress.com/2017/02/15/r-formulas-in-spark-and-un-nesting-data-in-sparklyr-nice-and-handy/rparsed/" alt="rparsed" data-medium-file="https://i2.wp.com/longhowlam.files.wordpress.com/2017/02/rparsed.png?w=456&amp;ssl=1?w=300" data-large-file="https://i2.wp.com/longhowlam.files.wordpress.com/2017/02/rparsed.png?w=456&amp;ssl=1?w=440" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" src="https://i2.wp.com/longhowlam.files.wordpress.com/2017/02/rparsed.png?w=456&ssl=1">After registering the output object, it is visible in the Spark interface and you can view the content.<img sizes="(max-width: 687px) 100vw, 687px" data-image-description="" data-orig-file="https://i1.wp.com/longhowlam.files.wordpress.com/2017/02/unnested.png?resize=456%2C123&amp;ssl=1" data-orig-size="1075,290" data-image-title="unnested" data-comments-opened="1" class="wp-image-2152" data-recalc-dims="1" data-attachment-id="2152" data-permalink="https://longhowlam.wordpress.com/2017/02/15/r-formulas-in-spark-and-un-nesting-data-in-sparklyr-nice-and-handy/unnested/" alt="unnested" data-medium-file="https://i1.wp.com/longhowlam.files.wordpress.com/2017/02/unnested.png?resize=456%2C123&amp;ssl=1?w=300" data-large-file="https://i1.wp.com/longhowlam.files.wordpress.com/2017/02/unnested.png?resize=456%2C123&amp;ssl=1?w=440" data-image-meta="{&quot;aperture&quot;:&quot;0&quot;,&quot;credit&quot;:&quot;&quot;,&quot;camera&quot;:&quot;&quot;,&quot;caption&quot;:&quot;&quot;,&quot;created_timestamp&quot;:&quot;0&quot;,&quot;copyright&quot;:&quot;&quot;,&quot;focal_length&quot;:&quot;0&quot;,&quot;iso&quot;:&quot;0&quot;,&quot;shutter_speed&quot;:&quot;0&quot;,&quot;title&quot;:&quot;&quot;,&quot;orientation&quot;:&quot;0&quot;}" src="https://i1.wp.com/longhowlam.files.wordpress.com/2017/02/unnested.png?resize=456%2C123&ssl=1"></p>
<p>Thanks for reading my two <em>tricks.</em> Cheers, Longhow.</p>
<p> <a target="_blank" rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/longhowlam.wordpress.com/2124/"><img alt="" src="http://feeds.wordpress.com/1.0/comments/longhowlam.wordpress.com/2124/" border="0"></a> <img alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" border="0" data-recalc-dims="1"></p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://longhowlam.wordpress.com/2017/02/15/r-formulas-in-spark-and-un-nesting-data-in-sparklyr-nice-and-handy/"> R – Longhow Lam's Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/_jm_j6T_AV8"></a></div>