<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/association-rules-using-fpgrowth-in-spark-mllib-through-sparklyr/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://longhowlam.wordpress.com/2017/11/23/association-rules-using-fpgrowth-in-spark-mllib-through-sparklyr/"> R – Longhow Lam's Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><img data-orig-file="https://longhowlam.files.wordpress.com/2017/11/sparkfp.png?w=456" data-orig-size="743,381" data-medium-file="https://longhowlam.files.wordpress.com/2017/11/sparkfp.png?w=456?w=300" data-attachment-id="2561" data-image-meta="{" data-image-description="" src="https://longhowlam.files.wordpress.com/2017/11/sparkfp.png?w=456" data-permalink="https://longhowlam.wordpress.com/2017/11/23/association-rules-using-fpgrowth-in-spark-mllib-through-sparklyr/sparkfp/" data-large-file="https://longhowlam.files.wordpress.com/2017/11/sparkfp.png?w=456?w=440" alt="sparkfp" data-image-title="sparkfp" sizes="(max-width: 440px) 100vw, 440px" data-comments-opened="1" class="wp-image-2561" data-recalc-dims="1"></p>
<h2>Introduction</h2>
<p>Market Basket Analysis or association rules mining can be a very useful technique to gain insights in transactional data sets, and it can be useful for product recommendation. The classical example is data in a supermarket. For each customer we know what the individual products (items) are that he has bought. With association rules mining we can identify items that are frequently bought together. Other use cases for MBA could be web click data, log files, and even questionnaires.</p>
<p>In R there is a package <strong>arules </strong>to calculate association rules, it makes use of the so-called Apriori algorithm. For data sets that are not too big, calculating rules with arules in R (on a laptop) is not a problem. But when you have very huge data sets, you need to do something else, you can:</p>
<ul>
<li>use more computing power (or cluster of computing nodes).</li>
<li>use another algorithm, for example FP Growth, which is more scalable. See <a rel="nofollow" href="https://www.singularities.com/blog/2015/08/apriori-vs-fpgrowth-for-frequent-item-set-mining" target="_blank">this blog</a> for some details on Apriori vs. FP Growth.</li>
</ul>
<p>Or do both of the above points by using FPGrowth in Spark MLlib on a cluster. And the nice thing is: you can stay in your familiar R Studio environment!</p>
<h2>Spark MLlib and sparklyr</h2>
<h3>Example Data set</h3>
<p>We use the example groceries transactions data in the arules package. It is not a big data set and you would definitely not need more than a laptop, but it is much more realistic than the example given in the<a rel="nofollow" href="https://spark.apache.org/docs/latest/ml-frequent-pattern-mining.html" target="_blank"> Spark MLlib documentation</a> :-).</p>
<h3>Preparing the data</h3>
<p>I am a fan of <a rel="nofollow" href="https://spark.rstudio.com/" target="_blank">sparklyr</a> <img alt="🙂" src="https://i0.wp.com/s0.wp.com/wp-content/mu-plugins/wpcom-smileys/twemoji/2/72x72/1f642.png?w=456&ssl=1" class="wp-smiley" data-recalc-dims="1"> It offers a good R interface to Spark and MLlib. You can use dplyr syntax to prepare data on Spark, it exposes many of the MLlib machine learning algorithms in a uniform way. Moreover, it is nicely integrated into the RStudio environment offering the user views on Spark data and a way to manage the Spark connection.</p>
<div><img src="https://i0.wp.com/media.licdn.com/mpr/mpr/AAIABADGAAAAAQAAAAAAAA0EAAAAJDJjOTE5OGVmLWM5NWMtNGRmZS04NWY1LWFkMDg1NTZiZGVjNQ.png?w=450&ssl=1" data-recalc-dims="1"></div>
<p>First connect to spark and read in the groceries transactional data, and upload the data to Spark. I am just using a local spark install on my Ubuntu laptop.</p>
<pre><span>######</span> sparklyr code to perform FPGrowth algorithm <span>############</span>

library(sparklyr)
library(dplyr)

<span>#### spark connect #########################################</span>
sc &lt;- spark_connect(master = <span>&quot;local&quot;</span>)

<span>#### first create some dummy data ###########################</span>
transactions = readRDS(<span>&quot;transactions.RDs&quot;</span>)

<span>#### upload to spark #########################################  </span>
trx_tbl  = copy_to(sc, transactions, overwrite = TRUE)
</pre>
<p>For demonstration purposes, data is copied in this example from the local R session to Spark. For large data sets this is not feasible anymore, in that case data can come from hive tables (on the cluster).</p>
<p>The figure above shows the products purchased by the first four customers in Spark in an RStudio grid. Although transactional systems will often output the data in this structure, it is not what the FPGrowth model in MLlib expects. It expects the data aggregated by id (customer) and the products inside an array. So there is one more preparation step.</p>
<pre><span># data needs to be aggregated by id, the items need to be in a list</span>
trx_agg = trx_tbl %>% 
   group_by(id) %>% 
   summarise(
      items = collect_list(item)
   )
</pre>
<div><img src="https://i0.wp.com/media.licdn.com/mpr/mpr/AAIABADGAAAAAQAAAAAAAAunAAAAJDlkNmQyZTQxLWM3M2YtNGIwMS04ODhkLWQ1NDc0ZmZkZDYwOA.png?w=456&ssl=1" data-recalc-dims="1"></div>
<p>The figure above shows the aggregated data, customer 12, has a list of 9 items that he has purchased.</p>
<h3>Running the FPGrowth algorithm</h3>
<p>We can now run the FPGrowth algorithm, but there is one more thing. Sparklyr does not expose the FPGrowth algorithm (yet), there is no R interface to the FPGrowth algorithm. Luckily, sparklyr allows the user to invoke the underlying Scala methods in Spark. We can define an new object with <strong>invoke_new</strong></p>
<pre>  uid = sparklyr::<span>:random_string</span>(<span>&quot;fpgrowth_&quot;</span>)
  jobj = invoke_new(sc, <span>&quot;org.apache.spark.ml.fpm.FPGrowth&quot;</span>, uid) 
</pre>
<p>Now jobj is an object of class FPGrowth in Spark.</p>
<pre>jobj
&lt;jobj[457]>
  class org.apache.spark.ml.fpm.FPGrowth
  fpgrowth_d4d41f71f3e0
</pre>
<p>And by looking at the <a rel="nofollow" href="https://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.ml.fpm.FPGrowth" target="_blank">Scala documentation</a> of FPGrowth we see that there are more methods that you can use. We need to use the function <strong>invoke, </strong>to specify which column contains the list of items, to specify the minimum confidence and to specify the minimum support.</p>
<pre>jobj %>% 
    invoke(<span>&quot;setItemsCol&quot;</span>, <span>&quot;items&quot;</span>) %>%
    invoke(<span>&quot;setMinConfidence&quot;</span>, 0.03) %>%
    invoke(<span>&quot;setMinSupport&quot;</span>, 0.01)  %>%
    invoke(<span>&quot;fit&quot;</span>, spark_dataframe(trx_agg))
</pre>
<p>By invoking <strong>fit</strong>, the FPGrowth algorithm is fitted and an FPGrowthModel object is returned where we can invoke <strong>associationRules</strong> to get the calculated rules in a spark data frame</p>
<pre>rules = FPGmodel %>% invoke(<span>&quot;associationRules&quot;</span>)
</pre>
<div><img src="https://i2.wp.com/media.licdn.com/mpr/mpr/AAIABADGAAAAAQAAAAAAAAo5AAAAJGYwMjhmYTE0LTMzZDMtNGUwYi05Y2NkLWUwZDg0MDllYjJmOQ.png?w=450&ssl=1" data-recalc-dims="1"></div>
<p>The rules in the spark data frame consists of an antecedent column (the left hand side of the rule), a consequent column (the right hand side of the rule) and a column with the confidence of the rule. Note that the antecedent and consequent are lists of items! If needed we can split these lists and collect them to R for plotting for further analysis.</p>
<p>The invoke statements and rules extractions statements can of course be wrapped inside functions to make it more reusable. So given the aggregated transactions in a spark table trx_agg, you can get something like:</p>
<pre>GroceryRules =  ml_fpgrowth(
  trx_agg
) <span>%>%</span>
  ml_fpgrowth_extract_rules()

plot_rules(GroceryRules)
</pre>
<div><img src="https://i1.wp.com/media.licdn.com/mpr/mpr/AAIABADGAAAAAQAAAAAAAAqlAAAAJDQxNmY5Y2Q0LTM5MmEtNGViNi05OTljLTBkNjdjODE2YTAwMg.png?w=456&ssl=1" data-recalc-dims="1"></div>
<h2>Conclusion</h2>
<p>The complete R script can be found on my <a rel="nofollow" href="https://github.com/longhowlam/BitsAndPieces" target="_blank">GitHub</a>. If arules in R on your laptop is not workable anymore because of the size of your data, consider FPGrowth in Spark through sparklyr.</p>
<p>cheers, Longhow</p>
<p> <a rel="nofollow" href="https://feeds.wordpress.com/1.0/gocomments/longhowlam.wordpress.com/2560/" target="_blank"><img border="0" alt="" src="https://feeds.wordpress.com/1.0/comments/longhowlam.wordpress.com/2560/"></a> <img border="0" alt="" src="https://i1.wp.com/pixel.wp.com/b.gif?resize=1%2C1&ssl=1" data-recalc-dims="1"></p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://longhowlam.wordpress.com/2017/11/23/association-rules-using-fpgrowth-in-spark-mllib-through-sparklyr/"> R – Longhow Lam's Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/q13v8KTPNf4" height="1"></a></div>