<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/integrating-r-with-apache-hadoop/"></div></div><div><a data-url="http://www.r-bloggers.com/integrating-r-with-apache-hadoop/" data-counturl="http://www.r-bloggers.com/integrating-r-with-apache-hadoop/" data-count="horizontal" data-text="Integrating R with Apache Hadoop" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://datascienceplus.com/integrating-r-with-apache-hadoop/"> DataScience+</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Integrating R to work on Hadoop is to address the requirement to scale R program to work with petabyte scale data. The primary goal of this post is to elaborate different techniques for integrating R with Hadoop.</p>
<h2>Approach 1: Using R and Streaming APIs in Hadoop</h2>
<p>In order to integrate an R function with Hadoop and see it running in a MapReduce mode, Hadoop supports Streaming APIs for R. These Streaming APIs primary help running any script that can access and operate with standard I/O in a map- reduce mode. So, in case of R, there wouldn’t be any explicit client side integration done with R. Following is an example for R and streaming:</p>
<pre>$ ${HADOOP_HOME}/bin/Hadoop jar
${HADOOP_HOME}/contrib/streaming/*.jar 
-inputformat
org.apache.hadoop.mapred.TextInputFormat 
-input input_data.txt 
-output output 
-mapper /home/tst/src/map.R 
-reducer /home/tst/src/reduce.R 
-file /home/tst/src/map.R 
-file /home/tst/src/reduce.R</pre>
<h2>Approach 2: Using Rhipe package of R</h2>
<p>There is a package in R called “Rhipe” that allows running a MapReduce job within R. To use this way of implementing R on Hadoop there are some pre-requisites. R needs to be installed on each Data node in the Hadoop Cluster, Protocol Buffers will be installed and available on each Data node (for more on Protocol Buffer refer http://wiki.apache.org/hadoop/ProtocolBuffers) and Rhipe should be available on each data node.</p>
<p>Following is a sample format for using Rhipe library in R to implement MapReduce:</p>
<pre>library(Rhipe)
rhinit(TRUE, TRUE);
map&lt;-expression ( {lapply (map.values, function(mapper)…)})
reduce&lt;-expression(
pre = {…},
reduce = {…},
post = {…},
)
x &lt;- rhmr(map=map, reduce=reduce,
 ifolder=inputPath,
 ofolder=outputPath,
 inout=c('text', 'text'),
 jobname='test name'))
rhex(x)</pre>
<h2>Approach 3: Using RHADOOP</h2>
<p>RHadoop, very similar to RHipe, facilitates running R functions in a MapReduce mode. It is an open source library built by Revolution Analytics. Following are some packages the are a part of the RHadoop library. <code>plyrmr</code> apackage that provides functions for common data manipulation requirements for large datasets running on Hadoop. <code>rmr</code> a package that has collection of functions that integrate R and Hadoop. <code>rdfs</code> a package with functions that help interface R and HDFS. <code>rhbase</code> a package with functions that help interface R and HBase</p>
<p>Following is an example that uses <code>rmr</code> package and demonstrates the steps to integrate R and Hadoop using the functions from that package.</p>
<pre>library(rmr)
maplogic&lt;-function(k,v) { …}
reducelogic&lt;-function(k,vv) { …}
mapreduce( input =&quot;data.txt&quot;,
output=&quot;output&quot;,
textinputformat =rawtextinputformat,
map = maplogic,
reduce=reducelogic
)</pre>
<h2>Summary of R / Hadoop integration approaches</h2>
<p>In summary, all the above three approaches yield results and facilitate integrating R and Hadoop and help scaling R to operate on large scale data that will be help on HDFS and each of these approaches has pros and cons. </p>
<h3>Below is a summary of conclusions:</h3>
<p>Hadoop Streaming API is the simplest of all the approaches as there are not any complications in terms of installation and set-up requirements. Both Rhipe and RHadoop requires some effort to set up R and related packages on the Hadoop cluster. In terms of implementation approach Streaming API is more of a command line map and reduce functions are inputs to the function while both Rhipe and RHadoop allows developers to define and call custom MapReduce function within R. In case of Hadoop Streaming API, there is no client side integration required while both Rhipe and RHadoop require client side integration. The other alternatives to scaling machine learning are Apache Mahout, Apache Hive and some commercials versions of R from Revolution Analytics, Segue framework and others.</p>
<p>This post is an extract from my latest publication on <a target="_blank" rel="nofollow" href="https://www.packtpub.com/big-data-and-business-intelligence/practical-machine-learning">Machine learning</a>.</p>
<p>Feel free to comment below if you have any question.</p>
<ul></ul>
<p><strong>
<p>Related Post</p>
<p>
<ol>
<li><a title="Permanent Link toPlotting App for ggplot2" target="_blank" rel="bookmark" href="http://datascienceplus.com/plotting-app-for-ggplot2/">Plotting App for ggplot2</a></li>
<li><a title="Permanent Link toPerforming SQL selects on R data frames" target="_blank" rel="bookmark" href="http://datascienceplus.com/performing-sql-selects-on-r-data-frames/">Performing SQL selects on R data frames</a></li>
<li><a title="Permanent Link toDeveloping an R Tutorial shiny dashboard app" target="_blank" rel="bookmark" href="http://datascienceplus.com/developing-a-r-tutorial-shiny-dashboard-app/">Developing an R Tutorial shiny dashboard app</a></li>
<li><a title="Permanent Link toStrategies to Speedup R Code" target="_blank" rel="bookmark" href="http://datascienceplus.com/strategies-to-speedup-r-code/">Strategies to Speedup R Code</a></li>
<li><a title="Permanent Link toSentiment analysis with machine learning in R" target="_blank" rel="bookmark" href="http://datascienceplus.com/sentiment-analysis-with-machine-learning-in-r/">Sentiment analysis with machine learning in R</a></li>
</ol></strong>
<div>
<a target="_blank" href="http://feeds.feedburner.com/~ff/datascienceplus?a=AN1B7jC-kcc:4HYHVyTZgjQ:D7DqB2pKExk"><img src="http://feeds.feedburner.com/~ff/datascienceplus?i=AN1B7jC-kcc:4HYHVyTZgjQ:D7DqB2pKExk" border="0"></a> <a target="_blank" href="http://feeds.feedburner.com/~ff/datascienceplus?a=AN1B7jC-kcc:4HYHVyTZgjQ:qj6IDK7rITs"><img src="http://feeds.feedburner.com/~ff/datascienceplus?d=qj6IDK7rITs" border="0"></a> <a target="_blank" href="http://feeds.feedburner.com/~ff/datascienceplus?a=AN1B7jC-kcc:4HYHVyTZgjQ:yIl2AUoC8zA"><img src="http://feeds.feedburner.com/~ff/datascienceplus?d=yIl2AUoC8zA" border="0"></a>
</div>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/integrating-r-with-apache-hadoop/"></div></div><div><a data-url="http://www.r-bloggers.com/integrating-r-with-apache-hadoop/" data-counturl="http://www.r-bloggers.com/integrating-r-with-apache-hadoop/" data-count="vertical" data-text="Integrating R with Apache Hadoop" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://datascienceplus.com/integrating-r-with-apache-hadoop/"> DataScience+</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/EoueLK0K0YQ"></a></div>