<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/building-scalable-data-pipelines-with-microsoft-r-server-and-azure-data-factory/"></div></div><div><a data-url="https://www.r-bloggers.com/building-scalable-data-pipelines-with-microsoft-r-server-and-azure-data-factory/" data-counturl="https://www.r-bloggers.com/building-scalable-data-pipelines-with-microsoft-r-server-and-azure-data-factory/" data-count="horizontal" data-text="Building Scalable Data Pipelines with Microsoft R Server and Azure Data Factory" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://blog.revolutionanalytics.com/2016/10/r-server-data-factory.html"> Revolutions</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<div>
<p>by Udayan Kumar, Data Scientist at Microsoft</p>
<p>Beginning in 2016, Microsoft rolled out a preview of  <a target="_blank" href="https://azure.microsoft.com/en-us/services/hdinsight/r-server/"><span>Microsoft R Server (MRS) for Azure HDInsight</span></a> clusters. This service provides a preconfigured instance of R server with Spark/Hadoop that can be provisioned within minutes. Recent blog posts (by <a target="_blank" href="https://blogs.msdn.microsoft.com/azuredatalake/2016/08/09/rapid-big-data-prototyping-with-microsoft-r-server-on-apache-spark-context-switching-spark-tuning"><span>Max Kaznady</span></a> and <a target="_blank" href="http://blog.revolutionanalytics.com/2016/04/mrs-nyc-taxi.html"><span>David Smith</span></a>) have highlighted how to use and tune this service for large scale machine learning tasks. In this post, we push the envelope and show how to build an end-to-end fully operationalized analytics pipeline using <a target="_blank" href="https://azure.microsoft.com/en-us/documentation/services/data-factory/"><span>Azure Data Factory</span></a> (ADF) and MRS with HDInsight (specifically Apache Spark). Herein, we provide a <a target="_blank" href="https://github.com/Azure-Samples/data-factory-r-server-apache-spark-pipeline"><span>walk-through tutorial</span></a> that shows how to use MRS with ADF. We also provide this <a target="_blank" href="https://azure.microsoft.com/en-us/documentation/articles/resource-manager-template-walkthrough/"><span>Azure Resource Management (ARM) template</span></a> for easy deployment. </p>
<div><a target="_blank" href="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d224d923970c-pi"><img title="HDinsight" alt="HDinsight" src="http://revolution-computing.typepad.com/.a/6a010534b1db25970b01b8d224d923970c-400wi"></a>
<div>Architecture diagram of Microsoft R Server with Azure Data Factory</div>
</div>
<p>Machine learning and data analysis tasks (such as training/re-training) are often run periodically and have dependence on other ETL tasks. ADF simplifies running and managing such repeated tasks to create production-ready pipelines. Essentially, ADF provides the following:</p>
<ol>
<li>ability to run tasks (aka pipeline activities) periodically,</li>
<li>ability to set dependency between tasks, and</li>
<li>easy maintainability and problem detection. </li>
</ol>
<p>By integrating Azure Data Factory with Microsoft R Server and Spark, we show how to configure a scalable training and testing pipeline that operates on large volumes of data. For this, we utilize the <a target="_blank" href="http://www.andresmh.com/nyctaxitrips">NYC taxis dataset</a>. A predictive model is used to determine the tip amount for each trip. The tip received after each completed trip can be used to retrain the model. This is where ADF comes into the picture. With ADF, it is possible to set up a pipeline that retrains the model and scores the incoming inputs at a cadence. Currently ADF only supports Map-reduce jobs on Linux HDI clusters. Since MRS runs only on Linux HDInsight clusters, as a workaround we use a <a target="_blank" href="https://github.com/Azure/Azure-DataFactory/tree/master/Samples/CustomizedJavaOnHDInsight"><span>custom task-runner</span></a> masquerading as a map-reduce job to run MRS script. Detailed instructions on how to do so are available at the link below. </p>
<p>Github: <a target="_blank" href="https://github.com/Azure-Samples/data-factory-r-server-apache-spark-pipeline">Building Scalable Data Pipelines with Microsoft R Server and Azure Data Factory: sample code and walk-through</a></p>
</div>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/building-scalable-data-pipelines-with-microsoft-r-server-and-azure-data-factory/"></div></div><div><a data-url="https://www.r-bloggers.com/building-scalable-data-pipelines-with-microsoft-r-server-and-azure-data-factory/" data-counturl="https://www.r-bloggers.com/building-scalable-data-pipelines-with-microsoft-r-server-and-azure-data-factory/" data-count="vertical" data-text="Building Scalable Data Pipelines with Microsoft R Server and Azure Data Factory" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://blog.revolutionanalytics.com/2016/10/r-server-data-factory.html"> Revolutions</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/fX_1HE5oYT8"></a></div>