<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/how-to-evaluate-r-models-in-azure-machine-learning-studio/"></div></div><div><a data-url="https://www.r-bloggers.com/how-to-evaluate-r-models-in-azure-machine-learning-studio/" data-counturl="https://www.r-bloggers.com/how-to-evaluate-r-models-in-azure-machine-learning-studio/" data-count="horizontal" data-text="How to evaluate R models in Azure Machine Learning Studio" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.nodalpoint.com/azure-ml-studio/"> R – Nodalpoint</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p align="justify">Azure Machine Learning Studio is a GUI-based integrated development environment for constructing and operationalizing machine learning workflows. The basic computational unit of an Azure ML Studio workflow (or Experiment) is a <strong>module</strong> which implements machine learning algorithms, data conversion and transformation functions etc. Modules can be connected by data flows, thus implementing a machine learning pipeline. A typical pipeline in Azure ML Studio is shown below:</p>
<p><a target="_blank" href="https://i1.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_train_score_evaluate.png"><img alt="azure_post_train_score_evaluate" class="wp-image-3154" src="https://i1.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_train_score_evaluate.png?resize=456%2C199" data-recalc-dims="1"></a></p>
<p align="justify">In the figure above, a native Azure ML Studio model (<b>Two-Class Boosted Decision Trees</b>) is connected successively to a <b>Train Model</b> which performs the training procedure, to a <b>Score Model</b> to predict the outcome on test data, and finally to an <b>Evaluate Model</b> that will assess the overall classification quality.</p>
<p align="justify">Microsoft currently offers an option for a <a target="_blank" href="https://msdn.microsoft.com/en-us/library/azure/dn955435.aspx"><strong>Create R Model</strong></a> module, which allows for creating custom ML models written in R and integrating them into Azure ML Studio pipelines, hence opening up a vast area of possibilities. However, currently the <strong>Evaluate Model</strong> module cannot be used with a <b>Create R Model</b> module (i.e. through the pipeline: model -> train -> score -> evaluate); quoting from the <a target="_blank" href="https://msdn.microsoft.com/en-us/library/azure/dn955435.aspx">documentation</a>:</p>
<blockquote>
<p align="justify"><span><img title="System_CAPS_warning" alt="System_CAPS_warning" src="https://i2.wp.com/i-msdn.sec.s-msft.com/dynimg/IC174466.jpeg?w=456&ssl=1" data-recalc-dims="1"> <strong><span>Warning</span></strong>:<br>
Currently it is not possible to pass the scored results of an R model to Evaluate Model or Cross-Validate Model. If you need to evaluate a model, you can write custom R script and run it using the Execute R Script module.</span></p>
</blockquote>
<p align="justify">This is not good news for a couple of reasons: the quality of a <b>Create R Model</b> module cannot be assessed uniformly and cannot be directly compared to native Azure models (<b>Evaluate Model</b> is used for comparison purposes too, as we will demonstrate below); hence, the user is forced to create custom comparison scripts, which is a tedious and non-standard process. Also, the <b>Evaluate Model</b> module offers handy out-of-the-box visualizations, which we would like to have available as they can help rapid prototyping. Finally, it is much more straightforward to save trained models in the Azure ML Studio workspace (without the need to work directly in an Azure blob storage) using the <strong>Create R Model</strong> module rather than the alternative option of <a target="_blank" href="https://msdn.microsoft.com/en-us/library/azure/dn905952.aspx"><strong>Execute R Script</strong></a>.</p>
<p align="justify">So, in this post we will present a tweak that bypasses this limitation, allowing the use of an <strong>Evaluate Model</strong> module with a <strong>Create R Model</strong> one.</p>
<h4>Demonstrating the issue</h4>
<p align="justify">We will use as our staring point an <a target="_blank" href="https://gallery.cortanaintelligence.com/Experiment/Create-R-Model-3">example</a> offered by Microsoft, which uses a two-class Naïve Bayes classifier from the <tt>e1071</tt> R package.</p>
<p>Here is the ‘Training Script’ of the <b>Create R Model</b>:</p>
<pre>

library(e1071)
features &lt;- get.feature.columns(dataset)
labels   &lt;- as.factor(get.label.column(dataset))
train.data &lt;- data.frame(features, labels)
feature.names &lt;- get.feature.column.names(dataset)
names(train.data) &lt;- c(feature.names, &quot;Class&quot;)
model &lt;- naiveBayes(Class ~ ., train.data)

</pre>
<p>and the respective ‘Scoring Script’:</p>
<pre>

library(e1071)
probabilities &lt;- predict(model, dataset, type=&quot;raw&quot;)[,2]
classes = 0.5))
scores &lt;- data.frame(classes, probabilities)

</pre>
<p>The complete experiment graph is shown in the figure below:<br>
<a target="_blank" href="https://i1.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_createRModel_2.png"><img alt="azure_post_createRModel_2" class="wp-image-3168" src="https://i1.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_createRModel_2.png?resize=456%2C185" data-recalc-dims="1"></a></p>
<p align="justify">The experiment ends to an <b>Execute Python Script</b> module that facilitates, programmatically (in Python!), the model evaluation. This script calculates quantities like “Accuracy”, “Precision”, “Recall”, and “AUC”, and produces a PNG plot of the ROC curve as shown below:</p>
<p align="center"><a target="_blank" href="https://i0.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_ROC_1.png"><img alt="azure_post_ROC_1" class="wp-image-3175" src="https://i2.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_ROC_1-300x257.png?resize=400%2C357" data-recalc-dims="1"></a></p>
<p align="justify">Let us first confirm Microsoft’s warning, by adding an <b>Evaluate Model</b> module and connecting it to the <b>Scored Model</b> like this:</p>
<p><a target="_blank" href="https://i0.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_evalauteModelPreError.png"><img alt="azure_post_evalauteModelPreError" class="wp-image-3139" src="https://i1.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_evalauteModelPreError-1024x684.png?resize=456%2C305" data-recalc-dims="1"></a></p>
<p align="justify">When we try to run the experiment we get the following error:</p>
<p><a target="_blank" href="https://i1.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_evalauteModelError.png"><img alt="azure_post_evalauteModelError" class="wp-image-3140" src="https://i1.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_evalauteModelError-1024x416.png?resize=456%2C185" data-recalc-dims="1"></a></p>
<p align="justify">It seems that the data passed from the <b>Score Model</b> to the <b>Evaluation Model</b> miss a special annotation for the scored columns. The error message states <code>There is no label column in &quot;Scored dataset&quot;</code>.</p>
<h4>Solving the issue</h4>
<p align="justify">Fortunately R again comes to the rescue. From the error message we deduce that <b>Evaluate Model</b> is missing a <i>label column</i> from the incoming dataset. <b> Score Model </b> on the other hand creates all the necessary information. This indicates a possible <i>metadata</i> issue. The appropriate columns do exist but need to be further annotated. The solution emerges when reading the appropriate <a target="_blank" href="https://msdn.microsoft.com/en-us/library/azure/dn905986.aspx">metadata documentation</a>. Typical metadata information includes:</p>
<ol>
<li>Treating Boolean or numeric columns as categorical values</li>
<li><b> Indicating which column contains the <i>true label</i>, the <i>assigned label </i> or the <i>classification score</i> </b></li>
<li>Marking columns as features</li>
<li>Changing date/time values to a numeric value</li>
<li>Adding or changing column names</li>
</ol>
<p align="justify">We need to indicate these special columns, i.e. <i>true label</i>, <i>assigned label, </i>and <i>classification score</i> in the dataset moved between the scoring and evaluation modules.<br>
The simplest way to do this is by using an <b>Execute R Script</b> as a bridge between the Score and the Evaluate modules. Our experiment now is as shown below:</p>
<p><a target="_blank" href="https://i1.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_evalauteModelErrorBridge.png"><img alt="azure_post_evalauteModelErrorBridge" class="wp-image-3141" src="https://i1.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_evalauteModelErrorBridge-1024x704.png?resize=456%2C313" data-recalc-dims="1"></a></p>
<p align="justify">Before we provide the code of the <b>Execute R Script</b> module, we must make three important observations:</p>
<ol>
<li>In the ‘Training Script’ of the example we can see that the classification column (<i>true label</i>) is called <code>Class</code></li>
<li>The ‘Scoring Script’ of the example ends with <code>scores &lt;- data.frame(classes, probabilities)</code>. The first corresponds to the <i>assigned label </i> and the second to the <i>classification score</i></li>
<li>The <b>Score Model</b> module has the option <code>Append score columns to output</code> checked, so we expect the ‘Scoring Script’ to add two extra columns on the input dataset: <code>classes</code> and <code>probabilities</code>.</li>
</ol>
<p align="justify">The final R script that will bridge the Score and Evaluation modules is as follows:</p>
<pre>
# Map 1-based optional input ports to variables
dataset1 &lt;- maml.mapInputPort(1) # class: data.frame

#
# Create a three column data.frame with columns 
#   true_labels: actual classes
#   assigned_label: predicted classes with threshold set to 0.5
#   calibrated_score: probability output from the Baye's classifier
data.set &lt;- data.frame(true_labels=dataset1$Class, 
                       assigned_label=dataset1$classes, 
                       calibrated_score=dataset1$probabilities)

#
# Change the &quot;label.type&quot; attribute of column &quot;Class&quot; to be of type &quot;True Labels&quot;
attr(data.set$true_labels, &quot;label.type&quot;) &lt;- &quot;True Labels&quot;
#
# Change the &quot;feature.channel&quot; and &quot;score.type&quot; attribute of &quot;scored&quot; column
attr(data.set$assigned_label, &quot;feature.channel&quot;) &lt;- &quot;Binary Classification Scores&quot;
attr(data.set$assigned_label, &quot;score.type&quot;) &lt;- &quot;Assigned Labels&quot;

#
# Change the &quot;feature.channel&quot; and &quot;score.type&quot; attribute of &quot;probs&quot; column
attr(data.set$calibrated_score, &quot;feature.channel&quot;) &lt;- &quot;Binary Classification Scores&quot;
attr(data.set$calibrated_score, &quot;score.type&quot;) &lt;- &quot;Calibrated Score&quot;

#
# Rename the columns
names(data.set) &lt;- c(&quot;Class&quot; ,&quot;Scored Labels&quot; , &quot;Scored Probabilities&quot;) 

#
# Export the dataset
maml.mapOutputPort(&quot;data.set&quot;);
</pre>
<p>We run the experiment and see that everything works. The <b>Evaluate Model</b> module is now operational, and we can click to visualize:</p>
<p><a target="_blank" href="https://i1.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_theEnd2.png"><img alt="azure_post_theEnd2" class="wp-image-3149" src="https://i0.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_theEnd2-1024x754.png?resize=456%2C336" data-recalc-dims="1"></a></p>
<p>The result of the visualization is shown below:</p>
<p><a target="_blank" href="https://i0.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_theEnd.png"><img alt="azure_post_theEnd" class="wp-image-3147" src="https://i1.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_theEnd-1024x475.png?resize=456%2C211" data-recalc-dims="1"></a></p>
<p align="justify">This is the ROC curve alright. We got the same figure from the <b>Execute Python Script</b> output. There is a huge difference though, as the result of <b>Evaluate Model</b> is <span>interactive</span> and the user can try different threshold combinations on the fly.</p>
<p align="justify">The R code provided above is tailored for the specific example; nevertheless, it is easily customized  by identifying the corresponding columns and making the appropriate modifications. Moreover, we can even integrate this code into the ‘Scoring Script’ part of our <strong>Create R Model</strong> module, hence eliminating the extra <strong>Execute R Script</strong> module altogether (we leave this as an exercise for the interested reader).</p>
<h4>Application: Comparing against native Azure ML Studio algorithms</h4>
<p align="justify">To demonstrate the usefulness of our workaround, we extend the existing experiment by adding a separate training workflow starting from a native Azure ML Studio algorithm. We then attach this secondary flow to our <b>Evaluate Model</b> module and perform a comparison. We have selected the native <b>Two-Class Bayes Point Machine</b>, and the experiment now looks like the figure below:</p>
<p><a target="_blank" href="https://i1.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_extended.png"><img alt="azure_post_extended" class="wp-image-3183" src="https://i1.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_extended-1024x659.png?resize=456%2C294" data-recalc-dims="1"></a></p>
<p align="justify">We can run the experiment and visualize the output of the <b>Evaluate Model</b>. In the figure below we present the two results side by side :</p>
<p><a target="_blank" href="https://i0.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_eval2-1.png"><img alt="azure_post_eval2" class="wp-image-3186" src="https://i0.wp.com/www.nodalpoint.com/wp-content/uploads/2016/08/azure_post_eval2-1.png?resize=456%2C224" data-recalc-dims="1"></a></p>
<p> </p>
<h4>Summary</h4>
<p align="justify">We have presented a workaround to connect scoring results from a <b>Create R Model</b> module to an <b>Evaluate Model</b> module in Azure ML Studio. This connection is not allowed in the current version of Azure ML Studio, and model evaluation has to be performed manually by user written scripts. The workaround consists of a rather simple R script that can be added in the existing ML pipeline between <b>Score Model</b> and <b>Evaluate Model</b>, altering the metadata of the scored dataset. With this modification, Azure ML Studio users can enjoy uniform evaluation of both native and custom machine learning models.</p>
<p>The post <a target="_blank" rel="nofollow" href="http://www.nodalpoint.com/azure-ml-studio/">How to evaluate R models in Azure Machine Learning Studio</a> appeared first on <a target="_blank" rel="nofollow" href="http://www.nodalpoint.com/">Nodalpoint</a>.</p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/how-to-evaluate-r-models-in-azure-machine-learning-studio/"></div></div><div><a data-url="https://www.r-bloggers.com/how-to-evaluate-r-models-in-azure-machine-learning-studio/" data-counturl="https://www.r-bloggers.com/how-to-evaluate-r-models-in-azure-machine-learning-studio/" data-count="vertical" data-text="How to evaluate R models in Azure Machine Learning Studio" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.nodalpoint.com/azure-ml-studio/"> R – Nodalpoint</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/Ul2hFnVzgNA"></a></div>