<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/evaluate-your-model-with-r-exercises/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=http%3A%2F%2Fwww.r-exercises.com%2F2017%2F05%2F31%2Fevaluate-your-model-with-r-exercises%2F%3Futm_source%3Drss%26utm_medium%3Drss%26utm_campaign%3Devaluate-your-model-with-r-exercises"> R-exercises</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=https%3A%2F%2Fi2.wp.com%2Fwww.r-exercises.com%2Fwp-content%2Fuploads%2F2017%2F05%2Ftime_series_exercises_part4-36.png" target="_blank"><img sizes="(max-width: 150px) 100vw, 150px" src="https://i1.wp.com/www.r-exercises.com/wp-content/uploads/2017/05/time_series_exercises_part4-36-150x150.png?resize=150%2C150" alt="" class="wp-image-5873" data-recalc-dims="1"></a>There was a time where statistician had to manually crunch number when they wanted to fit their data to a model. Since this process was so long, those statisticians usually did a lot of preliminary work researching other model who worked in the past or looking for studies in other scientific field like psychology or sociology who can influence their model with the goal to maximize their chance to make a relevant model. Then they would create a model and an alternative model and choose the one which seem more efficient.</p>
<p>Now that even an average computer give us incredible computing power, it’s easy to make multiple models and choose the one that best fit the data. Even though it is better to have good prior knowledge of the process you are trying to analyze and of other model used in the past, coming to a conclusion using mostly only the data help you avoid bias and help you create better models.</p>
<p>In this set of exercises, we’ll see how to apply the most used error metrics on your models with the intention to rate them and choose the one that is the more appropriate for the situation.<br>
Most of those errors metrics are not part of any R package, in consequence you have to apply the equation I give you on your data. Personally, I preferred to write a function which I can easily<br>
use on everyone of my models, but there’s many ways to code those equations. If your code is different from the one in the solution, feel free to post your code in the comments.</p>
<p>Answers to the exercises are available <a rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=http%3A%2F%2Fwww.r-exercises.com%2F2017%2F05%2F31%2Fevaluate-your-model-with-r-solutions%2F" target="_blank">here</a>.</p>
<p><strong>Exercise 1</strong><br>
We start by looking at error metrics we can use for regression model. For linear regression problem, the more used metrics are the coefficient of determination <i>R</i><sup>2</sup> which show what percentage of variance is explained by the model and the adjusted <i>R</i><sup>2</sup> which penalize model who use variables that doesn’t have an effective contribution to the model (see <a rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FCoefficient_of_determination%23Adjusted_R2" target="_blank">this page</a> for more details). Load the <code>attitude</code> data set from the package of the same name and make three linear models with the goal to explain the <code>rating</code> variable. The first one use all the variables from the dataset, the second one use the variable <code>complaints</code>, <code>privileges</code>, <code>learning</code> and <code>advance</code> as independent variables and the third one use only the <code>complaints</code>, <code>learning</code> and <code>advance</code> variables. Then use the <code>summary()</code> function to print <i>R</i><sup>2</sup> and the adjusted <i>R</i><sup>2</sup>.</p>
<p><strong>Exercise 2</strong><br>
Another way to measure how your model fit your data is to use the Root Mean Squared Error (RMSE), which is defined as the square root of the average of the square of all the error made by your model. You can find the mathematical definition of the RMSE <a>on this page.</a><br>
Calculate the RMSE of the prediction made by your three models.</p>
<p><strong>Exercise 3</strong><br>
The mean absolute error (MAE) is a good alternative to the RMSE if you don’t want to penalise too much the large estimation error of your model. The MAE is given by the equation<br>
The mathematical definition of the MAE can be found <a>here.</a><br>
Calculate the MAE of the prediction made by the 3 models.</p>
<p><strong>Exercise 4</strong><br>
Sometime some prediction error hurt your model than other. For example, if you are trying to predict the financial loss of a business over a period of time, under estimation of the loss would<br>
put the business at risk of bankruptcy, while overestimation of the loss will result in a conservative model. In those case, using the Root Mean Squared Logarithmic Error (RMSLE) as an error<br>
metric is useful since this metric penalize under estimation. The RMSLE is given by the equation given on this <a>page.</a><br>
Calculate the RMSLE of the prediction made by the three models.</p>
<p><strong>Exercise 5</strong><br>
Now that we’ve seen some examples of error metrics which can be used in a regression context, let’s see five examples of error metrics which can be used when you perform clustering analysis. But<br>
first, we must create a clustering model to test those metrics on. Load the iris dataset and apply the kmeans algorithm. Since the iris dataset has three distinct<br>
labels, use the kmeans algorithm with three centers. Also, use set the maximum number of iterations at 50 and use the “Lloyd” algorithm. Once it’s done, take time to rearrange the labels of your<br>
prediction so they are compatible with the factors in the iris dataset.</p>
<div>
<aside><img src="http://www.r-exercises.com/2017/05/31/evaluate-your-model-with-r-exercises/?utm_source=rss&utm_medium=rss&utm_campaign=evaluate-your-model-with-r-exercises"></aside>
<div><strong>Learn more</strong> about Model Evaluation in the online course <a rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=http%3A%2F%2Fwww.r-exercises.com%2Fproduct%2Fregression-machine-learning-with-r%2F" target="_blank">Regression Machine Learning with R</a>. In this course you will learn how to:
<ul>
<li>Avoid model over-fitting using cross-validation for optimal parameter selection</li>
<li>Explore maximum margin methods such as best penalty of error term support vector machines with linear and non-linear kernels.</li>
<li>And much more</li>
</ul>
</div>
</div>
<p><strong>Exercise 6</strong><br>
Print the confusion matrix of your model.</p>
<p><strong>Exercise 7</strong><br>
The easiest way to measure how well your model did categorize the data, is to calculate the accuracy, the recall and the precision of your results. Write three functions which return those individual values and calculate those metrics for your models.</p>
<p><strong>Exercise 8</strong><br>
The <a rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FPrecision_and_recall%23F-measure" target="_blank">F-measure</a> summarize the precision and recall value of your model by calculating the harmonic mean of those two values.<br>
Write a function which return the F-measure of your model and compute twice this measure for your data: once with a parameter of 2 and then with a parameter of 0.5.</p>
<p><strong>Exercise 9</strong><br>
The <a rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=https%3A%2F%2Fen.wikipedia.org%2Fwiki%2FCluster_analysis%23External_evaluation" target="_blank">Purity</a> is a measure of the homogeneity of your cluster: if all your cluster regroup object of the same class, you’ll get a purity score of one and if there’s no majority class in any of the cluster, you’ll get a purity score of 0. Write a function which return the purity score of your model and test it on your predictions.</p>
<p><strong>Exercise 10</strong><br>
The last error metrics we’ll see today is the Dunn index, which indicate if the clusters are compact and separated. You can find the mathematical definition of the Dunn index <a rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=https%3A%2F%2Fwww.kaggle.com%2Fwiki%2FMultiClassLogLoss" target="_blank">here</a>. Load the <code>cvalid</code> package and use the <code>dunn()</code> on your model, to compute the Dunn index of your classification. Note that this function take an integer vector representing the cluster partitioning as parameter.</p>
<div>
<h3>Related exercise sets:</h3>
<ol>
<li><a rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=http%3A%2F%2Fwww.r-exercises.com%2F2017%2F05%2F02%2Fforecasting-for-small-business-exercises-part-4%2F" title="Forecasting for small business Exercises (Part-4)" target="_blank">Forecasting for small business Exercises (Part-4) </a></li>
<li><a rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=http%3A%2F%2Fwww.r-exercises.com%2F2017%2F05%2F05%2Fforecasting-arimax-model-exercises-part-5%2F" title="Forecasting: ARIMAX Model Exercises (Part-5)" target="_blank">Forecasting: ARIMAX Model Exercises (Part-5) </a></li>
<li><a rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=http%3A%2F%2Fwww.r-exercises.com%2F2017%2F04%2F17%2Fforecasting-exponential-smoothing-exercises-part-3%2F" title="Forecasting: Exponential Smoothing Exercises (Part-3)" target="_blank">Forecasting: Exponential Smoothing Exercises (Part-3) </a></li>
<li><a rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=http%3A%2F%2Fwww.r-exercises.com%2Fstart-here-to-learn-r%2F" target="_blank">Explore all our (>1000) R exercises</a></li>
<li><a rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=http%3A%2F%2Fwww.r-exercises.com%2Fr-courses%2F" target="_blank">Find an R course using our R Course Finder</a> directory</li>
</ol>
</div>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=http%3A%2F%2Fwww.r-exercises.com%2F2017%2F05%2F31%2Fevaluate-your-model-with-r-exercises%2F%3Futm_source%3Drss%26utm_medium%3Drss%26utm_campaign%3Devaluate-your-model-with-r-exercises"> R-exercises</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=https%3A%2F%2Ffeedburner.google.com%2Ffb%2Fa%2Fmailverify%3Furi%3DRBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=https%3A%2F%2Fwww.r-project.org%2F" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="http://apicdn.viglink.com/api/click?format=go&key=949efb41171ac6ec1bf7f206d57e90b8&loc=https%3A%2F%2Fwww.r-bloggers.com%2Ffeed%2F&out=https%3A%2F%2Fwww.r-users.com%2F" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/YslYjhQgZp4" height="1"></a></div>