<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/model-evaluation-2/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://r-exercises.com/2016/12/22/model-evaluation-2/"> R-exercises</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a target="_blank" rel="nofollow" href="http://r-exercises.com/wp-content/uploads/2016/12/judgment-889219_960_720.jpg?x61433"><img sizes="(max-width: 150px) 100vw, 150px" alt="" class="wp-image-3521" src="https://i2.wp.com/r-exercises.com/wp-content/uploads/2016/12/judgment-889219_960_720-150x150.jpg?resize=150%2C150" data-recalc-dims="1"></a>We are committed to bringing you 100% authentic exercise sets. We even try to include as different datasets as possible to give you an understanding of different problems. No more classifying Titanic dataset. R has tons of datasets in its library. This is to encourage you to try as many datasets as possible. We will be comparing two models by checking their accuracy, Area under the curve, ROC performance etc.</p>
<p>It will be helpful to go over Tom Fawcett’s <a target="_blank" rel="nofollow" href="http://people.inf.elte.hu/kiss/11dwhdm/roc.pdf">research paper</a> on ‘An introduction to ROC analysis’</p>
<p>Answers to the exercises are available <a target="_blank" rel="nofollow" href="http://r-exercises.com/2016/12/22/model-evaluation-2-2/">here</a>.</p>
<p>If you obtained a different (correct) answer than those listed on the solutions page, please feel free to post your answer as a comment on that page.</p>
<p><strong>Exercise 1</strong></p>
<p>Run the following code. If you do not have ROCR package installed, you can use install.packages() command to install it.</p>
<p><code><br>
library(ROCR)<br>
library(caTools)<br>
library(caret)<br>
data(&quot;GermanCredit&quot;)<br>
df1=GermanCredit<br>
df1$Class=ifelse(df1$Class==&quot;Bad&quot;,1,0)<br>
set.seed(100)<br>
spl=sample.split(df1$Class,SplitRatio = 0.7)<br>
Train1=df1[spl==TRUE,]<br>
Test1=df1[spl==FALSE,]<br>
model1=glm(Class~.,data=Train1,family = binomial)<br>
pred1=predict(model1,Test1)<br>
table(Test1$Class,pred1>0.5)<br>
</code></p>
<p><strong>Exercise 2</strong></p>
<p>Using the confusion matrix, please state what is the accuracy of this model?</p>
<p><strong>Exercise 3</strong></p>
<p>Great. Now let’s see the ROC curve of the model. Use this code below and then use plot() command to plot ROCRperf2</p>
<p><code><br>
ROCRpred1=prediction(pred1,Test1$Class)<br>
ROCRperf1=performance(ROCRpred1,&quot;tpr&quot;,&quot;fpr&quot;)<br>
</code></p>
<p>The plot above gives us an idea of the performance of the model. Is this a a good or bad model? State reasons</p>
<p><strong>Exercise 4</strong> </p>
<p>use the summary function on the model to see the summary. Note that if there are more stars next to a feature, then it is highly corelated with our target variable. </p>
<p><strong>Exercise 5</strong> </p>
<p>Although we found out the accuracy of the model in Q2, it is still not the best measure. A better measure is area under the curve. AUC takes account of class distribution in the model and is in the range of 0 to 1. 1 being the best and 0 being the worse. It can also be taken as a probability score. If the AUC is 0.70 then that means there is a 0.7 chance of the model to predict positive.</p>
<p>Insert the code below to obtain AUC. What is the AUC score? Is it better than the accuracy obrained at Q2?<br>
<code><br>
auc= performance(ROCRpred1,measure=&quot;auc&quot;)<br>
auc=auc@y.values[[1]]
<p><br>
<strong>Exercise 6</strong></p>
<p>Now create another model called model2 and include 11 variables that have atleast a star next to their name.Hint: use the summary() command and intercept does not count.</p>
<p><strong>Exercise 7</strong></p>
<p>Now predict the target variable using the Test1 sample using model2 and store it in pred2.</p>
<p><strong>Exercise 8</strong><br>
Use the <code>table()</code> command to get the confusion matrix. Note the accuracy.</p>
<p><strong>Exercise 9</strong><br>
What is the auc of model2?</p>
<p><strong>Exercise 10</strong><br>
Is model2 better than model 1? If so, then why?</p>

<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://r-exercises.com/2016/12/22/model-evaluation-2/"> R-exercises</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/yaTEVDPRQCI"></a></div></code>