<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/be-careful-evaluating-model-predictions/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2016/12/be-careful-evaluating-model-predictions/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>One thing I teach is: when evaluating the performance of regression models <a target="_blank" rel="nofollow" href="http://www.win-vector.com/blog/2013/02/dont-use-correlation-to-track-prediction-performance/">you should not use correlation as your score</a>. </p>
<p>This is because correlation tells you if a re-scaling of your result is useful, but you want to know if the result in your hand is in fact useful. For example: the Mars Climate Orbiter software <a target="_blank" rel="nofollow" href="https://en.wikipedia.org/wiki/Mars_Climate_Orbiter">issued thrust commands in pound-seconds units to an engine expecting the commands to be in newton-seconds units</a>. The two quantities are related by a constant ratio of 1.4881639, and therefore anything measured in pound-seconds units will have a correlation of 1.0 with the same measurement in newton-seconds units. However, one is not the other and the difference is why the Mars Climate Orbiter “encountered Mars at a lower than anticipated altitude and disintegrated due to atmospheric stresses.”</p>
<p>The need for a convenient direct F-test without accidentally triggering the implicit re-scaling that is associated with calculating a correlation is one of the reasons we supply the <a target="_blank" rel="nofollow" href="https://github.com/WinVector/sigr">sigr</a> <a target="_blank" rel="nofollow" href="https://cran.r-project.org/">R</a> library. However, even then things can become confusing.</p>
<p><center><br>
<a target="_blank" rel="nofollow" href="https://i0.wp.com/commons.wikimedia.org/wiki/File:%22USE_PROPER_TOOLS_FOR_THE_JOB_-_TAKE_CARE_OF_THEM%22_-_NARA_-_515954.jpg?ssl=1"><img title="NewImage.png" alt="NewImage" src="https://i0.wp.com/www.win-vector.com/blog/wp-content/uploads/2016/11/NewImage-8.png?resize=393%2C512" border="0" data-recalc-dims="1"></a><br>
</center></p>
<p>Please read on for a nasty little example.<span></span></p>
<p>
<p>Consider the following “harmless data frame.”</p>
<p><code>
<pre>
d &lt;- data.frame(prediction=c(0,0,-1,-2,0,0,-1,-2),
                   actual=c(2,3,1,2,2,3,1,2))
</pre>
<p></p>
<p>The recommended test for checking the quality of “<code>prediction</code>” related to “<code>actual</code>” is an F-test (this is the test <code>stats::lm</code> uses). We can directly run this test with <code>sigr</code> (assuming we have installed the package) as follows:</p>
<p><code>
<pre>
sigr::formatFTest(d,'prediction','actual',format='html')$formatStr
</pre>
<p><center><br>
<b>F Test</b> summary: (<i>R<sup>2</sup></i>=-16, <i>F</i>(1,6)=-5.6, <i>p</i>=n.s.).<br>
</center><br>
</p>
<p><code>sigr</code> reports an R-squared of -16 (please see <a target="_blank" rel="nofollow" href="http://www.win-vector.com/blog/2011/11/correlation-and-r-squared/">here</a> for some discussion of R-squared). This may be confusing, but it correctly communicates we have no model and in fact “<code>prediction</code>” is worse than just using the average (a very traditional null-model).</p>
<p>However, <code>cor.test</code> appears to think “<code>prediction</code>” is a usable model:</p>
<p><code>
<pre>
cor.test(d$prediction,d$actual)

	Pearson's product-moment correlation

data:  d$prediction and d$actual
t = 1.1547, df = 6, p-value = 0.2921
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 -0.3977998  0.8697404
sample estimates:
      cor 
0.4264014 
</pre>
<p></p>
<p>This is all for a prediction where <code>sum((d$actual-d$prediction)^2)==66</code> which is larger than <code>sum((d$actual-mean(d$actual))^2)==4</code>. We concentrate on effects measures (such as R-squared) as we can drive the p-values wherever we want just by adding more data rows. Our point is: you are worse off using this model than using the mean-value of the actual (2) as your constant predictor. To my mind that is not a good prediction. And <code>lm</code> seems similarly excited about “<code>prediction</code>.”</p>
<p><code>
<pre>
summary(lm(actual~prediction,data=d))

Call:
lm(formula = actual ~ prediction, data = d)

Residuals:
     Min       1Q   Median       3Q      Max 
-0.90909 -0.43182  0.09091  0.52273  0.72727 

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept)   2.2727     0.3521   6.455 0.000655 ***
prediction    0.3636     0.3149   1.155 0.292121    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 0.7385 on 6 degrees of freedom
Multiple R-squared:  0.1818,	Adjusted R-squared:  0.04545 
F-statistic: 1.333 on 1 and 6 DF,  p-value: 0.2921
</pre>
<p></p>
<p>One reason to not trust the <code>lm</code> result is it didn’t score the quality of “<code>prediction</code>“. It scored the quality of “<code>0.3636*prediction + 2.2727</code>.” It can be the case that “<code>0.3636*prediction + 2.2727</code>” is in fact a good predictor. But<br>
that doesn’t help us if it is “<code>prediction</code>” we are showing to our boss or putting into production. We can <em>try</em> to mitigate this by insisting <code>lm</code> try to stay closer to the original by turning off the intercept or offset with the “<code>0+</code>” notation. That looks like the following.</p>
<p><code>
<pre>
summary(lm(actual~0+prediction,data=d))

Call:
lm(formula = actual ~ 0 + prediction, data = d)

Residuals:
   Min     1Q Median     3Q    Max 
  0.00   0.00   1.00   2.25   3.00 

Coefficients:
           Estimate Std. Error t value Pr(>|t|)
prediction  -1.0000     0.6094  -1.641    0.145

Residual standard error: 1.927 on 7 degrees of freedom
Multiple R-squared:  0.2778,	Adjusted R-squared:  0.1746 
F-statistic: 2.692 on 1 and 7 DF,  p-value: 0.1448
</pre>
<p></p>
<p>Even the <code>lm(0+)</code>‘s adjusted prediction is bad as we see below:</p>
<p><code>
<pre>
d$lmPred &lt;- predict(lm(actual~0+prediction,data=d))
sum((d$actual-d$lmPred)^2)
[1] 26
</pre>
<p></p>
<p>Yes, the <code>lm(0+)</code> found a way to improve the prediction; but the improved prediction is still very bad (worse than using a well chosen constant). And it is hard to argue that “<code>-prediction</code>” is the same model as “<code>prediction</code>.”</p>
<p>Now <code>sigr</code> is fairly new code, so it is a bit bold saying it is right when it disagrees with the standard methods. However <code>sigr</code> is right in this case. The standard methods are not so much wrong as different, for two reasons:</p>
<ol>
<li>They are answering different questions. The F-test is designed to check if the predictions in-hand are good or not; “<code>cor.test</code>” and “<code>lm %>% summary</code>” are designed to check if any re-scaling of the prediction is in fact good. These are different questions. Using “<code>cor.test</code>” or “<code>lm %>% summary</code>” to test the utility of a potential variable is a good idea. The reprocessing hidden in these tests is consistent with the later use of a variable in a model. Using them to score model results that are supposed be directly used is wrong.</li>
<li>From the standard R code’s point of view it isn’t obvious what the right “null model” is. Remember our original point: the quality measures on <code>lm(0+)</code> are designed to see how well <code>lm(0+)</code> is working. This means the <code>lm(0+)</code> scores the quality of its output (not its inputs) so it gets credit for flipping the sign on the prediction. Also it considers the natural null-model to be one it can form where there are no variable driven effects. Since there is no intercept or “dc-term” in these models (caused by the “<code>0+</code>” notation) the grand average is not considered a plausible null-model as it isn’t in the concept space of the modeling situation the <code>lm</code> was presented with. Or from <code>help(summary.lm)</code>:<br>
<blockquote><p>R^2, the ‘fraction of variance explained by the model’,<br>
R^2 = 1 – Sum(R[i]^2) / Sum((y[i]- y*)^2),<br>
where y* is the mean of y[i] if there is an intercept and zero otherwise.</p>
</blockquote>
</li>
</ol>
<p>I admit, this <em>is</em> very confusing. But it corresponds to documentation, and makes sense from a modeling perspective. It is correct. The silent switching of null model from average to zero make sense in the context it is defined in. It does not make sense for testing our prediction, but that is just one more reason to use the proper F-test directly instead of trying to hack “<code>cor.test</code>” or “<code>lm(0+) %>% summary</code>” to calculate it for you.</p>
<p>And that is what <code>sigr</code> is about: standard tests (using <code>R</code> supplied implementations) with a slightly different calling convention to better document intent (which in our case is almost always measuring the quality of a model separate from model construction). It is a new library, so it doesn’t yet have the documentation needed to achieve its goal, but we will eventually get there.</p>

<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2016/12/be-careful-evaluating-model-predictions/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/UVrRlqxVIT4"></a></div></code></code></code></code></code></code>