<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/on-calculating-auc/"></div></div><div><a data-url="https://www.r-bloggers.com/on-calculating-auc/" data-counturl="https://www.r-bloggers.com/on-calculating-auc/" data-count="horizontal" data-text="On calculating AUC" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2016/10/on-calculating-auc/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Recently Microsoft Data Scientist Bob Horton wrote a <a target="_blank" href="http://blog.revolutionanalytics.com/2016/08/roc-curves-in-two-lines-of-code.html">very nice article on ROC plots</a>. We expand on this a bit and discuss some of the issues in computing “area under the curve” (AUC).<span></span></p>
<p>
<p><a target="_blank" href="https://cran.r-project.org/">R</a> has a number of ROC/AUC packages; for example <a target="_blank" href="https://cran.r-project.org/package=ROCR">ROCR</a>, <a target="_blank" href="https://cran.r-project.org/package=pROC">pROC</a>, and <a target="_blank" href="https://cran.r-project.org/package=plotROC">plotROC</a>. But it is instructive to see how ROC plots are produced and how AUC can be calculated. Bob Horton’s article showed how elegantly the points on the ROC plot are expressed in terms of sorting and cumulative summation.</p>
<p>The next step is computing AUC. Obviously computing area is a solved problem. The issue is how you deal with interpolating between points and the conventions of what to do with data that has identical scores. An elegant interpretation of the usual tie breaking rules is: for every point on the ROC curve we must have either all of the data above a given score threshold or none of the data above a given score threshold. This is the issue alluded to when the original article states:</p>
<blockquote><p>This brings up another limitation of this simple approach; by assuming that the rank order of the outcomes embodies predictive information from the model, it does not properly handle sequences of cases that all have the same score.</p>
</blockquote>
<p>This problem is quite easy to explain with an example. Consider the following data.</p>
<p><code>
<pre>
d &lt;- data.frame(pred=c(1,1,2,2),y=c(FALSE,FALSE,TRUE,FALSE))
print(d)
## pred     y
## 1    1 FALSE
## 2    1 FALSE
## 3    2  TRUE
## 4    2 FALSE
</pre>
<p></p>
<p>Using code adapted from the original article we can quickly get an interesting summary.</p>
<p><code>
<pre>
ord &lt;- order(d$pred, decreasing=TRUE) # sort by prediction reversed
labels &lt;- d$y[ord]
data.frame(TPR=cumsum(labels)/sum(labels), 
           FPR=cumsum(!labels)/sum(!labels),
           labels=labels,
           pred=d$pred[ord])
##   TPR       FPR labels pred
## 1   1 0.0000000   TRUE    2
## 2   1 0.3333333  FALSE    2
## 3   1 0.6666667  FALSE    1
## 4   1 1.0000000  FALSE    1
</pre>
<p></p>
<p>The problem is: we need to take all of the points with the same prediction score as an atomic unit (we take all of them or none of them). Notice also <code>TPR</code> is always 1 (an undesirable effect).</p>
<p>We do not really want rows 1 and 3 in our plot or area calculations. In fact the values in row 1 and 3 are not fully determined as they can vary depending on details of tie breaking in the sorting (though the values recorded in rows 2 and 4 can not so vary). Also (especially after deleting rows) we may need to add in ideal points with <code>(FPR,TPR)=(0,0)</code> and <code>(FPR,TPR)=(1,1)</code> to complete our plot and area calculations. </p>
<p>What we want is a plot where ties are handled. Such plots look like the following:</p>
<p><code>
<pre>
# devtools::install_github('WinVector/WVPlots')
library('WVPlots') # see: https://github.com/WinVector/WVPlots
WVPlots::ROCPlot(d,'pred','y',TRUE,'example plot')
</pre>
<p></p>
<p><img title="NewImage.png" alt="NewImage" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2016/10/NewImage-1.png?w=660" border="0" data-recalc-dims="1"></p>
<p>There is a fairly elegant way to get the necessary adjusted plotting frame: use differencing (the opposite of cumulative sums) to find where the <code>pred</code> column changes, and limit to those rows.</p>
<p>The code is as follows (also found in our <code>sigr</code> library <a target="_blank" href="https://github.com/WinVector/sigr/blob/master/R/fns.R">here</a>):</p>
<p><code>
<pre>
calcAUC &lt;- function(modelPredictions,yValues) {
  ord &lt;- order(modelPredictions, decreasing=TRUE)
  yValues &lt;- yValues[ord]
  modelPredictions &lt;- modelPredictions[ord]
  x &lt;- cumsum(!yValues)/max(1,sum(!yValues)) # FPR = x-axis
  y &lt;- cumsum(yValues)/max(1,sum(yValues))   # TPR = y-axis
  # each point should be fully after a bunch of points or fully before a
  # decision level. remove dups to achieve this.
  dup &lt;- c(modelPredictions[-1]>=modelPredictions[-length(modelPredictions)],
           FALSE)
  # And add in ideal endpoints just in case (redundancy here is not a problem).
  x &lt;- c(0,x[!dup],1)
  y &lt;- c(0,y[!dup],1)
  # sum areas of segments (triangle topped vertical rectangles)
  n &lt;- length(y)
  area &lt;- sum( ((y[-1]+y[-n])/2) * (x[-1]-x[-n]) )
  area
}
</pre>
<p></p>
<p>This correctly calculates the AUC.</p>
<p><code>
<pre>
# devtools::install_github('WinVector/sigr')
library('sigr') # see: https://github.com/WinVector/sigr
calcAUC(d$pred,d$y)
## [1] 0.8333333
</pre>
<p></p>
<p>I think this extension maintains the spirit of the original. We have also shown how complexity increases as you move from code known to work on a particular data set at hand, to library code that may be exposed to data with unanticipated structures or degeneracies (this is why Quicksort, which has an elegant description, often has monstrous implementations; please see here for <a target="_blank" href="http://www.win-vector.com/blog/2008/04/sorting-in-anger/">a rant on that topic</a>).</p>

<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="https://www.r-bloggers.com/on-calculating-auc/"></div></div><div><a data-url="https://www.r-bloggers.com/on-calculating-auc/" data-counturl="https://www.r-bloggers.com/on-calculating-auc/" data-count="vertical" data-text="On calculating AUC" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>

<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2016/10/on-calculating-auc/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/DAGd9M7xWjs"></a></div></code></code></code></code></code>