<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/probabilistic-interpretation-of-auc/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.alexejgossmann.com/auc/"> Alexej's blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Unfortunately this was not taught in any of my statistics or data analysis classes at university (wtf it so needs to be :scream_cat:).<br>
So it took me some until I learned that the <a rel="nofollow" href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic#Area_under_the_curve" target="_blank">AUC</a> has a nice probabilistic meaning.</p>
<h2>What’s AUC anyway?</h2>
<p>Consider:</p>
<ol>
<li>A dataset <img src="http://latex.codecogs.com/png.latex?S%5Cinline">: <img src="http://latex.codecogs.com/png.latex?(%5Cmathbf%7Bx%7D_1,%20y_1),%20%5Cldots,%20(%5Cmathbf%7Bx%7D_n,%20y_n)%20%5Cin%20%5Cmathbb%7BR%7D%5Ep%20%5Ctimes%20%5C%7B0,%201%5C%7D%5Cinline">, where
<ul>
<li><img src="http://latex.codecogs.com/png.latex?%5Cmathbf%7Bx%7D_i%5Cinline"> is a vector of <img src="http://latex.codecogs.com/png.latex?p%5Cinline"> features collected for the <img src="http://latex.codecogs.com/png.latex?i%5Cinline">th subject,</li>
<li><img src="http://latex.codecogs.com/png.latex?y_i%5Cinline"> is the <img src="http://latex.codecogs.com/png.latex?i%5Cinline">th subject’s label (binary outcome variable of interest, like a disease status, class membership, or whatever binary label).</li>
</ul>
</li>
<li>A classification algorithm (like logistic regression, SVM, deep neural net, or whatever you like), trained on <img src="http://latex.codecogs.com/png.latex?S%5Cinline">, that assigns a score (or probability) <img src="http://latex.codecogs.com/png.latex?%5Chat%7Bp%7D(%5Cmathbf%7Bx%7D_%7B%5Cast%7D)%5Cinline"> to any new observation <img src="http://latex.codecogs.com/png.latex?%5Cmathbf%7Bx%7D_%7B%5Cast%7D%20%5Cin%20%5Cmathbb%7BR%7D%5Ep%5Cinline"> signifying how likely its label is <img src="http://latex.codecogs.com/png.latex?y_%7B%5Cast%7D%20=%201%5Cinline">.</li>
</ol>
<p>Then:</p>
<ol>
<li>A <em>decision threshold</em> (or <em>operating point</em>) can be chosen to assign a class label (<img src="http://latex.codecogs.com/png.latex?y_%7B%5Cast%7D%20=%200%5Cinline"> or <img src="http://latex.codecogs.com/png.latex?1%5Cinline">) to <img src="http://latex.codecogs.com/png.latex?%5Cmathbf%7Bx%7D_%7B%5Cast%7D%5Cinline"> based on the value of <img src="http://latex.codecogs.com/png.latex?%5Chat%7Bp%7D(%5Cmathbf%7Bx%7D_%7B%5Cast%7D)%5Cinline">.<br>
The chosen threshold determines the balance between how many <em>false positives</em> and <em>false negatives</em> will result from this classification.</li>
<li>Plotting the <em>true positive rate</em> (TPR) against the <em>false positive rate</em> (FPR) <em>as the operating point changes from its minimum to its maximum value</em> yields the <a rel="nofollow" href="https://en.wikipedia.org/wiki/Receiver_operating_characteristic" target="_blank"><em>receiver operating characteristic (ROC) curve</em></a>. Check the <a rel="nofollow" href="https://en.wikipedia.org/wiki/Sensitivity_and_specificity#Confusion_matrix" target="_blank">confusion matrix</a> if you are not sure what TPR and FPR refer to.</li>
<li>The area under the ROC curve, or AUC, is used as a measure of classifier performance.</li>
</ol>
<p>Here is some R code for clarification (not even using <code>tidyverse</code> :stuck_out_tongue:):</p>
<div>
<div>
<pre><code><span># load some data, fit a logistic regression classifier</span><span>
</span><span>data</span><span>(</span><span>iris</span><span>)</span><span>
</span><span>versicolor_virginica</span><span> </span><span>&lt;-</span><span> </span><span>iris</span><span>[</span><span>iris</span><span>$</span><span>Species</span><span> </span><span>!=</span><span> </span><span>&quot;setosa&quot;</span><span>,</span><span> </span><span>]</span><span>
</span><span>logistic_reg_fit</span><span> </span><span>&lt;-</span><span> </span><span>glm</span><span>(</span><span>Species</span><span> </span><span>~</span><span> </span><span>Sepal.Width</span><span> </span><span>+</span><span> </span><span>Sepal.Length</span><span>,</span><span>
</span><span>data</span><span> </span><span>=</span><span> </span><span>versicolor_virginica</span><span>,</span><span>
</span><span>family</span><span> </span><span>=</span><span> </span><span>&quot;binomial&quot;</span><span>)</span><span>
</span><span>y</span><span> </span><span>&lt;-</span><span> </span><span>ifelse</span><span>(</span><span>versicolor_virginica</span><span>$</span><span>Species</span><span> </span><span>==</span><span> </span><span>&quot;versicolor&quot;</span><span>,</span><span> </span><span>0</span><span>,</span><span> </span><span>1</span><span>)</span><span>
</span><span>y_pred</span><span> </span><span>&lt;-</span><span> </span><span>logistic_reg_fit</span><span>$</span><span>fitted.values</span><span>
</span><span># get TPR and FPR at different values of the decision threshold</span><span>
</span><span>threshold</span><span> </span><span>&lt;-</span><span> </span><span>seq</span><span>(</span><span>0</span><span>,</span><span> </span><span>1</span><span>,</span><span> </span><span>length</span><span> </span><span>=</span><span> </span><span>100</span><span>)</span><span>
</span><span>FPR</span><span> </span><span>&lt;-</span><span> </span><span>sapply</span><span>(</span><span>threshold</span><span>,</span><span>
</span><span>function</span><span>(</span><span>thresh</span><span>)</span><span> </span><span>{</span><span>
</span><span>sum</span><span>(</span><span>y_pred</span><span> </span><span>>=</span><span> </span><span>thresh</span><span> </span><span>&amp;</span><span> </span><span>y</span><span> </span><span>!=</span><span> </span><span>1</span><span>)</span><span> </span><span>/</span><span> </span><span>sum</span><span>(</span><span>y</span><span> </span><span>!=</span><span> </span><span>1</span><span>)</span><span>
</span><span>})</span><span>
</span><span>TPR</span><span> </span><span>&lt;-</span><span> </span><span>sapply</span><span>(</span><span>threshold</span><span>,</span><span>
</span><span>function</span><span>(</span><span>thresh</span><span>)</span><span> </span><span>{</span><span>
</span><span>sum</span><span>(</span><span>y_pred</span><span> </span><span>>=</span><span> </span><span>thresh</span><span> </span><span>&amp;</span><span> </span><span>y</span><span> </span><span>==</span><span> </span><span>1</span><span>)</span><span> </span><span>/</span><span> </span><span>sum</span><span>(</span><span>y</span><span> </span><span>==</span><span> </span><span>1</span><span>)</span><span>
</span><span>})</span><span>
</span><span># plot an ROC curve</span><span>
</span><span>plot</span><span>(</span><span>FPR</span><span>,</span><span> </span><span>TPR</span><span>)</span><span>
</span><span>lines</span><span>(</span><span>FPR</span><span>,</span><span> </span><span>TPR</span><span>)</span><span>
</span></code></pre>
</div>
</div>
<p>A rather ugly ROC curve emerges:</p>
<p><img alt="ROC curve R example" title="An ugly ROC curve" src="https://i1.wp.com/www.alexejgossmann.com/images/20180124-AUC/ROC.png?w=456" data-recalc-dims="1"></p>
<p>The area under the ROC curve, or AUC, seem like a nice heuristic to evaluate and compare the overall performance of classification models independent of the exact decision threshold chosen. But there’s more to it.</p>
<h2>Probabilistic interpretation</h2>
<p>As above, assume that we are looking at a dataset where we want to distinguish data points of <em>type 0</em> from those of <em>type 1</em>. Consider a classification algorithm that assigns to a random observation <img src="http://latex.codecogs.com/png.latex?%5Cmathbf%7Bx%7D%5Cin%5Cmathbb%7BR%7D%5Ep%5Cinline"> a score (or probability) <img src="http://latex.codecogs.com/png.latex?%5Chat%7Bp%7D(%5Cmathbf%7Bx%7D)%20%5Cin%20%5B0,1%5D%5Cinline"> signifying membership in <em>class 1</em>. If the final classification between <em>class 1</em> and <em>class 0</em> is determined by a decision threshold <img src="http://latex.codecogs.com/png.latex?t%5Cin%5B0,%201%5D%5Cinline">, then the <em>true positive rate</em> (a.k.a. <em>sensitivity</em> or <em>recall</em>) can be written as a conditional probability</p>
<div><img alt="T(t) := P[hat{p}(mathbf{x}) &gt; t ,|, mathbf{x},text{belongs to class 1}],” /&gt;&lt;/div&gt;
&lt;p&gt;and the &lt;em&gt;false positive rate&lt;/em&gt; (or &lt;em&gt;1 – specificity&lt;/em&gt;) can be written as&lt;/p&gt;
&lt;div style=" src="http://latex.codecogs.com/png.latex?T(t)%20:=%20P%5B%5Chat%7Bp%7D(%5Cmathbf%7Bx%7D)%20%3E%20t%20%5C,%7C%5C,%20%5Cmathbf%7Bx%7D%5C,%5Ctext%7Bbelongs%20to%20class%201%7D%5D,"><img src="http://latex.codecogs.com/png.latex?F(t)%20:=%20P%5B%5Chat%7Bp%7D(%5Cmathbf%7Bx%7D)%20%3E%20t%20%5C,%7C%5C,%20%5Cmathbf%7Bx%7D%5C,%5Ctext%7Bdoes%20not%20belong%20to%20class%201%7D%5D." alt="F(t) := P[hat{p}(mathbf{x}) &gt; t ,|, mathbf{x},text{does not belong to class 1}].” /&gt;&lt;/div&gt;
&lt;p&gt;For brevity of notation let’s say &lt;img src="> instead of “<img src="http://latex.codecogs.com/png.latex?%5Cmathbf%7Bx%7D%5Cinline"> belongs to class 1”, and <img src="http://latex.codecogs.com/png.latex?y(%5Cmathbf%7Bx%7D)=0%5Cinline"> instead of “<img src="http://latex.codecogs.com/png.latex?%5Cmathbf%7Bx%7D%5Cinline"> doesn’t belong to class 1”.
<p>The ROC curve simply plots <img src="http://latex.codecogs.com/png.latex?T(t)%5Cinline"> against <img src="http://latex.codecogs.com/png.latex?F(t)%5Cinline"> while varying <img src="http://latex.codecogs.com/png.latex?t%5Cinline"> from 0 to 1.<br>
Thus, if we view <img src="http://latex.codecogs.com/png.latex?T%5Cinline"> as a function of <img src="http://latex.codecogs.com/png.latex?F%5Cinline">, the AUC can be rewritten as follows.</p>
<p></p>
<p>So, given a randomly chosen observation <img src="http://latex.codecogs.com/png.latex?%5Cmathbf%7Bx%7D%5Cinline"> belonging to <em>class 1</em>, and a randomly chosen observation <img src="http://latex.codecogs.com/png.latex?%5Cmathbf%7Bx%5E%7B%5Cprime%7D%7D%5Cinline"> belonging to <em>class 0</em>, the AUC is the probability that the evaluated classification algorithm will assign a higher score to <img src="http://latex.codecogs.com/png.latex?%5Cmathbf%7Bx%7D%5Cinline"> than to <img src="http://latex.codecogs.com/png.latex?%5Cmathbf%7Bx%5E%7B%5Cprime%7D%7D%5Cinline">, i.e., the conditional probability of <img src="http://latex.codecogs.com/png.latex?%5Chat%7Bp%7D(%5Cmathbf%7Bx%7D)%20%3e%20%5Chat%7Bp%7D(%5Cmathbf%7Bx%5E%7B%5Cprime%7D%7D)%5Cinline">.</p>
<p>In other words, if the classification algorithm distinguishes “positive” and “negative” examples (e.g., disease status), then</p>
<blockquote>
<p>AUC is the probability of correct ranking of a random “positive”-“negative” pair.</p>
</blockquote>
<h2>Computing AUC</h2>
<p>The above probabilistic interpretation suggest a simple formula to compute AUC on a finite sample.</p>
<blockquote>
<p>Among all “positive”-“negative” pairs in the dataset compute the proportion of those which are ranked correctly by the evaluated classification algorithm.</p>
</blockquote>
<p>Here is an inefficient implementation using results from the above logistic regression example:</p>
<div>
<div>
<pre><code><span>s</span><span> </span><span>&lt;-</span><span> </span><span>0</span><span>
</span><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>which</span><span>(</span><span>y</span><span> </span><span>==</span><span> </span><span>1</span><span>))</span><span> </span><span>{</span><span>
</span><span>for</span><span> </span><span>(</span><span>j</span><span> </span><span>in</span><span> </span><span>which</span><span>(</span><span>y</span><span> </span><span>==</span><span> </span><span>0</span><span>))</span><span> </span><span>{</span><span>
</span><span>if</span><span> </span><span>(</span><span>y_pred</span><span>[</span><span>i</span><span>]</span><span> </span><span>></span><span> </span><span>y_pred</span><span>[</span><span>j</span><span>])</span><span> </span><span>{</span><span>
</span><span>s</span><span> </span><span>&lt;-</span><span> </span><span>s</span><span> </span><span>+</span><span> </span><span>1</span><span>
</span><span>}</span><span> </span><span>else</span><span> </span><span>if</span><span> </span><span>(</span><span>y_pred</span><span>[</span><span>i</span><span>]</span><span> </span><span>==</span><span> </span><span>y_pred</span><span>[</span><span>j</span><span>])</span><span> </span><span>{</span><span>
</span><span>s</span><span> </span><span>&lt;-</span><span> </span><span>s</span><span> </span><span>+</span><span> </span><span>0.5</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span><span>}</span><span>
</span><span>s</span><span> </span><span>&lt;-</span><span> </span><span>s</span><span> </span><span>/</span><span> </span><span>(</span><span>sum</span><span>(</span><span>y</span><span> </span><span>==</span><span> </span><span>1</span><span>)</span><span> </span><span>*</span><span> </span><span>sum</span><span>(</span><span>y</span><span> </span><span>==</span><span> </span><span>0</span><span>))</span><span>
</span><span>s</span><span>
</span><span># [1] 0.7918</span><span>
</span></code></pre>
</div>
</div>
<p>The <strong>proportion of correctly ranked “positive”-“negative” pairs</strong> yields estimated <img src="http://latex.codecogs.com/png.latex?%5Cmathrm%7BAUC%7D%20=%200.7918%5Cinline">.</p>
<p>We can compare this value to the area under the ROC curve computed with the <a rel="nofollow" href="https://en.wikipedia.org/wiki/Trapezoidal_rule" target="_blank">trapezoidal rule</a>.</p>
<div>
<div>
<pre><code><span>s</span><span> </span><span>&lt;-</span><span> </span><span>0</span><span>
</span><span>for</span><span> </span><span>(</span><span>i</span><span> </span><span>in</span><span> </span><span>1</span><span>:</span><span>(</span><span>length</span><span>(</span><span>FPR</span><span>)</span><span> </span><span>-</span><span> </span><span>1</span><span>))</span><span> </span><span>{</span><span>
</span><span>dFPR</span><span> </span><span>&lt;-</span><span> </span><span>abs</span><span>(</span><span>FPR</span><span>[</span><span>i</span><span>+1</span><span>]</span><span> </span><span>-</span><span> </span><span>FPR</span><span>[</span><span>i</span><span>])</span><span>
</span><span>s</span><span> </span><span>&lt;-</span><span> </span><span>s</span><span> </span><span>+</span><span> </span><span>0.5</span><span> </span><span>*</span><span> </span><span>dFPR</span><span> </span><span>*</span><span> </span><span>(</span><span>TPR</span><span>[</span><span>i</span><span>+1</span><span>]</span><span> </span><span>+</span><span> </span><span>TPR</span><span>[</span><span>i</span><span>])</span><span>
</span><span>}</span><span>
</span><span>s</span><span>
</span><span># [1] 0.7922</span><span>
</span></code></pre>
</div>
</div>
<p><strong>Trapezoidal rule</strong> yields estimated <img src="http://latex.codecogs.com/png.latex?%5Cmathrm%7BAUC%7D%20=%200.7922%5Cinline">. The difference of <img src="http://latex.codecogs.com/png.latex?0.0004%5Cinline"> can be explained by the fact that we evaluated the ROC curve at only 100 points.</p>
<p>Since there is a minor disagreement, let’s use some standard R package to compute AUC.</p>
<div>
<div>
<pre><code><span>library</span><span>(</span><span>ROCR</span><span>)</span><span>
</span><span>pred</span><span> </span><span>&lt;-</span><span> </span><span>prediction</span><span>(</span><span>y_pred</span><span>,</span><span> </span><span>y</span><span>)</span><span>
</span><span>auc</span><span> </span><span>&lt;-</span><span> </span><span>as.numeric</span><span>(</span><span>performance</span><span>(</span><span>pred</span><span>,</span><span> </span><span>measure</span><span> </span><span>=</span><span> </span><span>&quot;auc&quot;</span><span>)</span><span>@</span><span>y.values</span><span>)</span><span>
</span><span>auc</span><span>
</span><span># [1] 0.7918</span><span>
</span></code></pre>
</div>
</div>
<p>Same as the proportion of correctly ranked pairs! <img alt="😀" src="https://s.w.org/images/core/emoji/2.3/72x72/1f600.png" class="wp-smiley"></p>
<h2>So what? Why care about AUC anyway?</h2>
<ul>
<li>It has a fu*** nice probabilistic meaning!</li>
</ul>
<p>Besides, as a measure of classification performance AUC has many advantages compared to other “single number” performance measures:</p>
<ul>
<li>Independence of the decision threshold.</li>
<li>Invariance to prior class probabilities or class prevalence in the data.</li>
<li>Can choose/change a decision threshold based on cost-benefit analysis after model training.</li>
<li>Extensively used in machine learning, and in medical research – and that for good reasons, as for example explained in an <a rel="nofollow" href="https://lukeoakdenrayner.wordpress.com/2017/12/06/do-machines-actually-beat-doctors-roc-curves-and-performance-metrics/" target="_blank">excellent blog post on deep learning research in medicine by Luke Oakden-Rayner</a>.</li>
</ul>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.alexejgossmann.com/auc/"> Alexej's blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/f364NY6MjN4" height="1"></a></div></div>