<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/interpretation-of-the-auc/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://datascienceplus.com/interpretation-of-the-auc/"> R Programming – DataScience+</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>The AUC* or concordance statistic <em>c</em> is the most commonly used measure for diagnostic accuracy of quantitative tests. It is a discrimination measure which tells us how well we can classify patients in two groups: those with and those without the outcome of interest. Since the measure is based on ranks, it is not sensitive to systematic errors in the calibration of the quantitative tests. </p>
<p>It is very well known that a test with no better accuracy than chance has an AUC of 0.5, and a test with perfect accuracy has an AUC of 1. But what is the exact interpretation of an AUC of for example 0.88? Did you know that the AUC is completely equivalent with the Mann-Whitney U test statistic? </p>
<p>*AUC: the Area Under the Curve (AUC) of the Receiver Operating Characteristic (ROC) curve.</p>
<h2>Example</h2>
<p>Around 27% of the patients with liver cirrhosis will develop Hepatocellular Carcinoma (HCC) within 5 years of follow-up. With our biomarker “peakA” we would like to predict which patients will develop HCC, and which won't. We will assess the diagnostic accuracy of biomarker “peakA” using the AUC. </p>
<p>To keep things visually clear, we suppose we have a dataset of only 12 patients. Four patients did develop HCC (the “cases”) and 8 didn't (the “controls”). (fictive data)</p>
<table>
<thead>
<tr>
<th align="right">HCC</th>
<th align="right">Biomarker_value</th>
</tr>
</thead>
<tbody>
<tr>
<td align="right">0</td>
<td align="right">1.063</td>
</tr>
<tr>
<td align="right">1</td>
<td align="right">1.132</td>
</tr>
<tr>
<td align="right">1</td>
<td align="right">1.122</td>
</tr>
<tr>
<td align="right">1</td>
<td align="right">1.058</td>
</tr>
<tr>
<td align="right">0</td>
<td align="right">0.988</td>
</tr>
<tr>
<td align="right">0</td>
<td align="right">1.182</td>
</tr>
<tr>
<td align="right">0</td>
<td align="right">1.037</td>
</tr>
<tr>
<td align="right">0</td>
<td align="right">1.052</td>
</tr>
<tr>
<td align="right">0</td>
<td align="right">0.925</td>
</tr>
<tr>
<td align="right">1</td>
<td align="right">1.232</td>
</tr>
<tr>
<td align="right">0</td>
<td align="right">0.911</td>
</tr>
<tr>
<td align="right">0</td>
<td align="right">0.967</td>
</tr>
</tbody>
</table>
<p>The AUC can be defined as “The probability that a randomly selected case will have a higher test result than a randomly selected control”. Let's use this definition to calculate and visualize the estimated AUC.<br>
In the figure below, the cases are presented on the left and the controls on the right.<br>
Since we have only 12 patients, we can easily visualize all 32 possible combinations of one case and one control. (Rcode below) </p>
<p><a rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2018/09/dotplot.png?ssl=1" target="_blank"><img sizes="(max-width: 1024px) 100vw, 1024px" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2018/09/dotplot-1024x1024.png?w=450&ssl=1" alt="" class="wp-image-16872" data-recalc-dims="1"></a></p>
<p>Those 32 different pairs of cases and controls are represented by lines on the plot above. 28 of them are indicated in green. For those pairs, the value for “PeakA” is higher for the case compared to the control. The remaining 4 pairs are indicated in blue. The AUC can be estimated as the proportion of pairs for which the case has a higher value compared to the control. Thus, the estimated AUC is the proportion of green lines or 28/32 = 0.875. This visualization might help to understand the concept of an AUC. Besides this educational purpose, this type of plot is not very useful. Hopefully, the sample size of your study is much larger than 12 patients. And in that situation, this type of plot will become very crowded.</p>
<h2>The ROC curve</h2>
<p>Now let's verify that the AUC is indeed equal to 0.875 in a classical way, by plotting a ROC curve and calculating the estimated AUC using the ROCR package.</p>
<p>The ROC curve plots the False Positive Rate (FPR) on the X-axis and the True Postive Rate (TPR) on the Y-axis for all possible thresholds (or cutoff values). </p>
<ul>
<li>True Positive Rate (TPR) or sensitivity: the proportion of actual positives that are correctly identified as such.</li>
<li>True Negative Rate (TNR) or specificiy: the proportion of actual negatives that are correctly identified as such.</li>
<li>False Positive Rate (FPR) or 1-specificity: the proportion of actual negatives that are wrongly identified as positives.</li>
</ul>
<pre><code>library(ROCR)
pred &lt;- prediction(df$Biomarker_value, df$HCC )
perf &lt;- performance(pred,&quot;tpr&quot;,&quot;fpr&quot;)
plot(perf,col=&quot;black&quot;)
abline(a=0, b=1, col=&quot;#8AB63F&quot;)
</code></pre>
<p><a rel="nofollow" href="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2018/09/ROC.png?ssl=1" target="_blank"><img sizes="(max-width: 1024px) 100vw, 1024px" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2018/09/ROC-1024x1024.png?w=450&ssl=1" alt="" class="wp-image-16874" data-recalc-dims="1"></a></p>
<p>The green line represents a completely uninformative test, which corresponds to an AUC of 0.5. A curve pulled close to the upper left corner indicates (an AUC close to 1 and thus) a better performing test. The ROC curve does not show the cutoff values </p>
<p>The ROCR package also allows to calculate the estimated AUC:</p>
<pre><code>auc&lt;- performance( pred,  c(&quot;auc&quot;))
unlist(slot(auc , &quot;y.values&quot;))</code>
<em>[1] 0.875
</em></pre>
<p>The estimated AUC based on this ROC curve is indeed equal to 0.875, the proportion of pairs for which the value of “PeakA” is larger for HCC compared to NoHCC.</p>
<h2>Relation to cutoff points of the biomarker</h2>
<p>Visualizing the sensitivity and specificity as a function of the cutoff points of the biomarker results in a plot that is at least as informative as a ROC curve and (in my opinion) easier to interpret. The plot can be created using the ROCR package.</p>
<pre><code>library(ROCR)
testy &lt;- performance(pred,&quot;tpr&quot;,&quot;fpr&quot;)
</code></pre>
<p>Using the <em>str()</em> function, we see that the following slots are part of the testy object: </p>
<ul>
<li>alpha.values: Cutoff</li>
<li>x.values: Specificity or True Negative Rate</li>
<li>y.values: Sensitivity or True Positive Rate</li>
</ul>
<pre><code>plot(testy@alpha.values[[1]], testy@x.values[[1]], type='n',  
     xlab='Cutoff points of the biomarker', 
     ylab='sensitivity or specificity')
lines(testy@alpha.values[[1]], testy@y.values[[1]], 
      type='s', col=&quot;#1A425C&quot;, lwd=2)
lines(testy@alpha.values[[1]], 1-testy@x.values[[1]], 
      type='s', col=&quot;#8AB63F&quot;, lwd=2)
legend(1.11,.85, c('sensitivity', 'specificity'), 
       lty=c(1,1), col=c(&quot;#1A425C&quot;, &quot;#8AB63F&quot;), cex=.9, bty='n')
</code></pre>
<p><a rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2018/09/spec_sens.png?ssl=1" target="_blank"><img sizes="(max-width: 1024px) 100vw, 1024px" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2018/09/spec_sens-1024x1024.png?w=450&ssl=1" alt="" class="wp-image-16876" data-recalc-dims="1"></a></p>
<p>The plot shows how the sensitivity increases as the specificity decreases and vice versa, in relation to the possible cutoff points of the biomarker.</p>
<h2>Mann-Whitney U test statistic</h2>
<p>The Mann-Whitney U test statistic (or Wilcoxon or Kruskall-Wallis test statistic) is equivalent to the AUC (Mason, 2002).<br>
The AUC can be calculated from the output of the <em>wilcox.test()</em> function:</p>
<pre><code>wt &lt;-wilcox.test(data=df, df$Biomarker_value ~ df$HCC)
1 - wt$statistic/(sum(df$HCC==1)*sum(df$HCC==0))</code>
<em> W 0.875 </em></pre>
<p>The p-value of the Mann-Whitney U test can thus safely be used to test whether the AUC differs significantly from 0.5 (AUC of an uninformative test). </p>
<pre><code>wt &lt;-wilcox.test(data=df, df$Biomarker_value ~ df$HCC)
wt$p.value</code>
<em>[1] 0.04848485
</em></pre>
<h2>Simulation: the completely uninformative test.</h2>
<p>Now, let's have a look how our plots look like if our biomarker is not informative at all. </p>
<p>Data creation: </p>
<pre><code>#simulation of the data
set.seed(12345)
HCC &lt;- rbinom (n=12, size=1, prob=0.27)
Biomarker_value &lt;- rnorm (12,mean=1,sd=0.1) + HCC*0 
# replacing the zero by a value would make the test informative
df&lt;-data.frame (HCC, Biomarker_value)
library(knitr)
kable(head(df))
</code></pre>
<table>
<thead>
<tr>
<th align="right">HCC</th>
<th align="right">Biomarker_value</th>
</tr>
</thead>
<tbody>
<tr>
<td align="right">0</td>
<td align="right">1.0630099</td>
</tr>
<tr>
<td align="right">1</td>
<td align="right">0.9723816</td>
</tr>
<tr>
<td align="right">1</td>
<td align="right">0.9715840</td>
</tr>
<tr>
<td align="right">1</td>
<td align="right">0.9080678</td>
</tr>
<tr>
<td align="right">0</td>
<td align="right">0.9883752</td>
</tr>
<tr>
<td align="right">0</td>
<td align="right">1.1817312</td>
</tr>
</tbody>
</table>
<p>The function <em>expand.grid()</em> is used to create all possible combinations of one case and one control:</p>
<pre><code>newdf&lt;- expand.grid (Biomarker_value [df$HCC==0],Biomarker_value [df$HCC==1])
colnames(newdf)&lt;- c(&quot;NoHCC&quot;, &quot;HCC&quot;)
newdf$Pair &lt;- seq(1,dim(newdf)[1])
</code></pre>
<p>For each pair the values of the biomarker are compared between case and control:</p>
<pre><code>newdf$Comparison  &lt;- 1*(newdf$HCC>newdf$NoHCC)
mean(newdf$Comparison)</code>
<em>[1] 0.40625
</em></pre>
<pre><code>newdf$Comparison&lt;-factor(newdf$Comparison, labels=c(&quot;HCC>NoHCC&quot;,&quot;HCC&lt;=NoHCC&quot;))
kable (head(newdf,4))
</code></pre>
<table>
<thead>
<tr>
<th align="right">NoHCC</th>
<th align="right">HCC</th>
<th align="right">Pair</th>
<th align="left">Comparison</th>
</tr>
</thead>
<tbody>
<tr>
<td align="right">1.0630099</td>
<td align="right">0.9723816</td>
<td align="right">1</td>
<td align="left">HCC>NoHCC</td>
</tr>
<tr>
<td align="right">0.9883752</td>
<td align="right">0.9723816</td>
<td align="right">2</td>
<td align="left">HCC>NoHCC</td>
</tr>
<tr>
<td align="right">1.1817312</td>
<td align="right">0.9723816</td>
<td align="right">3</td>
<td align="left">HCC>NoHCC</td>
</tr>
<tr>
<td align="right">1.0370628</td>
<td align="right">0.9723816</td>
<td align="right">4</td>
<td align="left">HCC>NoHCC</td>
</tr>
</tbody>
</table>
<pre><code>library(data.table)
longdf = melt(newdf, id.vars = c(&quot;Pair&quot;, &quot;Comparison&quot;),
              variable.name = &quot;Group&quot;,
                measure.vars = c(&quot;HCC&quot;, &quot;NoHCC&quot;))

lab&lt;-paste(&quot;AUC = Proportion \n of green lines \nAUC=&quot;, round(table(newdf$Comparison)[2]/sum(table(newdf$Comparison)),3))
library(ggplot2)
fav.col=c(&quot;#1A425C&quot;, &quot;#8AB63F&quot;)
ggplot(longdf, aes(x=Group, y=value))+geom_line(aes(group=Pair, col=Comparison)) + 
        scale_color_manual(values=fav.col)+theme_bw() + 
        ylab(&quot;Biomarker value&quot;) + geom_text(x=0.75,y=0.95,label=lab) + 
        geom_point(shape=21, size=2) + 
        theme(legend.title=element_blank(), legend.position=&quot;bottom&quot;)
</code></pre>
<p><a rel="nofollow" href="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2018/09/dotplot2.png?ssl=1" target="_blank"><img sizes="(max-width: 1024px) 100vw, 1024px" src="https://i2.wp.com/datascienceplus.com/wp-content/uploads/2018/09/dotplot2-1024x1024.png?w=450&ssl=1" alt="" class="wp-image-16873" data-recalc-dims="1"></a></p>
<pre><code>library(ROCR)
pred &lt;- prediction(df$Biomarker_value, df$HCC )
perf &lt;- performance(pred,&quot;tpr&quot;,&quot;fpr&quot;)
plot(perf,col=&quot;black&quot;)
abline(a=0, b=1, col=&quot;#8AB63F&quot;)
</code></pre>
<p><a rel="nofollow" href="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2018/09/ROC2.png?ssl=1" target="_blank"><img sizes="(max-width: 1024px) 100vw, 1024px" src="https://i0.wp.com/datascienceplus.com/wp-content/uploads/2018/09/ROC2-1024x1024.png?w=450&ssl=1" alt="" class="wp-image-16875" data-recalc-dims="1"></a></p>
<p>Calculating the AUC: </p>
<pre><code>auc&lt;- performance( pred,  c(&quot;auc&quot;))
unlist(slot(auc , &quot;y.values&quot;))</code>
<em>[1] 0.40625
</em></pre>
<p>Sensitivity and specificity as a function of the cutoff points of the biomarker:</p>
<pre><code>library(ROCR)
testy &lt;- performance(pred,&quot;tpr&quot;,&quot;fpr&quot;)
plot(testy@alpha.values[[1]], testy@x.values[[1]], type='n', xlab='Cutoff points of the biomarker', ylab='sensitivity or specificity')
lines(testy@alpha.values[[1]], testy@y.values[[1]], type='s', col=&quot;#1A425C&quot;)
lines(testy@alpha.values[[1]], 1-testy@x.values[[1]], type='s', col=&quot;#8AB63F&quot;)
legend(1.07,.85, c('sensitivity', 'specificity'), lty=c(1,1), col=c(&quot;#1A425C&quot;, &quot;#8AB63F&quot;), cex=.9, bty='n')
</code></pre>
<p><a rel="nofollow" href="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2018/09/spec_sens_2.png?ssl=1" target="_blank"><img sizes="(max-width: 1024px) 100vw, 1024px" src="https://i1.wp.com/datascienceplus.com/wp-content/uploads/2018/09/spec_sens_2-1024x1024.png?w=450&ssl=1" alt="" class="wp-image-16877" data-recalc-dims="1"></a></p>
<p>Equivalence with the Mann-Whitney U test:</p>
<pre><code>wt &lt;-wilcox.test(data=df, df$Biomarker_value ~ df$HCC)
1 - wt$statistic/(sum(df$HCC==1)*sum(df$HCC==0))</code>
<em> W 0.40625 </em></pre>
<pre><code>wt$p.value</code>
<em>[1] 0.6828283
</em></pre>
<h2>General remarks on the AUC</h2>
<p>Often, a combination of new markers is selected from a large set. This can result in overoptimistic expectations of the marker's performance. Any performance measure should be estimated with correction for optimism, for example by applying cross-validation or bootstrap resampling. However, validation in fully independent, external data is the best way to validate a new marker.</p>
<p>When we want to assess the incremental value of an additional marker (e.g. molecular, genetic, imaging) to an existing model, the increase of the AUC can be reported.</p>
<h2>References</h2>
<ul>
<li>Mason, S. J. and Graham, N. E. (2002), Areas beneath the relative operating characteristics (ROC) and relative operating levels (ROL) curves: Statistical significance and interpretation. Q.J.R. Meteorol. Soc., 128: 2145-2166.</li>
<li>Steyerberg, Ewout W. et al. “Assessing the Performance of Prediction Models: A Framework for Some Traditional and Novel Measures.” Epidemiology (Cambridge, Mass.) 21.1 (2010): 128-138.</li>
<li> <a rel="nofollow" href="https://www.unc.edu/courses/2008fall/ecol/563/001/docs/lectures/lecture19.htm" target="_blank">Tutorial: Lecture 19 (lab 9)</a></li>
<li>Xavier Verhelst, Dieter Vanderschaeghe, Laurent Castéra, Tom Raes, Anja Geerts, Claire Francoz, Roos Colman, François Durand, Nico Callewaert, and Hans Van Vlierberghe (2017). A Glycomics-Based Test Predicts the Development of Hepatocellular Carcinoma in Cirrhosis. Clin Cancer Res (23) (11) 2750-2758</li>
</ul>
<p><strong>
<p>Related Post</p>
<p></p></strong>
<ol>
<li><a rel="nofollow" href="https://datascienceplus.com/simple-experiments-with-smoothed-scatterplots/" title="Permanent Link toSimple Experiments with Smoothed Scatterplots" target="_blank">Simple Experiments with Smoothed Scatterplots</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/understanding-the-covariance-matrix/" title="Permanent Link toUnderstanding the Covariance Matrix" target="_blank">Understanding the Covariance Matrix</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/six-sigma-dmaic-series-in-r-part-3/" title="Permanent Link toSix Sigma DMAIC Series in R – Part 3" target="_blank">Six Sigma DMAIC Series in R – Part 3</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/prediction-interval-the-wider-sister-of-confidence-interval/" title="Permanent Link toPrediction Interval, the wider sister of Confidence Interval" target="_blank">Prediction Interval, the wider sister of Confidence Interval</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/six-sigma-dmaic-series-in-r-part-2/" title="Permanent Link toSix Sigma DMAIC Series in R – Part 2" target="_blank">Six Sigma DMAIC Series in R – Part 2</a></li>
</ol>
<p><strong>
<p>Categories</p>
<p></p></strong>
<ol>
<li><a rel="nofollow" href="https://datascienceplus.com/category/basic-statistics/" title="Permanent Link toBasic Statistics" target="_blank">Basic Statistics</a></li>
</ol>
<p><strong>
<p>Tags</p>
<p></p></strong>
<ol>
<li><a rel="nofollow" href="https://datascienceplus.com/tag/auc/" title="Permanent Link toAUC" target="_blank">AUC</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/tag/data-visualisation/" title="Permanent Link toData Visualisation" target="_blank">Data Visualisation</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/tag/logistic-regression/" title="Permanent Link toLogistic Regression" target="_blank">Logistic Regression</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/tag/rstats/" title="Permanent Link toR Programming" target="_blank">R Programming</a></li>
<li><a rel="nofollow" href="https://datascienceplus.com/tag/roc/" title="Permanent Link toROC" target="_blank">ROC</a></li>
</ol>
<ul></ul>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://datascienceplus.com/interpretation-of-the-auc/"> R Programming – DataScience+</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/TxlTmZ1DjCA" height="1"></a></div>