<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="https://www.r-bloggers.com/sigr-simple-significance-reporting/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://www.win-vector.com/blog/2017/03/sigr-simple-significance-reporting/"> R – Win-Vector Blog</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p><a target="_blank" rel="nofollow" href="https://cran.r-project.org/package=sigr"><code>sigr</code></a> is a simple <a target="_blank" rel="nofollow" href="https://cran.r-project.org/"><code>R</code></a> package that conveniently formats a few statistics and their significance tests. This allows the analyst to use the correct test no matter what modeling package or procedure they use.</p>
<p><img title="sigr.png" alt="Sigr" src="https://i0.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/03/sigr.png?w=660" border="0" data-recalc-dims="1"><span></span></p>
<p>
<h2>Model Example</h2>
<p>Let’s take as our example the following linear relation between <code>x</code> and <code>y</code>:</p>
<div>
<pre><code><span>library</span>(<span>'sigr'</span>)
<span>set.seed</span>(<span>353525</span>)
d &lt;-<span> </span><span>data.frame</span>(<span>x=</span> <span>rnorm</span>(<span>5</span>))
d$y &lt;-<span> </span><span>2</span>*d$x +<span> </span><span>0.5</span>*<span>rnorm</span>(<span>nrow</span>(d))</code></pre>
</div>
<p><code>stats::lm()</code> has among the most complete summaries of all models in <code>R</code>, so we easily get can see the quality of fit and its significance:</p>
<div>
<pre><code>model &lt;-<span> </span><span>lm</span>(y~x, <span>d=</span>d)
<span>summary</span>(model)</code></pre>
</div>
<pre><code>## 
## Call:
## lm(formula = y ~ x, data = d)
## 
## Residuals:
##       1       2       3       4       5 
##  0.3292  0.3421  0.0344 -0.1671 -0.5387 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)   
## (Intercept)   0.4201     0.2933   1.432  0.24747   
## x             2.1117     0.2996   7.048  0.00587 **
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.4261 on 3 degrees of freedom
## Multiple R-squared:  0.943,  Adjusted R-squared:  0.9241 
## F-statistic: 49.67 on 1 and 3 DF,  p-value: 0.005871</code></pre>
<p><code>sigr::wrapFTest()</code> can render the relevant model quality summary.</p>
<div>
<pre><code><span>cat</span>(<span>render</span>(<span>wrapFTest</span>(model),
    <span>pLargeCutoff=</span><span>1</span>))</code></pre>
</div>
<p><strong>F Test</strong> summary: (<i>R<sup>2</sup></i>=0.94, <em>F</em>(1,3)=50, <em>p</em>=0.0059).</p>
<p>Note: <code>sigr</code> reports the un-adjusted R-squared, as this is the one that significance is reported for in <code>summary::lm()</code>.</p>
<p><code>sigr</code> also carries around the important summary components for use in code.</p>
<div>
<pre><code><span>unclass</span>(<span>wrapFTest</span>(model))</code></pre>
</div>
<pre><code>## $test
## [1] &quot;F Test&quot;
## 
## $numdf
## [1] 1
## 
## $dendf
## [1] 3
## 
## $FValue
## [1] 49.66886
## 
## $R2
## [1] 0.9430403
## 
## $pValue
## [1] 0.005871236</code></pre>
<p>In this function <code>sigr</code> is much like <code>broom::glance()</code> or <code>modelr::rsquare()</code>.</p>
<div>
<pre><code>broom::<span>glance</span>(model)</code></pre>
</div>
<pre><code>##   r.squared adj.r.squared     sigma statistic     p.value df    logLik
## 1 0.9430403     0.9240538 0.4261232  49.66886 0.005871236  2 -1.552495
##       AIC      BIC deviance df.residual
## 1 9.10499 7.933304 0.544743           3</code></pre>
<div>
<pre><code>modelr::<span>rsquare</span>(model, d)</code></pre>
</div>
<pre><code>## [1] 0.9430403</code></pre>
<p>This is something like what is reported by <code>caret::train()</code> (where <code>caret</code> controls the model training process).</p>
<div>
<pre><code>cr &lt;-<span> </span>caret::<span>train</span>(y~x, d, 
                   <span>method =</span> <span>'lm'</span>)</code></pre>
</div>
<pre><code>## Warning in nominalTrainWorkflow(x = x, y = y, wts = weights, info =
## trainInfo, : There were missing values in resampled performance measures.</code></pre>
<div>
<pre><code>cr$results</code></pre>
</div>
<pre><code>##   intercept      RMSE  Rsquared    RMSESD   RsquaredSD
## 1      TRUE 0.9631662 0.9998162 0.6239989 0.0007577834</code></pre>
<div>
<pre><code><span>summary</span>(cr$finalModel)</code></pre>
</div>
<pre><code>## 
## Call:
## lm(formula = .outcome ~ ., data = dat)
## 
## Residuals:
##      X1      X2      X3      X4      X5 
##  0.3292  0.3421  0.0344 -0.1671 -0.5387 
## 
## Coefficients:
##             Estimate Std. Error t value Pr(>|t|)   
## (Intercept)   0.4201     0.2933   1.432  0.24747   
## x             2.1117     0.2996   7.048  0.00587 **
## ---
## Signif. codes:  0 '***' 0.001 '**' 0.01 '*' 0.05 '.' 0.1 ' ' 1
## 
## Residual standard error: 0.4261 on 3 degrees of freedom
## Multiple R-squared:  0.943,  Adjusted R-squared:  0.9241 
## F-statistic: 49.67 on 1 and 3 DF,  p-value: 0.005871</code></pre>
<p>(I presume <code>cr$results$Rsquared</code> is a model quality report and not a consistency of cross-validation procedure report. If it is a model quality report it is somehow inflated, perhaps by the resampling procedure. So I apologize for using either <code>caret::train()</code> or its results incorrectly.)</p>
<h2>Data example</h2>
<p>The issues in taking summary statistics (and significances) from models include:</p>
<ul>
<li>Working only from models limits you to models that include the statistic you want.</li>
<li>Working only from models is mostly &quot;in-sample.&quot; You need additional procedures for test or hold-out data.</li>
</ul>
<p>With <code>sigr</code> it is also easy to reconstruct quality and significance from the predictions, no matter where they came from (without needing the model data structures).</p>
<h3>In-sample example</h3>
<div>
<pre><code>d$pred &lt;-<span> </span><span>predict</span>(model, <span>newdata =</span> d)</code></pre>
</div>
<div>
<pre><code><span>cat</span>(<span>render</span>(<span>wrapFTest</span>(d, <span>'pred'</span>, <span>'y'</span>),
    <span>pLargeCutoff=</span><span>1</span>))</code></pre>
</div>
<p><strong>F Test</strong> summary: (<i>R<sup>2</sup></i>=0.94, <em>F</em>(1,3)=50, <em>p</em>=0.0059).</p>
<p>Notice we reconstruct the summary statistic and significance, independent of the model data structures. This means the test is generic and can be used on any regression (modulo informing the significance model of the appropriate number of parameters). It also can be used on held-out or test data.</p>
<p>In this mode it is a lot like <code>ModelMetrics::rmse()</code> or <code>caret::postResample()</code>.</p>
<div>
<pre><code>ModelMetrics::<span>rmse</span>(d$y, d$pred)</code></pre>
</div>
<pre><code>## [1] 0.3300736</code></pre>
<div>
<pre><code>caret::<span>postResample</span>(d$y, d$pred)</code></pre>
</div>
<pre><code>##      RMSE  Rsquared 
## 0.3300736 0.9430403</code></pre>
<h3>Out of sample example</h3>
<p>If we had more data we can look at the quality of the prediction on that data (though you have to take care in understanding the number of degrees of freedom is different for held-out data).</p>
<div>
<pre><code>d2 &lt;-<span> </span><span>data.frame</span>(<span>x=</span> <span>rnorm</span>(<span>5</span>))
d2$y &lt;-<span> </span><span>2</span>*d2$x +<span> </span><span>0.5</span>*<span>rnorm</span>(<span>nrow</span>(d2))
d2$pred &lt;-<span> </span><span>predict</span>(model, <span>newdata =</span> d2)</code></pre>
</div>
<div>
<pre><code><span>cat</span>(<span>render</span>(<span>wrapFTest</span>(d2, <span>'pred'</span>, <span>'y'</span>),
    <span>pLargeCutoff=</span><span>1</span>))</code></pre>
</div>
<p><strong>F Test</strong> summary: (<i>R<sup>2</sup></i>=0.76, <em>F</em>(1,3)=9.6, <em>p</em>=0.054).</p>
<h3>Exotic model example</h3>
<p>We could have used <code>glmnet</code> instead of <code>lm</code>:</p>
<div>
<pre><code><span>library</span>(<span>&quot;glmnet&quot;</span>)</code></pre>
</div>
<div>
<pre><code>d$one &lt;-<span> </span><span>1</span> <span># make sure we have at least 2 columns</span>
xmat &lt;-<span> </span><span>as.matrix</span>(d[, <span>c</span>(<span>'one'</span>,<span>'x'</span>)])
glmnetModel &lt;-<span> </span><span>cv.glmnet</span>(xmat, d$y)</code></pre>
</div>
<pre><code>## Warning: Option grouped=FALSE enforced in cv.glmnet, since &lt; 3 observations
## per fold</code></pre>
<div>
<pre><code><span>summary</span>(glmnetModel)</code></pre>
</div>
<pre><code>##            Length Class  Mode     
## lambda     53     -none- numeric  
## cvm        53     -none- numeric  
## cvsd       53     -none- numeric  
## cvup       53     -none- numeric  
## cvlo       53     -none- numeric  
## nzero      53     -none- numeric  
## name        1     -none- character
## glmnet.fit 12     elnet  list     
## lambda.min  1     -none- numeric  
## lambda.1se  1     -none- numeric</code></pre>
<div>
<pre><code>d$predg &lt;-<span> </span><span>as.numeric</span>(<span>predict</span>(glmnetModel, 
                              <span>newx=</span> xmat))</code></pre>
</div>
<div>
<pre><code><span>cat</span>(<span>render</span>(<span>wrapFTest</span>(d, <span>'predg'</span>, <span>'y'</span>),
    <span>pLargeCutoff=</span><span>1</span>))</code></pre>
</div>
<p><strong>F Test</strong> summary: (<i>R<sup>2</sup></i>=0.91, <em>F</em>(1,3)=30, <em>p</em>=0.012).</p>
<h2>Plotting</h2>
<p>Because <code>sigr</code> can render to &quot;<code>LaTex</code>&quot; it can (when used in conjunction with <code>latex2exp</code>) also produce formatted titles for plots.</p>
<div>
<pre><code><span>library</span>(<span>&quot;ggplot2&quot;</span>)
<span>library</span>(<span>&quot;latex2exp&quot;</span>)


f &lt;-<span> </span><span>paste0</span>(<span>format</span>(model$coefficients[<span>'x'</span>], <span>digits=</span> <span>3</span>), 
            <span>'*x + '</span>,
            <span>format</span>(model$coefficients[<span>'(Intercept)'</span>], <span>digits=</span> <span>3</span>))
title &lt;-<span> </span><span>paste0</span>(<span>&quot;linear y ~ &quot;</span>, f, <span>&quot; relation&quot;</span>)
subtitle &lt;-<span> </span>latex2exp::<span>TeX</span>(<span>render</span>(<span>wrapFTest</span>(d, <span>'pred'</span>, <span>'y'</span>), 
                                          <span>format=</span> <span>'latex'</span>))
<span>ggplot</span>(<span>data=</span>d, <span>mapping=</span><span>aes</span>(<span>x=</span>pred, <span>y=</span>y)) +<span> </span>
<span> </span><span>geom_point</span>() +<span> </span><span>geom_abline</span>(<span>color=</span><span>'blue'</span>) +
<span> </span><span>xlab</span>(f) +
<span> </span><span>ggtitle</span>(title, 
          <span>subtitle=</span> subtitle)</code></pre>
</div>
<div>
<img title="NewImage.png" alt="NewImage" src="https://i2.wp.com/www.win-vector.com/blog/wp-content/uploads/2017/03/NewImage.png?w=660" border="0" data-recalc-dims="1">
</div>
<h2>Conclusion</h2>
<p><code>sigr</code> computes a few statistics or summaries (with standard significance estimates) and returns the calculation in both machine readable and formatted forms. For non-standard summaries <code>sigr</code> includes some resampling methods for significance estimation. For formatting <code>sigr</code> tries to get near the formats specified by &quot;The American Psychological Association.&quot; <code>sigr</code> works with models (such as <code>lm</code> and <code>glm(family='binomial')</code>) or data, and is a small package with few dependencies.</p>
<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://www.win-vector.com/blog/2017/03/sigr-simple-significance-reporting/"> R – Win-Vector Blog</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="https://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="https://www.r-bloggers.com/search/data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="https://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="https://www.r-bloggers.com/search/map">maps</a>, <a title="Animation in R" rel="nofollow" href="https://www.r-bloggers.com/search/animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="https://www.r-bloggers.com/search/sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="https://www.r-bloggers.com/search/SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="https://www.r-bloggers.com/search/eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="https://www.r-bloggers.com/search/git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="https://www.r-bloggers.com/search/hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="https://www.r-bloggers.com/search/regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="https://www.r-bloggers.com/search/PCA">PCA</a>, <a title="Time series" rel="nofollow" href="https://www.r-bloggers.com/search/time+series">time series</a>, <a title="finance trading" rel="nofollow" href="https://www.r-bloggers.com/search/trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/FZht5o6aqjE"></a></div>