<p><div><div><div><div data-width="100" data-show-faces="false" data-send="true" data-layout="button_count" data-height="21" data-href="http://www.r-bloggers.com/exploring-p-values-with-simulations-in-r/"></div></div><div><a data-url="http://www.r-bloggers.com/exploring-p-values-with-simulations-in-r/" data-counturl="http://www.r-bloggers.com/exploring-p-values-with-simulations-in-r/" data-count="horizontal" data-text="Exploring P-values with Simulations in R" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="https://stablemarkets.wordpress.com/2016/05/21/exploring-p-values-with-simulations-in-r/"> R – Stable Markets</a></strong>, and kindly contributed to <a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>The recent flare-up in discussions on p-values inspired me to conduct a brief simulation study.</p>
<p>In particularly, I wanted to illustrate just how p-values vary with different effect and sample sizes.<br>
Here are the details of the simulation. I simulated <img title="n " alt="n " class="latex" src="https://s0.wp.com/latex.php?latex=n+&bg=ffffff&%23038;fg=2b2b2b&%23038;s=0"> draws of my independent variable <img title="X " alt="X " class="latex" src="https://s0.wp.com/latex.php?latex=X+&bg=ffffff&%23038;fg=2b2b2b&%23038;s=0">:</p>
<p><img title="X_n \sim N(100, 400)" alt="X_n \sim N(100, 400)" class="latex" src="https://s0.wp.com/latex.php?latex=X_n+%5Csim+N%28100%2C+400%29&bg=ffffff&%23038;fg=2b2b2b&%23038;s=0"></p>
<p>where</p>
<p><img title="n \in \{5,6,...,25\}" alt="n \in \{5,6,...,25\}" class="latex" src="https://s0.wp.com/latex.php?latex=n+%5Cin+%5C%7B5%2C6%2C...%2C25%5C%7D&bg=ffffff&%23038;fg=2b2b2b&%23038;s=0"></p>
<p>For each <img title="X_n " alt="X_n " class="latex" src="https://s0.wp.com/latex.php?latex=X_n+&bg=ffffff&%23038;fg=2b2b2b&%23038;s=0">, I define a <img title="Y_n " alt="Y_n " class="latex" src="https://s0.wp.com/latex.php?latex=Y_n+&bg=ffffff&%23038;fg=2b2b2b&%23038;s=0"> as</p>
<p><img title="Y_n := 10+\beta X_n +\epsilon " alt="Y_n := 10+\beta X_n +\epsilon " class="latex" src="https://s0.wp.com/latex.php?latex=Y_n+%3A%3D+10%2B%5Cbeta+X_n+%2B%5Cepsilon+&bg=ffffff&%23038;fg=2b2b2b&%23038;s=0"></p>
<p>where</p>
<p><img title="\epsilon \sim N(0,1) " alt="\epsilon \sim N(0,1) " class="latex" src="https://s0.wp.com/latex.php?latex=%5Cepsilon+%5Csim+N%280%2C1%29+&bg=ffffff&%23038;fg=2b2b2b&%23038;s=0"><br>
<img title="\beta \in \{.05, .06,..., .25 \} " alt="\beta \in \{.05, .06,..., .25 \} " class="latex" src="https://s0.wp.com/latex.php?latex=%5Cbeta+%5Cin+%5C%7B.05%2C+.06%2C...%2C+.25+%5C%7D+&bg=ffffff&%23038;fg=2b2b2b&%23038;s=0"></p>
<p>In other words, for each effect size, <img title="\beta " alt="\beta " class="latex" src="https://s0.wp.com/latex.php?latex=%5Cbeta+&bg=ffffff&%23038;fg=2b2b2b&%23038;s=0">, the simulation draws <img title="X " alt="X " class="latex" src="https://s0.wp.com/latex.php?latex=X+&bg=ffffff&%23038;fg=2b2b2b&%23038;s=0"> and <img title="Y " alt="Y " class="latex" src="https://s0.wp.com/latex.php?latex=Y+&bg=ffffff&%23038;fg=2b2b2b&%23038;s=0"> with some error <img title="\epsilon " alt="\epsilon " class="latex" src="https://s0.wp.com/latex.php?latex=%5Cepsilon+&bg=ffffff&%23038;fg=2b2b2b&%23038;s=0">. The following regression model is estimated and the p-value of <img title="\beta " alt="\beta " class="latex" src="https://s0.wp.com/latex.php?latex=%5Cbeta+&bg=ffffff&%23038;fg=2b2b2b&%23038;s=0"> is observed.</p>
<p><img title="Y_n = \beta_0 + \beta X_n " alt="Y_n = \beta_0 + \beta X_n " class="latex" src="https://s0.wp.com/latex.php?latex=Y_n+%3D+%5Cbeta_0+%2B+%5Cbeta+X_n+&bg=ffffff&%23038;fg=2b2b2b&%23038;s=0"></p>
<p>The drawing and the regression is done 1,000 times so that for each effect size – sample size combination, the simulation yields 1,000 p-values. The average of these 1,000 p-values for each effect size and sample size combination is plotted below.</p>
<p>Note, these results are for a fixed <img title="var(\epsilon)=1" alt="var(\epsilon)=1" class="latex" src="https://s0.wp.com/latex.php?latex=var%28%5Cepsilon%29%3D1&bg=ffffff&%23038;fg=2b2b2b&%23038;s=0">. Higher sampling error would typically shift these curves upward, meaning that for each effect size, the same sample would yield a lower signal.</p>
<p><a target="_blank" href="https://stablemarkets.files.wordpress.com/2016/05/2dplot.jpeg"><img alt="2dplot" class="wp-image-883" src="https://stablemarkets.files.wordpress.com/2016/05/2dplot.jpeg?w=474"></a></p>
<p>There are many take-aways from this plot.</p>
<p>First, for a given sample size, larger effect sizes are “detected” more easily. By detected, I mean found to be statistically significant using the .05 threshold. It’s possible to detect larger effect sizes (e.g. .25) with relatively low sample sizes (in this case &lt;10). By contrast, if the effect size is small (e.g. .05), then a larger sample is needed to detect the effect (>10).</p>
<p>Second, this figure illustrates an oft-heard warning about p-values: always interpret them within the context of sample size. Lack of statistical significance does not imply lack of an effect. An effect may exist, but the sample size may be insufficient to detect it (or the variability in the data set is too high). On the other hand, just because a p-value signals statistical significance does not mean that the effect is actually meaningful. Consider an effect size of .00000001 (effectively 0). According to the chart, even the p-value of this effect size tends to 0 as the sample size increases, eventually crossing the statistical significance threshold.</p>
<p> <a target="_blank" rel="nofollow" href="http://feeds.wordpress.com/1.0/gocomments/stablemarkets.wordpress.com/804/"><img alt="" src="http://feeds.wordpress.com/1.0/comments/stablemarkets.wordpress.com/804/" border="0"></a> <img height="1" alt="" width="1" src="https://pixel.wp.com/b.gif?host=stablemarkets.wordpress.com&blog=53001117&%23038;post=804&%23038;subd=stablemarkets&%23038;ref=&%23038;feed=1" border="0"></p>
<p><div><div><div><div data-width="55" data-show-faces="false" data-send="true" data-layout="box_count" data-height="62" data-href="http://www.r-bloggers.com/exploring-p-values-with-simulations-in-r/"></div></div><div><a data-url="http://www.r-bloggers.com/exploring-p-values-with-simulations-in-r/" data-counturl="http://www.r-bloggers.com/exploring-p-values-with-simulations-in-r/" data-count="vertical" data-text="Exploring P-values with Simulations in R" class="twitter-share-button" data-via="rbloggers" href="https://twitter.com/share"></a></div><div></div></div><div></div></div>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="https://stablemarkets.wordpress.com/2016/05/21/exploring-p-values-with-simulations-in-r/"> R – Stable Markets</a></strong>.</div>
<hr>
<a rel="nofollow" href="http://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="http://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a title="The R Project for Statistical Computing" rel="nofollow" href="http://www.r-project.org/">R</a> news and <a title="R tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=tutorial">tutorials</a> on topics such as: <a title="Data science" rel="nofollow" href="http://www.r-bloggers.com/?s=data%20science">Data science</a>, <a title="Big Data" rel="nofollow" href="http://www.r-bloggers.com/?s=Big%20Data">Big Data, <a title="R jobs" rel="nofollow" href="http://www.r-users.com/">R jobs</a>, visualization (<a title="ggplot and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=ggplot2">ggplot2</a>, <a title="Boxplots using lattice and ggplot2 tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=boxplot">Boxplots</a>, <a title="Maps and gis" rel="nofollow" href="http://www.r-bloggers.com/?s=map">maps</a>, <a title="Animation in R" rel="nofollow" href="http://www.r-bloggers.com/?s=animation">animation</a>), programming (<a title="RStudio IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=RStudio">RStudio</a>, <a title="Sweave and literate programming" rel="nofollow" href="http://www.r-bloggers.com/?s=sweave">Sweave</a>, <a title="LaTeX in R" rel="nofollow" href="http://www.r-bloggers.com/?s=LaTeX">LaTeX</a>, <a title="SQL and databases" rel="nofollow" href="http://www.r-bloggers.com/?s=SQL">SQL</a>, <a title="Eclipse IDE for R" rel="nofollow" href="http://www.r-bloggers.com/?s=eclipse">Eclipse</a>, <a title="git and github, Version Control System" rel="nofollow" href="http://www.r-bloggers.com/?s=git">git</a>, <a title="Large data in R using Hadoop" rel="nofollow" href="http://www.r-bloggers.com/?s=hadoop">hadoop</a>, <a title="Web Scraping of google, facebook, yahoo, twitter and more using R" rel="nofollow" href="http://www.r-bloggers.com/?s=Web+Scraping">Web Scraping</a>) statistics (<a title="Regressions and ANOVA analysis tutorials" rel="nofollow" href="http://www.r-bloggers.com/?s=regression">regression</a>, <a title="principal component analysis tutorial" rel="nofollow" href="http://www.r-bloggers.com/?s=PCA">PCA</a>, <a title="Time series" rel="nofollow" href="http://www.r-bloggers.com/?s=time+series">time series</a>, <a title="finance trading" rel="nofollow" href="http://www.r-bloggers.com/?s=trading">trading</a>) and more...
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/rQ15eap_iqY"></a></div>