<div><em>[This is a guest post by Ned Bicare, PhD]</em></div>
<div></div>
<div> <br>


<p><a data-font-primary="Acme" target="_blank" href="http://shinyapps.org/apps/p-hacker/"><br>
<span>Start the p-hacker app!</span></a></p>
<p>
</div>
<div></div>
<div><strong>My dear fellow scientists!</strong></div>
<div></div>
<div>
<blockquote>
<div><em>“If you torture the data long enough, it will confess.”</em></div>
</blockquote>
</div>
<div>This aphorism, attributed to <a target="_blank" href="http://en.wikipedia.org/wiki/Ronald_Coase">Ronald Coase</a>, sometimes has been used in a disrespective manner, as if it was wrong to do creative data analysis.</div>
<div></div>
<div>In fact, the art of creative data analysis has experienced despicable attacks over the last years. A small but annoyingly persistent group of second-stringers tries to denigrate our scientific achievements. They drag psychological science through the mire.</div>
<div></div>
<div>These people propagate stupid method repetitions; and what was once one of the supreme disciplines of scientific investigation – a creative data analysis of a data set – has been crippled to conducting an empty-headed step-by-step pre-registered analysis plan. (Come on: If I lay out the full analysis plan in a pre-registration, even an <i>undergrad</i> student can do the final analysis, right? Is that really the high-level scientific work we were trained for so hard?).</div>
<div></div>
<div>They broadcast in an annoying frequency that p-hacking leads to more significant results, and that researcher who use <em>p</em>-hacking have higher chances of getting things published.</div>
<div></div>
<div>What are the consequence of these findings? The answer is clear. Everybody should be equipped with these powerful tools of research enhancement!</div>
<div></div>
<h3>The art of creative data analysis</h3>
<div>Some researchers describe a performance-oriented data analysis as “data-dependent analysis”. We go one step further, and call this technique <strong><i>data-optimal analysis (<a target="_blank" href="http://www.urbandictionary.com/define.php?term=DOA">DOA</a>)</i></strong>, as our goal is to produce the optimal, most significant outcome from a data set.</div>
<div></div>
<div>I developed an <strong>online app that allows to practice creative data analysis and how to polish your <i>p</i>-values</strong>. It’s primarily aimed at young researchers who do not have our level of expertise yet, but I guess even old hands might learn one or two new tricks! It’s called “The p-hacker” (please note that ‘hacker’ is meant in a very positive way here. You should think of the cool hackers who fight for world peace). You can use the app in teaching, or to practice <em>p</em>-hacking yourself.</div>
<div></div>
<div>Please test the app, and give me feedback! You can also send it to colleagues: <a target="_blank" href="http://shinyapps.org/apps/p-hacker">http://shinyapps.org/apps/p-hacker</a></div>
<div></div>
<div> <br>


<p><a data-font-primary="Acme" target="_blank" href="http://shinyapps.org/apps/p-hacker/"><br>
<span>Start the p-hacker app!</span></a></p>
<p>
</div>
<div></div>
<div>The full R code for this Shiny app is on <a target="_blank" href="https://github.com/nicebread/p-hacker">Github</a>.</div>
<h3>Train your p-hacking skills: Introducing the p-hacker app</h3>
<div>Here’s a quick walkthrough of the app. Please see also the quick manual at the top of the app for more details.</div>
<div></div>
<div>First, you have to run an initial study in the “New study” tab:</div>
<div></div>
<div><a target="_blank" href="http://i0.wp.com/www.nicebread.de/WP/wp-content/uploads/2016/06/69CFFA3C-7144-4D3E-889D-38D6493FF9E2.png"><img sizes="(max-width: 700px) 100vw, 700px" srcset="http://i0.wp.com/www.nicebread.de/WP/wp-content/uploads/2016/06/69CFFA3C-7144-4D3E-889D-38D6493FF9E2.png?resize=456%2C296 913w, http://www.nicebread.de/WP/wp-content/uploads/2016/06/69CFFA3C-7144-4D3E-889D-38D6493FF9E2-300x195.png 300w, http://www.nicebread.de/WP/wp-content/uploads/2016/06/69CFFA3C-7144-4D3E-889D-38D6493FF9E2-768x499.png 768w" alt="69CFFA3C-7144-4D3E-889D-38D6493FF9E2" class="wp-image-1045" src="http://i0.wp.com/www.nicebread.de/WP/wp-content/uploads/2016/06/69CFFA3C-7144-4D3E-889D-38D6493FF9E2.png?resize=456%2C296" data-recalc-dims="1"></a></div>
<div></div>
<div></div>
<div>When you ran your first study, inspect the results in the middle pane. Let’s take a look at our results, which are quite promising:</div>
<div></div>
<div><a target="_blank" href="http://i2.wp.com/www.nicebread.de/WP/wp-content/uploads/2016/06/EAE34502-7F93-4100-B0DE-D739593A12EC.png"><img sizes="(max-width: 674px) 100vw, 674px" srcset="http://i2.wp.com/www.nicebread.de/WP/wp-content/uploads/2016/06/EAE34502-7F93-4100-B0DE-D739593A12EC.png?resize=456%2C417 674w, http://www.nicebread.de/WP/wp-content/uploads/2016/06/EAE34502-7F93-4100-B0DE-D739593A12EC-300x275.png 300w" alt="EAE34502-7F93-4100-B0DE-D739593A12EC" class="wp-image-1046" src="http://i2.wp.com/www.nicebread.de/WP/wp-content/uploads/2016/06/EAE34502-7F93-4100-B0DE-D739593A12EC.png?resize=456%2C417" data-recalc-dims="1"></a></div>
<div></div>
<div><b>After exclusion of this obvious outlier, your first study is already a success!</b> Click on “Save” next to your significant result to save the study to your study stack on the right panel:</div>
<div></div>
<div><a target="_blank" href="http://i0.wp.com/www.nicebread.de/WP/wp-content/uploads/2016/06/8AE9BA13-1A8A-4293-A0B5-FC99872A63A0.png"><img sizes="(max-width: 615px) 100vw, 615px" srcset="http://i0.wp.com/www.nicebread.de/WP/wp-content/uploads/2016/06/8AE9BA13-1A8A-4293-A0B5-FC99872A63A0.png?resize=456%2C362 615w, http://www.nicebread.de/WP/wp-content/uploads/2016/06/8AE9BA13-1A8A-4293-A0B5-FC99872A63A0-300x238.png 300w" alt="8AE9BA13-1A8A-4293-A0B5-FC99872A63A0" class="wp-image-1047" src="http://i0.wp.com/www.nicebread.de/WP/wp-content/uploads/2016/06/8AE9BA13-1A8A-4293-A0B5-FC99872A63A0.png?resize=456%2C362" data-recalc-dims="1"></a></div>
<div></div>
<div></div>
<div>Sometimes outlier exclusion is not enough to improve your result.</div>
<div></div>
<div>Now comes the magic. Click on the “Now: p-hack!” tab – <b>this gives you all the great tools to improve your current study</b>. Here you can fully utilize your data analytic skills and creativity.</div>
<div></div>
<div>In the following example, we could not get a significant result by outlier exclusion alone. But after adding 10 participants (in two batches of 5), controlling for age and gender, and focusing on the variable that worked best – voilà!</div>
<div></div>
<div><a target="_blank" href="http://i2.wp.com/www.nicebread.de/WP/wp-content/uploads/2016/06/8E270173-E65E-40E2-B3DD-0E93E911A938.png"><img sizes="(max-width: 700px) 100vw, 700px" srcset="http://i2.wp.com/www.nicebread.de/WP/wp-content/uploads/2016/06/8E270173-E65E-40E2-B3DD-0E93E911A938.png?resize=456%2C160 881w, http://www.nicebread.de/WP/wp-content/uploads/2016/06/8E270173-E65E-40E2-B3DD-0E93E911A938-300x106.png 300w, http://www.nicebread.de/WP/wp-content/uploads/2016/06/8E270173-E65E-40E2-B3DD-0E93E911A938-768x270.png 768w" alt="8E270173-E65E-40E2-B3DD-0E93E911A938" class="wp-image-1048" src="http://i2.wp.com/www.nicebread.de/WP/wp-content/uploads/2016/06/8E270173-E65E-40E2-B3DD-0E93E911A938.png?resize=456%2C160" data-recalc-dims="1"></a></div>
<div></div>
<div>Do you see how easy it is to craft a significant study?</div>
<div></div>
<div>Now it is important to <b>show even more productivity</b>: Go for the next conceptual replication (i.e., go back to Step 1 and collect a new sample, with a new manipulation and a new DV). Whenever your study reached significance, click on the <em>Save</em> button next to each DV and the study is saved to your stack, awaiting some additional conceptual replications that show the robustness of the effect.</div>
<div><b>Many journals require multiple studies</b>. Four to six studies should make a compelling case for your subtile, counterintuitive, and shocking effects:</div>
<div></div>
<div><a target="_blank" href="http://i2.wp.com/www.nicebread.de/WP/wp-content/uploads/2016/06/67F00220-678D-4D21-BB34-16E8FB062AF6.png"><img alt="67F00220-678D-4D21-BB34-16E8FB062AF6" class="wp-image-1049" src="http://i2.wp.com/www.nicebread.de/WP/wp-content/uploads/2016/06/67F00220-678D-4D21-BB34-16E8FB062AF6.png?resize=284%2C266" data-recalc-dims="1"></a></div>
<div></div>
<div>Honor to whom honor is due: Find the best outlet for your achievements!</div>
<div></div>
<div></div>
<div>My friends, let’s stand together and Make Psychological Science Great Again! I really hope that the <em>p</em>-hacker app can play its part in bringing psychological science back to its old days of glory.</div>
<div></div>
<div><a target="_blank" href="http://i0.wp.com/www.nicebread.de/WP/wp-content/uploads/2016/06/make.png"><img alt="make" class="wp-image-1059" src="http://i1.wp.com/www.nicebread.de/WP/wp-content/uploads/2016/06/make-150x150.png?resize=150%2C150" data-recalc-dims="1"></a></div>
<div>


<p><a data-font-primary="Acme" target="_blank" href="http://shinyapps.org/apps/p-hacker/"><br>
<span>Start the p-hacker app!</span></a></p>
<p>
</div>
<div></div>
<div>Best regards,</div>
<div></div>
<div><i>Ned Bicare, PhD</i></div>
<div><i> </i></div>
<div>PS: A similar app can be found on FiveThirtyEight: <a target="_blank" href="http://projects.fivethirtyeight.com/p-hacking/">Hack Your Way To Scientific Glory</a></div>
<div></div>
<div></div>
<img height="1" alt="" width="1" src="http://feeds.feedburner.com/~r/RBloggers/~4/RgsEOxhmC_k">