<h2>bayesAB 0.7.0</h2>
<p>Quick announcement that my package for Bayesian AB Testing, bayesAB, has been updated to 0.7.0 on CRAN. Some improvements on the backend as well a few tweaks for a more fluid UX/API. Some links:</p>
<ul><li><a href="https://github.com/FrankPortman/bayesAB">bayesAB Github Repo</a></li>
<li><a href="https://cran.r-project.org/package=bayesAB">CRAN Page</a></li>
</ul><p>Now, on to the good stuff.</p>
<h2>Why should we care about priors?</h2>
<p>Most questions I’ve gotten since I released bayesAB have been along the lines of:</p>
<ul><li>Why/how is Bayesian AB testing better than Frequentist hypothesis AB testing?</li>
<li>Why do I need priors?</li>
<li>Do I really really really need priors?</li>
<li>How do I choose priors?</li>
</ul><p>Question 1 has a few objective and a few subjective answers to it. The main benefits are ones that I’ve already highlighted in the README/vignette of the bayesAB package. To briefly summarize, we get direct probabilities for A > B (rather than p-values) and distributions over the parameter estimates rather than point estimates. Finally, we can also leverage <strong>priors</strong> which help with the low sample size and low base rate problems.</p>
<p>To start, let’s go back to what a <strong>prior</strong> actually is in a Bayesian context. There are countless mathematical resources out there (including part of my previous blog post) so I’ll only about this conceptually. Simply put, a <strong>prior</strong> lets you specify some sort of, ahem, prior information about a certain parameter so that the end <strong>posterior</strong> on that parameter encapsualtes both the <strong>data</strong> you saw and the <strong>prior</strong> you inputted. Priors can come from a variety of places including past experiments, literature, and domain expertise into the problem. <a href="http://www.sumsar.net/blog/2015/11/a-bayesian-model-to-calculate-whether-my-wife-is-pregnant/">See this blogpost</a> for a great example of somebody combining their own past data and literature to form very strong priors.</p>
<p><strong>Priors</strong> can be weak or strong. The weakest prior will be completely <strong>objective</strong> and thus assign an equal probability to each value for the parameter. Examples of this include a Beta(1, 1) prior for the Bernoulli distribution. In these cases, the <strong>posterior</strong> distribution is completely reliant on the <strong>data</strong>. A strong <strong>prior</strong> will convey a very precise belief as to where a parameter’s values may lie. For example:</p>
<div><span></span><div><table><tr><td><pre><span>1</span>
<span>2</span>
<span>3</span>
</pre></td><td><pre><code><span><span>library</span><span>(</span>bayesAB<span>)</span>
</span><span>
</span><span>plotBeta<span>(</span><span>1000</span><span>,</span> <span>1000</span><span>)</span>
</span></code></pre></td></tr></table></div></div>
<p align="center"></p>
<p>The stronger the <strong>prior</strong> the more say it has in the <strong>posterior</strong> distribution. Of course, according to the <a href="https://en.wikipedia.org/wiki/Bernstein%E2%80%93von_Mises_theorem">Bernstein–von Mises theorem</a> the <strong>posterior</strong> is effectively independent of the <strong>prior</strong> once a large enough sample size has been reached for the <strong>data</strong>. How quickly this is the case, depends on the strength of your <strong>prior</strong>.</p>
<p>Do you need (weak/strong) <strong>priors</strong>? Not necessarily. You can still leverage the interpretability benefits of Bayesian AB testing even without priors. At worst, you’ll also get slightly more pertinent results since you can parametrize your metrics as the appropriate distribution random variable. However, without <strong>priors</strong> of some kind (and to be clear, not random bullshit priors either) you run into similar issues as with Frequentist AB testing, namely Type 1 and Type 2 errors. A Type 1 error is calling one version better when it really isn’t, and a Type 2 error is calling a better version equal or worse. Both typically arise from low sample size/base rate and are controlled by reaching appropriate sample size as per a <a href="https://en.wikipedia.org/wiki/Statistical_power">power calculation</a>.</p>
<h3>So what can we do?</h3>
<p>Have no fear! Even without good and/or strong <strong>priors</strong> there are still ways to control for false positives and all that good stuff. We use something called <strong>Expected Posterior Loss</strong> or “based on the current winner, what is the expected loss you would see should you choose wrongly”. If this value is lower than your <strong>threshold of caring</strong> (<code>abs(A - b)</code>) then you can go ahead and call your test. This value implictly encompasses the uncertainty about your <strong>posteriors</strong>.</p>
<p>Okay cool, that roughly answers Questions 1-4 in some order.</p>
<h2>Simulation</h2>
<p>Let’s do a quick simulation to illustrate some of the above points. Let’s make three examples: weak priors, strong priors, and diffuse priors (quick tip: the <a href="https://en.wikipedia.org/wiki/Jeffreys_prior">Jeffrey’s Prior</a> of a Gamma distribution is Gamma(eps, eps) where eps is smallllll). We’ll be taking 2 x 100 samples from a Poisson distribution with the same $\lambda$ parameters. The strong and weak priors will be centered around this value of 2.3.</p>
<div><span></span><div><table><tr><td><pre><span>1</span>
<span>2</span>
<span>3</span>
<span>4</span>
<span>5</span>
<span>6</span>
<span>7</span>
<span>8</span>
<span>9</span>
<span>10</span>
<span>11</span>
<span>12</span>
<span>13</span>
<span>14</span>
<span>15</span>
<span>16</span>
<span>17</span>
<span>18</span>
<span>19</span>
<span>20</span>
<span>21</span>
<span>22</span>
<span>23</span>
<span>24</span>
<span>25</span>
<span>26</span>
</pre></td><td><pre><code><span><span>library</span><span>(</span>magrittr<span>)</span>
</span><span>
</span><span>n <span>&lt;-</span> <span>1e3</span>
</span><span>out_weaker_priors <span>&lt;-</span> <span>rep</span><span>(</span><span>NA</span><span>,</span> n<span>)</span>
</span><span>out_stronger_priors <span>&lt;-</span> <span>rep</span><span>(</span><span>NA</span><span>,</span> n<span>)</span>
</span><span>out_diffuse <span>&lt;-</span> <span>rep</span><span>(</span><span>NA</span><span>,</span> n<span>)</span>
</span><span>
</span><span>getProb <span>&lt;-</span> <span>function</span><span>(</span>x<span>)</span> <span>summary</span><span>(</span>x<span>)</span><span>$</span>probability<span>$</span>Lambda
</span><span>
</span><span><span>for</span><span>(</span>i <span>in</span> <span>1</span><span>:</span>n<span>)</span> <span>{</span>
</span><span> A <span>&lt;-</span> rpois<span>(</span><span>100</span><span>,</span> <span>2.3</span><span>)</span>
</span><span> B <span>&lt;-</span> rpois<span>(</span><span>100</span><span>,</span> <span>2.3</span><span>)</span>
</span><span>
</span><span> out_weaker_priors<span>[</span>i<span>]</span> <span>&lt;-</span> bayesTest<span>(</span>A<span>,</span> B<span>,</span> priors <span>=</span> <span>c</span><span>(</span><span>'shape'</span> <span>=</span> <span>23</span><span>,</span> <span>'rate'</span> <span>=</span> <span>10</span><span>),</span> distribution <span>=</span> <span>'poisson'</span><span>)</span> <span>%>%</span>
</span><span> getProb
</span><span>
</span><span> out_stronger_priors<span>[</span>i<span>]</span> <span>&lt;-</span> bayesTest<span>(</span>A<span>,</span> B<span>,</span> priors <span>=</span> <span>c</span><span>(</span><span>'shape'</span> <span>=</span> <span>230</span><span>,</span> <span>'rate'</span> <span>=</span> <span>100</span><span>),</span> distribution <span>=</span> <span>'poisson'</span><span>)</span> <span>%>%</span>
</span><span> getProb
</span><span>
</span><span> out_diffuse<span>[</span>i<span>]</span> <span>&lt;-</span> bayesTest<span>(</span>A<span>,</span> B<span>,</span> priors <span>=</span> <span>c</span><span>(</span><span>'shape'</span> <span>=</span> <span>0.00001</span><span>,</span> <span>'rate'</span> <span>=</span> <span>0.00001</span><span>),</span> distribution <span>=</span> <span>'poisson'</span><span>)</span> <span>%>%</span>
</span><span> getProb
</span><span><span>}</span>
</span><span>
</span><span>out_weaker_priors <span>&lt;-</span> <span>ifelse</span><span>(</span>out_weaker_priors <span>&lt;=</span> <span>0.05</span> <span>|</span> out_weaker_priors <span>>=</span> <span>.95</span><span>,</span> <span>1</span><span>,</span> <span>0</span><span>)</span>
</span><span>out_stronger_priors <span>&lt;-</span> <span>ifelse</span><span>(</span>out_stronger_priors <span>&lt;=</span> <span>0.05</span> <span>|</span> out_stronger_priors <span>>=</span> <span>.95</span><span>,</span> <span>1</span><span>,</span> <span>0</span><span>)</span>
</span><span>out_diffuse <span>&lt;-</span> <span>ifelse</span><span>(</span>out_diffuse <span>&lt;=</span> <span>0.05</span> <span>|</span> out_diffuse <span>>=</span> <span>.95</span><span>,</span> <span>1</span><span>,</span> <span>0</span><span>)</span>
</span></code></pre></td></tr></table></div></div>
<p>Now, A and B shouldn’t have any difference between the two but occasionally we will see a Type 1 error. That’s what the bottom 3 lines are doing. If P(A > B) is &lt;=0.05 or >= .95 we call one of the recipes “significantly” better. Observe what happens with each case of prior.</p>
<div><span></span><div><table><tr><td><pre><span>1</span>
</pre></td><td><pre><code><span><span>mean</span><span>(</span>out_weaker_priors<span>)</span>
</span></code></pre></td></tr></table></div></div>
<div><span></span><div><table><tr><td><pre><span>1</span>
</pre></td><td><pre><code><span><span>## [1] 0.081</span>
</span></code></pre></td></tr></table></div></div>
<div><span></span><div><table><tr><td><pre><span>1</span>
</pre></td><td><pre><code><span><span>mean</span><span>(</span>out_stronger_priors<span>)</span>
</span></code></pre></td></tr></table></div></div>
<div><span></span><div><table><tr><td><pre><span>1</span>
</pre></td><td><pre><code><span><span>## [1] 0.02</span>
</span></code></pre></td></tr></table></div></div>
<div><span></span><div><table><tr><td><pre><span>1</span>
</pre></td><td><pre><code><span><span>mean</span><span>(</span>out_diffuse<span>)</span>
</span></code></pre></td></tr></table></div></div>
<div><span></span><div><table><tr><td><pre><span>1</span>
</pre></td><td><pre><code><span><span>## [1] 0.095</span>
</span></code></pre></td></tr></table></div></div>
<p>The diffuse priors have the most Type 1 errors, followed by the weak priors, followed by the strong priors; to be expected.</p>
<p>Finally, we can fit another bayesTest (:D) to determine whether the differences between Type 1 error percents across priors are different from one another.</p>
<div><span></span><div><table><tr><td><pre><span>1</span>
<span>2</span>
<span>3</span>
<span>4</span>
</pre></td><td><pre><code><span>t1 <span>&lt;-</span> bayesTest<span>(</span>out_diffuse<span>,</span> out_weaker_priors<span>,</span> priors <span>=</span> <span>c</span><span>(</span><span>'alpha'</span> <span>=</span> <span>1</span><span>,</span> <span>'beta'</span> <span>=</span> <span>1</span><span>),</span> distribution <span>=</span> <span>'bernoulli'</span><span>)</span>
</span><span>t2 <span>&lt;-</span> bayesTest<span>(</span>out_diffuse<span>,</span> out_stronger_priors<span>,</span> priors <span>=</span> <span>c</span><span>(</span><span>'alpha'</span> <span>=</span> <span>1</span><span>,</span> <span>'beta'</span> <span>=</span> <span>1</span><span>),</span> distribution <span>=</span> <span>'bernoulli'</span><span>)</span>
</span><span>
</span><span>plot<span>(</span>t1<span>)</span>
</span></code></pre></td></tr></table></div></div>
<p align="center"></p>
<p align="center"></p>
<p align="center"></p>
<div><span></span><div><table><tr><td><pre><span>1</span>
</pre></td><td><pre><code><span>plot<span>(</span>t2<span>,</span> priors <span>=</span> <span>FALSE</span><span>)</span>
</span></code></pre></td></tr></table></div></div>
<p align="center"></p>
<p align="center"></p>
<p>As we can see, it’s somewhat clear that the diffuse is worse than the weak and very clear that the diffuse is worse than the stronger priors. Note that in our case I use a diffuse prior of Beta(1, 1) since I have no idea what’s normal going into this simulation.</p>
<p>Finally we can check the output of <code>summary</code> to see if the <strong>Posterior Expected Loss</strong> is within our constraints.</p>
<div><span></span><div><table><tr><td><pre><span>1</span>
</pre></td><td><pre><code><span><span>summary</span><span>(</span>t1<span>)</span>
</span></code></pre></td></tr></table></div></div>
<div><span></span><div><table><tr><td><pre><span>1</span>
<span>2</span>
<span>3</span>
<span>4</span>
<span>5</span>
<span>6</span>
<span>7</span>
<span>8</span>
<span>9</span>
<span>10</span>
<span>11</span>
<span>12</span>
<span>13</span>
<span>14</span>
<span>15</span>
<span>16</span>
<span>17</span>
<span>18</span>
<span>19</span>
<span>20</span>
<span>21</span>
<span>22</span>
<span>23</span>
<span>24</span>
<span>25</span>
<span>26</span>
<span>27</span>
<span>28</span>
<span>29</span>
<span>30</span>
<span>31</span>
<span>32</span>
<span>33</span>
</pre></td><td><pre><code><span><span>## Quantiles of posteriors for A and B:</span>
</span><span><span>## </span>
</span><span><span>## $Probability</span>
</span><span><span>## $Probability$A_probs</span>
</span><span><span>## 0% 25% 50% 75% 100% </span>
</span><span><span>## 0.06019053 0.08936888 0.09552425 0.10188340 0.14012802 </span>
</span><span><span>## </span>
</span><span><span>## $Probability$B_probs</span>
</span><span><span>## 0% 25% 50% 75% 100% </span>
</span><span><span>## 0.04926047 0.07582110 0.08158084 0.08758630 0.12639536 </span>
</span><span><span>## </span>
</span><span><span>## </span>
</span><span><span>## --------------------------------------------</span>
</span><span><span>## </span>
</span><span><span>## P(A > B) by (0)%: </span>
</span><span><span>## </span>
</span><span><span>## $Probability</span>
</span><span><span>## [1] 0.86214</span>
</span><span><span>## </span>
</span><span><span>## --------------------------------------------</span>
</span><span><span>## </span>
</span><span><span>## Credible Interval on (A - B) / B for interval length(s) (0.9) : </span>
</span><span><span>## </span>
</span><span><span>## $Probability</span>
</span><span><span>## 5% 95% </span>
</span><span><span>## -0.07634565 0.48644269 </span>
</span><span><span>## </span>
</span><span><span>## --------------------------------------------</span>
</span><span><span>## </span>
</span><span><span>## Posterior Expected Loss for choosing B over A:</span>
</span><span><span>## </span>
</span><span><span>## $Probability</span>
</span><span><span>## [1] 0.01270441</span>
</span></code></pre></td></tr></table></div></div>
<div><span></span><div><table><tr><td><pre><span>1</span>
</pre></td><td><pre><code><span><span>summary</span><span>(</span>t2<span>)</span>
</span></code></pre></td></tr></table></div></div>
<div><span></span><div><table><tr><td><pre><span>1</span>
<span>2</span>
<span>3</span>
<span>4</span>
<span>5</span>
<span>6</span>
<span>7</span>
<span>8</span>
<span>9</span>
<span>10</span>
<span>11</span>
<span>12</span>
<span>13</span>
<span>14</span>
<span>15</span>
<span>16</span>
<span>17</span>
<span>18</span>
<span>19</span>
<span>20</span>
<span>21</span>
<span>22</span>
<span>23</span>
<span>24</span>
<span>25</span>
<span>26</span>
<span>27</span>
<span>28</span>
<span>29</span>
<span>30</span>
<span>31</span>
<span>32</span>
<span>33</span>
</pre></td><td><pre><code><span><span>## Quantiles of posteriors for A and B:</span>
</span><span><span>## </span>
</span><span><span>## $Probability</span>
</span><span><span>## $Probability$A_probs</span>
</span><span><span>## 0% 25% 50% 75% 100% </span>
</span><span><span>## 0.06244681 0.08940537 0.09554403 0.10195937 0.13931010 </span>
</span><span><span>## </span>
</span><span><span>## $Probability$B_probs</span>
</span><span><span>## 0% 25% 50% 75% 100% </span>
</span><span><span>## 0.007008903 0.017726756 0.020601621 0.023791156 0.050439822 </span>
</span><span><span>## </span>
</span><span><span>## </span>
</span><span><span>## --------------------------------------------</span>
</span><span><span>## </span>
</span><span><span>## P(A > B) by (0)%: </span>
</span><span><span>## </span>
</span><span><span>## $Probability</span>
</span><span><span>## [1] 1</span>
</span><span><span>## </span>
</span><span><span>## --------------------------------------------</span>
</span><span><span>## </span>
</span><span><span>## Credible Interval on (A - B) / B for interval length(s) (0.9) : </span>
</span><span><span>## </span>
</span><span><span>## $Probability</span>
</span><span><span>## 5% 95% </span>
</span><span><span>## 2.173703 5.994963 </span>
</span><span><span>## </span>
</span><span><span>## --------------------------------------------</span>
</span><span><span>## </span>
</span><span><span>## Posterior Expected Loss for choosing B over A:</span>
</span><span><span>## </span>
</span><span><span>## $Probability</span>
</span><span><span>## [1] NaN</span>
</span></code></pre></td></tr></table></div></div>
<p>If the <strong>Posterior Expected Loss</strong> is lower than our threshold for caring on abs(A - B) then we can call this test and accept the current results. The PEL is small in both cases, and possibly 0/NaN for <code>t2</code> so it’s quite clear that priors, even weak ones, have a significant positive effect on Type 1 Errors. Remember that we see this effect partially because our <strong>priors</strong> were of a similar shape to the <strong>data</strong>. If the <strong>priors</strong> and the <strong>data</strong> disagree, the effects might not be so clear cut and you will need more <strong>data</strong> to have a stable <strong>posterior</strong>.</p>
<p>Check out bayesAB 0.7.0 to supercharge your AB tests and reach out to me if you have any questions!</p>