<p><div><div><div><div data-show-faces="false" data-height="21" data-width="100" data-layout="button_count" data-send="true" data-href="https://www.r-bloggers.com/multilevel-modeling-solves-the-multiple-comparison-problem-an-example-with-r/"></div></div><div></div></div><div></div></div>
<div>
(This article was first published on <strong><a href="http://methodmatters.blogspot.com/2018/10/multilevel-modeling-solves-multiple.html"> Method Matters</a></strong>, and kindly contributed to <a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers)</a> </div>
<p>Multiple comparisons of group-level means is a tricky problem in statistical inference. A standard practice is to adjust the threshold for statistical significance according to the number of pairwise tests performed. For example, according to the widely-known <a rel="nofollow" href="https://en.wikipedia.org/wiki/Bonferroni_correction" target="_blank">Bonferonni method</a>, if we have 3 different groups for which we want to compare the means of a given variable, we would divide the standard significance level (.05 by convention) by the number of tests performed (3 in this case), and only conclude that a given comparison is statistically significant if its <i>p</i>-value falls below .017 (e.g. .05/3).</p>
<p>In this post, we’ll employ an approach often recommended by <a rel="nofollow" href="http://andrewgelman.com/" target="_blank">Andrew Gelman</a>, and use a multi-level model to examine pairwise comparisons of group means. In this method, we use the results of a model built with all the available data to draw simulations for each group from the posterior distribution estimated from our model. We use these simulations to compare group means, without resorting to the expensive corrections typical of standard multiple comparison analyses (e.g. the Bonferroni method referenced above).</p>
<div><b><br></b></div>
<div><b>The Data</b></div>
<p>We will return to a <a rel="nofollow" href="http://methodmatters.blogspot.com/2017/01/analyzing-accupedo-step-count-data-in-r_4.html" target="_blank">data source</a> that we have <a rel="nofollow" href="http://methodmatters.blogspot.com/2017/02/analyzing-accupedo-step-count-data-in-r.html" target="_blank">examined</a> <a rel="nofollow" href="http://methodmatters.blogspot.com/2017/03/analyzing-accupedo-step-count-data-in-r.html" target="_blank">previously</a> on this blog: walking data from the Accupedo app on my phone. <a rel="nofollow" href="http://www.accupedo.com/" target="_blank">Accupedo</a> is a free step-counting app that I’ve been using for more than 3 years to track how many steps I walk each day. It’s <a rel="nofollow" href="http://methodmatters.blogspot.com/2017/01/analyzing-accupedo-step-count-data-in-r_4.html" target="_blank">relatively straightforward</a> to import these data into R and to extract the cumulative number of steps taken per hour.</p>
<p>In the data used for this post, I exclude all measurements taken before 6 AM (because the step counts are cumulative we don’t lose any steps by excluding these observations). This results in 18,117 observations of cumulative hourly step counts for 1113 days. The first observation is on 3 March 2015 and the last on 22 March 2018, resulting in just over 3 years of data!</p>
<p>A sample of the dataset, called <i>aggregate_day_hour</i>, is shown below: </p>
<table>
<thead>
<tr>
<th></th>
<th>oneday</th>
<th>dow</th>
<th>hour</th>
<th>steps</th>
<th>week_weekend</th>
</tr>

</thead>
<tbody>
<tr>
<td>1</td>
<td>2015-03-07</td>
<td>Sat</td>
<td>16</td>
<td>14942</td>
<td>Weekend</td>
</tr>
<tr>
<td>2</td>
<td>2015-03-07</td>
<td>Sat</td>
<td>17</td>
<td>14942</td>
<td>Weekend</td>
</tr>
<tr>
<td>3</td>
<td>2015-03-07</td>
<td>Sat</td>
<td>18</td>
<td>14991</td>
<td>Weekend</td>
</tr>
<tr>
<td>4</td>
<td>2015-03-07</td>
<td>Sat</td>
<td>19</td>
<td>15011</td>
<td>Weekend</td>
</tr>
<tr>
<td>5</td>
<td>2015-03-07</td>
<td>Sat</td>
<td>20</td>
<td>15011</td>
<td>Weekend</td>
</tr>
<tr>
<td>6</td>
<td>2015-03-07</td>
<td>Sat</td>
<td>21</td>
<td>15011</td>
<td>Weekend</td>
</tr>
<tr>
<td>7</td>
<td>2015-03-07</td>
<td>Sat</td>
<td>22</td>
<td>15011</td>
<td>Weekend</td>
</tr>
<tr>
<td>8</td>
<td>2015-03-07</td>
<td>Sat</td>
<td>23</td>
<td>15011</td>
<td>Weekend</td>
</tr>
<tr>
<td>9</td>
<td>2015-03-08</td>
<td>Sun</td>
<td>11</td>
<td>2181</td>
<td>Weekend</td>
</tr>
<tr>
<td>10</td>
<td>2015-03-08</td>
<td>Sun</td>
<td>12</td>
<td>2428</td>
<td>Weekend</td>
</tr>
</tbody>
</table>
<p><i>Oneday </i>represents the calendar year/month/day the observation was recorded, <i>dow </i>contains the name of the day of the week, <i>hour </i>is the hour of the day (in 24-hour or “military” style), <i>steps </i>gives the cumulative step count and <i>week_weekend </i>indicates whether the day in question is a weekday or a weekend.</p>
<div><b>Analytical Goal </b></div>
<p>Our ultimate purpose here is to compare step counts across a large number of days, without having to adjust our threshold for statistical significance. In this example, we will compare the step counts for 50 different days. The standard Bonferonni correction would require us to divide the significance level by the number of possible pairwise comparisons, 1225 in this case. The resulting significance level is .05/1225, or .00004, which imposes a very strict threshold, making it more difficult to conclude that a given comparison is statistically significant.</p>
<p>As mentioned above, the method adopted in this post is one that Andrew Gelman often advocates on his blog. This advice often comes in the form of: “fit a multilevel model!,” with a link to <a rel="nofollow" href="http://www.stat.columbia.edu/~gelman/research/published/multiple2f.pdf" target="_blank">this paper</a> by Gelman, Hill and Yajima (2012). I have to admit that I never fully understood the details of the method from the blog posts. However, by looking at <a rel="nofollow" href="http://www.stat.columbia.edu/~gelman/research/published/multiple2f.pdf" target="_blank">the paper</a> and the explanations and code presented in Chapter 12 of <a rel="nofollow" href="http://www.stat.columbia.edu/~gelman/arm/" target="_blank">Gelman and Hill’s (2007) book</a>, I think I’ve figured out what he means and how to do it. This post represents my best understanding of this method.*</p>
<p>In brief (see the paper for more details), we calculate a multilevel model using flat priors (e.g. the standard out-of-the-box model in the <a rel="nofollow" href="https://cran.r-project.org/package=lme4" target="_blank"><b>lme4</b></a> package). We extract the coefficients from the model and the standard deviation of the residual error of estimated step counts per day (also known as <i>sigma hat</i>), and use these values to simulate step count totals for each day (based on the estimated total step count and <i>sigma hat</i>). We then compare the simulations for each day against each other in order to make pairwise comparisons across days.</p>
<p>Sound confusing? You’re not alone! I’m still grappling with how this works, but I’ll gently walk through the steps below, explaining the basic ideas and the code along the way.</p>
<div><b>Step 1: Calculate the Model</b></div>
<p>In order to perform our pairwise comparisons of 50 days of step counts, we will construct a multi-level model using all of the available data. We’ll employ a model similar to one we used in a <a rel="nofollow" href="http://methodmatters.blogspot.com/2017/02/analyzing-accupedo-step-count-data-in-r.html" target="_blank">previous post</a> on this blog. The model predicts step count from A) the hour of the day and B) whether the day is a weekday or a weekend. We specify fixed effects (meaning regression coefficients that are the same across all days) for hour of the day and time period (weekday vs. weekend). We specify a random effect for the variable that indicates the day each measurement was taken (e.g. a random intercept per day, allowing the average number of estimated steps to be different on each day). We center the hour variable at 6 PM (more on this below).</p>
<p>We can calculate this model using the <a rel="nofollow" href="https://cran.r-project.org/package=lme4" target="_blank"><b>lme4</b></a> package. We will also load the <b><a rel="nofollow" href="https://cran.r-project.org/package=arm" target="_blank">arm</a> </b>package (written to accompany Gelman and Hill’s (2007) book) to extract model parameters we’ll need to compute our simulations below.</p>
<p></p>
<pre><span># load the lme4 package</span><br>library(<span>lme4</span>)<br><span># load the arm package</span><br><span># to extract the estimate of the </span><br><span># standard deviation of our residuals </span><br><span># (sigma hat)</span><br>library(<span>arm</span>)<br><br><span># center the hour variable for 6 pm</span><br><span>aggregate_day_hour</span><span>$</span><span>hour_centered</span> <span>&lt;-</span> <span>aggregate_day_hour</span><span>$</span><span>hour</span> <span>-</span><span>18</span> <br><br><span># compute the model</span><br><span>lme_model</span> <span>&lt;-</span> lmer(<span>steps</span> <span>~</span> <span>1</span> <span>+</span> <span>hour_centered</span> <span>+</span> <span>week_weekend</span>  <span>+</span> (<span>1</span> <span>|</span> <span>oneday</span>), <br>                  <span>data</span>=<span>aggregate_day_hour</span>)<br><br><span># view the model results</span><br>summary(<span>lme_model</span>)<br><br></pre>
<p>The estimates of the fixed effects are as follows:</p>
<table>
<thead>
<tr>
<th></th>
<th>Estimate</th>
<th>Std. Error</th>
<th>t value</th>
</tr>
</thead>
<tbody>
<tr>
<td>(Intercept)</td>
<td>13709.05</td>
<td>143.67</td>
<td>95.42</td>
</tr>
<tr>
<td>hour_centered</td>
<td>1154.39</td>
<td>4.55</td>
<td>253.58</td>
</tr>
<tr>
<td>week_weekendWeekend</td>
<td>-1860.15</td>
<td>268.35</td>
<td>-6.93</td>
</tr>
</tbody>
</table>
<p>Because we have centered our hour variable at 6 PM, the intercept value gives the estimated step count at 6 PM during a weekday (e.g. when the <i>week_weekend </i>variable is 0, as is the case for weekdays in our data). The model thus estimates that at 6 PM on a weekday I have walked 13,709.05 steps. The <i>hour_centered </i>coefficient gives the estimate of the number of steps per hour: 1,154.39. Finally, the <i>week_weekendWeekend </i>variable gives the estimated difference in the total number of steps per day I walk on a weekend, compared to a weekday. In other words, I walk 1,860.15 fewer steps on a weekend day compared to a weekday.</p>
<div><b>Step 2: Extract Model Output </b></div>
<p><i>Coefficients</i></p>
<p>In order to compare the step counts across days, we will make use of the coefficients from our multilevel model. We can examine the coefficients (random intercepts and fixed effects for hour and day of week) with the following code:</p>
<pre><span># examine the coefficients</span><br>coef(<span>lme_model</span>)<br><br></pre>
<p>Which the gives the coefficient estimates for each day (first 10 rows shown):</p>
<table>
<thead>
<tr>
<th></th>
<th>(Intercept)</th>
<th>hour_centered</th>
<th>week_weekendWeekend</th>
</tr>
</thead>
<tbody>
<tr>
<td>2015-03-03</td>
<td>-184.31</td>
<td>1154.39</td>
<td>-1860.15</td>
</tr>
<tr>
<td>2015-03-04</td>
<td>11088.64</td>
<td>1154.39</td>
<td>-1860.15</td>
</tr>
<tr>
<td>2015-03-05</td>
<td>9564.42</td>
<td>1154.39</td>
<td>-1860.15</td>
</tr>
<tr>
<td>2015-03-06</td>
<td>9301.65</td>
<td>1154.39</td>
<td>-1860.15</td>
</tr>
<tr>
<td>2015-03-07</td>
<td>15159.48</td>
<td>1154.39</td>
<td>-1860.15</td>
</tr>
<tr>
<td>2015-03-08</td>
<td>10097.27</td>
<td>1154.39</td>
<td>-1860.15</td>
</tr>
<tr>
<td>2015-03-09</td>
<td>15163.94</td>
<td>1154.39</td>
<td>-1860.15</td>
</tr>
<tr>
<td>2015-03-10</td>
<td>18008.48</td>
<td>1154.39</td>
<td>-1860.15</td>
</tr>
<tr>
<td>2015-03-11</td>
<td>15260.7</td>
<td>1154.39</td>
<td>-1860.15</td>
</tr>
<tr>
<td>2015-03-12</td>
<td>10892.19</td>
<td>1154.39</td>
<td>-1860.15</td>
</tr>
</tbody>
</table>
<p>We will extract these coefficients from the model output, and merge in the day of the week (week vs. weekend) for every date in our dataset. We will then create a binary variable for <i>week_weekend</i>, which takes on a value of 0 for weekdays and 1 for weekends (akin to that automatically created when running the multilevel model with the <b>lme4 </b>package):</p>
<pre><span># extract coefficients from the lme_model</span><br><span>coefficients_lme</span> <span>&lt;-</span> as.data.frame(coef(<span>lme_model</span>)[<span>1</span>])<br>names(<span>coefficients_lme</span>) <span>&lt;-</span> c(<span>'intercept'</span>, <span>&quot;hour_centered&quot;</span>, <span>&quot;week_weekendWeekend&quot;</span>)<br><span>coefficients_lme</span><span>$</span><span>date</span> <span>&lt;-</span> row.names(<span>coefficients_lme</span>)<br><br><span># create a day-level dataset with the indicator</span><br><span># of day (week vs. weekend)</span><br><span>week_weekend_df</span> <span>&lt;-</span> unique( <span>aggregate_day_hour</span>[ , c(<span>'oneday'</span>, <span>'week_weekend'</span>) ] )<br><span># join the week/weekend dataframe to the coefficients dataframe</span><br>library(<span>plyr</span>); library(<span>dplyr</span>)<br><span>coefficients_lme </span><span>&lt;-</span> left_join(<span>coefficients_lme</span>, <br>                                  <span>week_weekend_df</span>[,c(<span>'oneday'</span>, <span>'week_weekend'</span>)], <br>                                <span>by</span> <span>=</span> c(<span>&quot;date&quot;</span> <span>=</span> <span>'oneday'</span>))<br><span># make a dummy variable for weekend which</span><br><span># takes on the value of 0 for weekday</span><br><span># and 1 for the weekend</span><br><span>coefficients_lme </span><span>$</span><span>weekend</span> <span>&lt;-</span> ifelse(<span>coefficients_lme </span><span>$</span><span>week_weekend</span><span>=</span><span>=</span><span>'Weekend'</span>,<span>1</span>,<span>0</span>)<br><br></pre>
<p>Our resulting dataframe looks like this:</p>
<p></p>
<table>
<thead>
<tr>
<th></th>
<th>intercept</th>
<th>hour_centered</th>
<th>week_weekendWeekend</th>
<th>date</th>
<th>week_weekend</th>
<th>weekend</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>-184.31</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2015-03-03</td>
<td>Weekday</td>
<td>0</td>
</tr>
<tr>
<td>2</td>
<td>11088.64</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2015-03-04</td>
<td>Weekday</td>
<td>0</td>
</tr>
<tr>
<td>3</td>
<td>9564.42</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2015-03-05</td>
<td>Weekday</td>
<td>0</td>
</tr>
<tr>
<td>4</td>
<td>9301.65</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2015-03-06</td>
<td>Weekday</td>
<td>0</td>
</tr>
<tr>
<td>5</td>
<td>15159.48</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2015-03-07</td>
<td>Weekend</td>
<td>1</td>
</tr>
<tr>
<td>6</td>
<td>10097.27</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2015-03-08</td>
<td>Weekend</td>
<td>1</td>
</tr>
<tr>
<td>7</td>
<td>15163.94</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2015-03-09</td>
<td>Weekday</td>
<td>0</td>
</tr>
<tr>
<td>8</td>
<td>18008.48</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2015-03-10</td>
<td>Weekday</td>
<td>0</td>
</tr>
<tr>
<td>9</td>
<td>15260.7</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2015-03-11</td>
<td>Weekday</td>
<td>0</td>
</tr>
<tr>
<td>10</td>
<td>10892.19</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2015-03-12</td>
<td>Weekday</td>
<td>0</td>
</tr>
</tbody>
</table>
<p>These coefficients will allow us to create an estimate of the daily step counts for each day.</p>
<p><i>Standard Deviation of Residual Errors: Sigma Hat </i></p>
<p>In order to simulate draws from the posterior distribution implied for each day, we need the estimated step counts per day (also known as <i>y hat</i>, which we’ll calculate using the coefficient dataframe above), and the <i>sigma hat</i> value (e.g. the standard deviation of the residual error from our estimated step counts) from the model. As Gelman and Hill explain in their book, we can simply extract the standard deviation using the <i>sigma.hat</i> command (this function is part of their <b>arm </b>package, which we loaded above). We’ll store the sigma hat value in a variable called <i>sigma_y_hat</i>. In this analysis, the standard deviation of the residual error of our estimated step counts is 2193.09.</p>
<pre><span># extract the standard deviation of the residual error of predicted step counts</span><br><span># store in a variable called &quot;sigma_y_hat&quot;</span><br><span>sigma_y_hat</span> <span>&lt;-</span> sigma.hat(<span>lme_model</span>)<span>$</span><span>sigma</span><span>$</span><span>data</span><br><span># 2913.093</span><br><br></pre>
<div><b>Step 3: Sample 50 days and plot</b></div>
<p>For this exercise, we’ll randomly select 50 days for our pairwise comparisons. We’ll then plot the model estimated step count for the selected days. Here, I calculate the estimated step counts at 6 PM (using the formula and coefficients from the model) directly in the <b><a rel="nofollow" href="https://cran.r-project.org/package=ggplot2" target="_blank">ggplot2</a> </b>code. </p>
<pre><span># sample 50 random days</span><br>set.seed(<span>1</span>)<br><span>sampled_dates</span> <span>&lt;-</span> <span>coefficients_lme</span>[sample(nrow(<span>coefficients_lme</span>), <span>50</span>), ]<br><br><span># visualize estimated step count</span><br><span># for each date</span><br>library(<span>ggplot2</span>)<br>ggplot(<span>sampled_dates</span>, aes(<span>x</span> <span>=</span> <span>date</span>, <span>y</span> <span>=</span> <span>intercept</span> <span>+</span> (<span>week_weekendWeekend</span><span>*</span><span>weekend</span>), <br>                          <span>color</span> <span>=</span> <span>week_weekend</span>)) <span>+</span><br>  <span># point plot</span><br>  geom_point() <span>+</span> <br>  <span># set the title</span><br>  labs(<span>title</span> <span>=</span> <span>&quot;Model Estimated Step Count at 6 PM&quot;</span>) <span>+</span><br>  <span># set the y axis title</span><br>  ylab(<span>'Estimated Number of Steps at 6 PM'</span>) <span>+</span><br>  <span># turn off x axis title, make dates on x axis</span><br>  <span># horizontal</span><br>  theme(<span>axis.title.x</span> <span>=</span> element_blank(),<br>        <span>axis.text.x</span> <span>=</span> element_text(<span>angle</span> <span>=</span> <span>90</span>, <span>vjust</span> <span>=</span> .<span>5</span>)) <span>+</span><br>  <span># set the legend title</span><br>  scale_colour_discrete(<span>name</span>  <span>=</span><span>&quot;Week/Weekend&quot;</span>)<br><br></pre>
<p>Our sampled dates dataframe looks like this (first 10 rows shown): </p>
<table>
<thead>
<tr>
<th></th>
<th>intercept</th>
<th>hour_centered</th>
<th>week_weekendWeekend</th>
<th>date</th>
<th>week_weekend</th>
<th>weekend</th>
</tr>
</thead>
<tbody>
<tr>
<td>296</td>
<td>3985.94</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2015-12-25</td>
<td>Weekday</td>
<td>0</td>
</tr>
<tr>
<td>414</td>
<td>16165.61</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2016-04-21</td>
<td>Weekday</td>
<td>0</td>
</tr>
<tr>
<td>637</td>
<td>11963.4</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2016-11-30</td>
<td>Weekday</td>
<td>0</td>
</tr>
<tr>
<td>1009</td>
<td>10950.75</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2017-12-07</td>
<td>Weekday</td>
<td>0</td>
</tr>
<tr>
<td>224</td>
<td>14669.36</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2015-10-14</td>
<td>Weekday</td>
<td>0</td>
</tr>
<tr>
<td>996</td>
<td>13273.51</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2017-11-24</td>
<td>Weekday</td>
<td>0</td>
</tr>
<tr>
<td>1046</td>
<td>17883.73</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2018-01-13</td>
<td>Weekend</td>
<td>1</td>
</tr>
<tr>
<td>731</td>
<td>10716</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2017-03-04</td>
<td>Weekend</td>
<td>1</td>
</tr>
<tr>
<td>696</td>
<td>16334.55</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2017-01-28</td>
<td>Weekend</td>
<td>1</td>
</tr>
<tr>
<td>69</td>
<td>15280.03</td>
<td>1154.39</td>
<td>-1860.15</td>
<td>2015-05-12</td>
<td>Weekday</td>
<td>0</td>
</tr>
</tbody>
</table>
<p>And our plot looks like this:</p>
<div><a rel="nofollow" href="https://i0.wp.com/2.bp.blogspot.com/-hRtINO4vy9o/WtMF0npX6eI/AAAAAAAAAbk/6z1RZFeryRobymUKZTLZGGCvNO68aVAXwCLcBGAs/s1600/comp_cloud_genre_blog.png?ssl=1" target="_blank"><img border="0" data-original-height="1067" src="https://i0.wp.com/2.bp.blogspot.com/-hRtINO4vy9o/WtMF0npX6eI/AAAAAAAAAbk/6z1RZFeryRobymUKZTLZGGCvNO68aVAXwCLcBGAs/s1600/comp_cloud_genre_blog.png?resize=450%2C1067&ssl=1" data-recalc-dims="1" data-original-width="450"></a></div>
<p>We can see that ‘2015-12-25’ (Christmas 2015) was day in which I walked very few steps. The highest step count is on ‘2016-08-08’, while the second-highest date is ‘2016-04-23’.</p>
<p>Note that the samples dataframe shown above is no longer in chronological order. However, <b>ggplot2</b> is smart enough to understand that the x-axis is a date, and orders the values in the plot accordingly.</p>
<p></p>
<div><b>Step 4: Simulate Posterior Distributions of Step Counts Per Day</b></div>
<p>We now need to simulate posterior distributions of step counts per day. The method below is, as far as I can understand it, that applied in the <a rel="nofollow" href="http://www.stat.columbia.edu/~gelman/research/published/multiple2f.pdf" target="_blank">Gelman et al. (2012) paper</a>. I have adapted the logic from this paper and the Gelman and Hill book (particularly Chapter 12). </p>
<p><i>What are we doing?</i></p>
<p>Our model estimates the overall step counts as a function of our predictor variables. However, the model is not perfect, and the predicted step counts do not perfectly match the observed step counts. This difference between the observed and the predicted step counts for each day is called “residual error” – it’s what your model misses.</p>
<p>We will make 1000 simulations for each day with step count values that we did not actually see, but according to the model estimate and the uncertainty of that estimate (e.g. <i>sigma hat</i>), we <i>could have seen</i>. To do this, we simply calculate the model-predicted step count for each day, and use this value along with the <i>sigma hat</i> value (the standard deviation of our residual error) to sample 1000 values from each day’s implied posterior distribution. </p>
<p><i>How do we do it? </i></p>
<p>I wrote a simple function to do this. It takes our sample data frame and, for each row (date) in the dataset, it calculates the estimated step count using the regression formula and parameter estimates and passes this value, along with the <i>sigma hat</i> value, to the <i><a rel="nofollow" href="https://stat.ethz.ch/R-manual/R-devel/library/stats/html/Normal.html" target="_blank">rnrom</a> </i>function, drawing 1000 samples from the implied posterior distribution for that day.</p>
<p>Note that I’m not including the <i>hour_centered </i>variable in this function. This is because I’m choosing to estimate the step count when <i>hour_centered </i>is zero (e.g. 6 PM because we centered on this value above).</p>
<pre><span># this function calculates the estimated step count</span><br><span># and then draws 1000 samples from the model-implied</span><br><span># posterior distribution</span><br>lme_create_samples <span>&lt;-</span> <span>function</span>(<span>intercept</span>, <span>w_wk_coef</span>, <span>w_wk_dum</span>){<br>  <span>intercept_f</span> <span>&lt;-</span> <span>intercept</span> <span>+</span> (<span>w_wk_coef</span> <span>*</span> <span>w_wk_dum</span>)<br>  <span>y_tilde</span> <span>&lt;-</span> rnorm(<span>1000</span>, <span>intercept_f</span>, <span>sigma_y_hat</span>)<br>}<br><br><span># here we apply the function to our sampled dates</span><br><span># and store the results in a matrix called</span><br><span># &quot;posterior_samples&quot;</span><br><span>posterior_samples</span> <span>&lt;-</span> mapply(<span>lme_create_samples</span>, <br>                            <span>sampled_dates</span><span>$</span><span>intercept</span>, <br>                            <span>sampled_dates</span><span>$</span><span>week_weekendWeekend</span>, <br>                            <span>sampled_dates</span><span>$</span><span>weekend</span>)<br>dim(<span>posterior_samples</span>)<br><span># [1] 1000 50</span><br><br></pre>
<p>Our resulting <i>posterior_samples</i> matrix has 1 column for each day, with 1000 samples for each day contained in the rows. Below I show a small subset of the entire matrix – just the first 10 rows for the first 5 columns. It is interesting to note that the first column, representing the first day (‘2015-12-25’) in our dataframe of samples above, has smaller values than the second column (representing the second day, ‘2016-04-21’). If we look at the intercepts in the samples dataset, we can see that the first day is much lower than the second. This difference in our model-estimated intercepts is propagated to the simulations of daily step counts. </p>
<table>
<tbody>
<tr>
<td>3822.43</td>
<td>12517.38</td>
<td>10611.98</td>
<td>7940.35</td>
<td>15046.33</td>
</tr>
<tr>
<td>3532.09</td>
<td>13224.83</td>
<td>10720.97</td>
<td>5916.09</td>
<td>15879.66</td>
</tr>
<tr>
<td>-298.5</td>
<td>18354.48</td>
<td>7002.13</td>
<td>7571.94</td>
<td>18489.92</td>
</tr>
<tr>
<td>2593.04</td>
<td>12354.25</td>
<td>8929.43</td>
<td>6891.52</td>
<td>9846.13</td>
</tr>
<tr>
<td>5203.44</td>
<td>17702.38</td>
<td>14202.8</td>
<td>8032.03</td>
<td>13051.09</td>
</tr>
<tr>
<td>7943.9</td>
<td>14611.36</td>
<td>9792.22</td>
<td>14878.74</td>
<td>9892.48</td>
</tr>
<tr>
<td>3686.51</td>
<td>15005.1</td>
<td>10546.27</td>
<td>5777.57</td>
<td>15511.05</td>
</tr>
<tr>
<td>5115.26</td>
<td>13865.52</td>
<td>10934.97</td>
<td>12029.68</td>
<td>21345.46</td>
</tr>
<tr>
<td>3829.2</td>
<td>15495.18</td>
<td>11781.92</td>
<td>11072.98</td>
<td>17209.84</td>
</tr>
<tr>
<td>-25.57</td>
<td>18720.93</td>
<td>16681.49</td>
<td>10564.03</td>
<td>13591.19</td>
</tr>
</tbody>
</table>
<div><b>Step 5: Compare the Simulations For Each Day </b></div>
<p>We can see in the above matrix of simulations that the first day (‘2015-12-25’, contained in the first column) has lower values than the second day (‘2016-04-21’, contained in the second column). In order to formally compare the the two columns, we will compare the simulations and see how many times the values in the first column are larger than those in the second column. If values in one column are larger more than 95% of the time, we will say that there is a meaningful (“significant”) difference in step counts between the two dates.</p>
<p>We will apply this logic to all of the possible pairwise comparisons in our sampled 50 dates. The following code accomplishes this (adopted from this <a rel="nofollow" href="https://stackoverflow.com/questions/19933788/r-compare-all-the-columns-pairwise-in-matrix" target="_blank">tremendous Stackoverflow question + answer</a>):</p>
<pre><span># do the pairwise comparisons</span><br><span># first construct an empty matrix</span><br><span># to contain results of comparisons</span><br><span>comparison_matrix</span><span>&lt;-</span>matrix(<span>nrow</span>=ncol(<span>posterior_samples</span>),<span>ncol</span>=ncol(<span>posterior_samples</span>))<br><span># give column and row names for the matrix</span><br><span># (these are our observed dates)</span><br>colnames(<span>comparison_matrix</span>) <span>&lt;-</span> <span>sampled_dates</span><span>$</span><span>date</span><br>rownames(<span>comparison_matrix</span>) <span>&lt;-</span> <span>sampled_dates</span><span>$</span><span>date</span><br><span># loop over the columns of the matrix</span><br><span># and count the number of times the values</span><br><span># in each column are greater than the values</span><br><span># in the other columns</span><br><span>for</span>(<span>col</span> <span>in</span> <span>1</span><span>:</span>ncol(<span>posterior_samples</span>)){<br>  <span>comparisons</span><span>&lt;-</span><span>posterior_samples</span>[,<span>col</span>]<span>></span><span>posterior_samples</span><br>  <span>comparisons_counts</span><span>&lt;-</span>colSums(<span>comparisons</span>)<br>  <span>comparisons_counts</span>[<span>col</span>]<span>&lt;-</span> <span>0</span> <span># Set the same column comparison to zero.</span><br>  <span>comparison_matrix</span>[,<span>col</span>]<span>&lt;-</span><span>comparisons_counts</span><br>}<br><br><span># shape of output comparison matrix</span><br>dim(<span>comparison_matrix</span>)<br><span># [1] 50 50</span><br><br></pre>
<p>The first 10 rows of the first 5 columns of our comparison matrix look like this:</p>
<table>
<thead>
<tr>
<th></th>
<th>2015-12-25</th>
<th>2016-04-21</th>
<th>2016-11-30</th>
<th>2017-12-07</th>
<th>2015-10-14</th>
</tr>
</thead>
<tbody>
<tr>
<td>2015-12-25</td>
<td>0</td>
<td>998</td>
<td>970</td>
<td>954</td>
<td>995</td>
</tr>
<tr>
<td>2016-04-21</td>
<td>2</td>
<td>0</td>
<td>173</td>
<td>103</td>
<td>360</td>
</tr>
<tr>
<td>2016-11-30</td>
<td>30</td>
<td>827</td>
<td>0</td>
<td>395</td>
<td>722</td>
</tr>
<tr>
<td>2017-12-07</td>
<td>46</td>
<td>897</td>
<td>605</td>
<td>0</td>
<td>822</td>
</tr>
<tr>
<td>2015-10-14</td>
<td>5</td>
<td>640</td>
<td>278</td>
<td>178</td>
<td>0</td>
</tr>
<tr>
<td>2017-11-24</td>
<td>13</td>
<td>755</td>
<td>384</td>
<td>299</td>
<td>656</td>
</tr>
<tr>
<td>2018-01-13</td>
<td>3</td>
<td>514</td>
<td>173</td>
<td>114</td>
<td>379</td>
</tr>
<tr>
<td>2017-03-04</td>
<td>114</td>
<td>962</td>
<td>786</td>
<td>698</td>
<td>915</td>
</tr>
<tr>
<td>2017-01-28</td>
<td>7</td>
<td>660</td>
<td>273</td>
<td>223</td>
<td>529</td>
</tr>
<tr>
<td>2015-05-12</td>
<td>3</td>
<td>569</td>
<td>223</td>
<td>152</td>
<td>436</td>
</tr>
</tbody>
</table>
<p>The counts in each cell indicate the number of times (out of 1000) that the column value is greater than the row value (we set the diagonals – comparisons of a day with itself – to zero in the above code). The comparison we identified above (‘2015-12-25’ and ‘2016-04-21’) is shown in the first row, second column (and the second row, first column – this matrix contains the same information but in reverse above and below the diagonals). The samples from ‘2016-04-21’ were greater than those from ‘2015-12-25’ 998 times out of 1000!</p>
<p></p>
<div><b>Step 6: Make the Pairwise Comparison Plot</b></div>
<p>We can make a heat map using <b>ggplot2</b> to show which pairwise comparisons are “significantly” different from one another, akin to that used in Gelman et al. (2007).</p>
<p>Because <b>ggplot2 </b>requires data to be in a tidy format, we’ll have to melt the comparison matrix so that it has 1 row per pairwise comparison. The code to do this was based on <a rel="nofollow" href="https://stackoverflow.com/questions/12081843/r-matrix-plot-with-colour-threshold-and-grid" target="_blank">these very clear</a> <a rel="nofollow" href="http://www.sthda.com/english/wiki/ggplot2-quick-correlation-matrix-heatmap-r-software-and-data-visualization" target="_blank">explanations</a> of how to make a heatmap with <b>ggplot2</b>:</p>
<pre><span># load the reshape2 package</span><br><span># for melting our data</span><br>library(<span>reshape2</span>)<br><span># melt the data</span><br><span>melted_cormat</span> <span>&lt;-</span> melt(<span>comparison_matrix</span>)<br><span># rename the variables</span><br>names(<span>melted_cormat</span>)<span>=</span>c(<span>&quot;x&quot;</span>,<span>&quot;y&quot;</span>,<span>&quot;count&quot;</span>)<br><span># identify which comparisons are &quot;significant&quot;</span><br><span>melted_cormat</span><span>$</span><span>meaningful_diff</span> <span>=</span> factor(<span>melted_cormat</span><span>$</span><span>count</span><span>></span><span>950</span>)<br><span># and set the levels</span><br>levels(<span>melted_cormat</span><span>$</span><span>meaningful_diff</span>) <span>=</span> c(<span>'No Difference'</span>, <br>                                          <span>'Row > Column'</span>)<br><span># sort the matrix by the dates</span><br><span># first turn the date columns</span><br><span># into date types in R using</span><br><span># the lubridate package</span><br>library(<span>lubridate</span>)<br><span>melted_cormat</span><span>$</span><span>x</span> <span>&lt;-</span> ymd(<span>melted_cormat</span><span>$</span><span>x</span>)<br><span>melted_cormat</span><span>$</span><span>y</span> <span>&lt;-</span> ymd(<span>melted_cormat</span><span>$</span><span>y</span>)<br><span># then arrange the dataset by x and y dates</span><br><span>melted_cormat</span> %<span>></span>% arrange(<span>x</span>,<span>y</span>) <span>-</span><span>></span> <span>melted_cormat</span> <br><br></pre>
<p>Our final formatted matrix for plotting, called <i>melted_cormat</i>, looks like this (first 10 rows shown):</p>
<table>
<thead>
<tr>
<th></th>
<th>x</th>
<th>y</th>
<th>count</th>
<th>meaningful_diff</th>
</tr>
</thead>
<tbody>
<tr>
<td>1</td>
<td>2015-03-18</td>
<td>2015-03-18</td>
<td>0</td>
<td>No Difference</td>
</tr>
<tr>
<td>2</td>
<td>2015-03-18</td>
<td>2015-03-29</td>
<td>222</td>
<td>No Difference</td>
</tr>
<tr>
<td>3</td>
<td>2015-03-18</td>
<td>2015-05-12</td>
<td>737</td>
<td>No Difference</td>
</tr>
<tr>
<td>4</td>
<td>2015-03-18</td>
<td>2015-06-29</td>
<td>222</td>
<td>No Difference</td>
</tr>
<tr>
<td>5</td>
<td>2015-03-18</td>
<td>2015-07-19</td>
<td>515</td>
<td>No Difference</td>
</tr>
<tr>
<td>6</td>
<td>2015-03-18</td>
<td>2015-09-15</td>
<td>768</td>
<td>No Difference</td>
</tr>
<tr>
<td>7</td>
<td>2015-03-18</td>
<td>2015-09-22</td>
<td>884</td>
<td>No Difference</td>
</tr>
<tr>
<td>8</td>
<td>2015-03-18</td>
<td>2015-10-14</td>
<td>683</td>
<td>No Difference</td>
</tr>
<tr>
<td>9</td>
<td>2015-03-18</td>
<td>2015-10-18</td>
<td>837</td>
<td>No Difference</td>
</tr>
<tr>
<td>10</td>
<td>2015-03-18</td>
<td>2015-10-22</td>
<td>728</td>
<td>No Difference</td>
</tr>
</tbody>
</table>
<p>The <i>count </i>variable here gives the number of times that the <i>y</i> date simulations were greater than the <i>x</i> date simulations. So the second row above indicates that the simulations on ‘2015-03-29’ were greater than those for ‘2015-03-18’ a total of 222 times. </p>
<p>We can make the heatmap with the following code:</p>
<pre><span># make the heatmap</span><br>ggplot(<span>melted_cormat</span>, aes(as.factor(<span>x</span>), as.factor(<span>y</span>), <br>                          <span>fill</span> <span>=</span> <span>meaningful_diff</span>)) <span>+</span> <br>  <span># tile plot</span><br>  geom_tile() <span>+</span><br>  <span># remove x and y axis titles</span><br>  <span># rotate x axis dates 90 degrees</span><br>  theme(<span>axis.title.x</span>=element_blank(),<br>        <span>axis.title.y</span>=element_blank(), <br>        <span>axis.text.x</span> <span>=</span> element_text(<span>angle</span> <span>=</span> <span>90</span>, <br>                                   <span>vjust</span> <span>=</span> .<span>5</span>)) <span>+</span> <br>  <span># choose the colors for the plot </span><br>  <span># and specify the legend title</span><br>  scale_fill_manual(<span>name</span> <span>=</span> <span>&quot;Comparison&quot;</span>, <br>                    <span>values</span>=c(<span>&quot;#999999&quot;</span>, <span>&quot;#800000&quot;</span>)) <span>+</span> <br>  <span># add a title to the plot</span><br>  labs(<span>title</span> <span>=</span> <span>&quot;Pairwise Comparisons of Model-Estimated Step Counts at 6 PM&quot;</span>)<br><br></pre>
<p>Which gives us the following plot:</p>
<div><a rel="nofollow" href="https://i0.wp.com/4.bp.blogspot.com/-ofDyxtkP0N8/WtN5Y9Dh2gI/AAAAAAAAAb0/ItPQL53tI_IoMKlCNOje18Dqs4-Rp-qlgCLcBGAs/s1600/comparison_heatmap.png?ssl=1" target="_blank"><img border="0" data-original-height="1067" src="https://i0.wp.com/4.bp.blogspot.com/-ofDyxtkP0N8/WtN5Y9Dh2gI/AAAAAAAAAb0/ItPQL53tI_IoMKlCNOje18Dqs4-Rp-qlgCLcBGAs/s1600/comparison_heatmap.png?resize=450%2C1067&ssl=1" data-recalc-dims="1" data-original-width="450"></a></div>
<p>How can we read this plot? All of our 50 dates are displayed on the x and y axes. When the row step count is meaningfully larger than the column step count (according to our method, e.g. if more than 950 of the simulations for the row date are greater than those for the column date), the cell is colored in maroon. Otherwise, cells are colored in grey.</p>
<p><u>Rows</u> that have a lot of maroon values are days that have <i>higher </i>step counts. To take a concrete example, the row representing ‘2016-08-08’ has many red values. If we look at the estimated step count for that day on the first graph above, we can see that it has the highest predicted step count in the sampled dates – over 25,000! It therefore makes sense that the estimated step count on this day is “significantly” larger than those from most of the other dates.</p>
<p><u>Columns</u> that have many red values are days that have especially <i>low </i>step counts. For example, the column representing ‘2015-12-25’ is mostly red. As we saw above, this date has the lowest predicted step count in our sample – under 5,000! It is no wonder then that this date has a “significantly” lower step count than most of the other days. </p>
<p></p>
<div><b>How Baysian Am I?</b></div>
<p>I’m a newbie to Baysian thinking, but I get the sense that Baysian statistics comes in many different flavors. The approach above strikes me as being a little bit, but not completely, Baysian.</p>
<p><i>That’s So Baysian</i></p>
<p>This approach is Baysian in that it uses posterior distributions of daily step counts to make the comparisons between days. The approach recognizes that observed daily step counts are just observations from a larger posterior distribution of possible step counts, and we make explicit use of this posterior distribution in the data analysis. In contrast, frequentist methods basically only make use of point estimates of coefficients and the standard errors of those estimates to draw conclusions from data. </p>
<p><i>Not So Fast!</i></p>
<p>There are some things about this perspective that aren’t so Baysian. First, we use flat priors in the model; a “more Baysian” approach would assign prior distributions to the intercept and the coefficients, which would then be updated during the model computation. Second, we use the point estimates of our coefficients to compute the estimated daily step counts. A “more Baysian” approach would recognize that the coefficient estimates also have posterior distributions, and would incorporate this uncertainty in the simulations of daily step counts.</p>
<p><i>What Do I Know?</i></p>
<p>This is my current best understanding of the differences in Baysian and frequentist perspectives as they apply to what we’ve done here. I’m reading <a rel="nofollow" href="http://xcelab.net/rm/statistical-rethinking/" target="_blank">Statistical Rethinking</a> (and watching the accompanying <a rel="nofollow" href="https://www.youtube.com/playlist?list=PLDcUM9US4XdMdZOhJWJJD4mDBMnbTWw_z" target="_blank">course videos</a>) by <a rel="nofollow" href="http://xcelab.net/rm/" target="_blank">Richard McElreath</a> (love it), and these differences are what I’ve understood from the book and videos thus far. </p>
<p>The method used in this blog post is a nice compromise for someone comfortable with frequentist use of multi-level models (like myself). It requires a little bit more work than just interpreting standard multilevel model output, but it’s not a tremendous stretch. Indeed, the more “exotic” procedures (for a Baysian newbie) like assigning priors for the model parameters are not necessary here.</p>
<p></p>
<div><b>Summary and Conclusion</b></div>
<p>In this post, we used multilevel modeling to construct pairwise comparisons of estimated step counts for 50 different days. We computed the multilevel model, extracted the coefficients and sigma hat value, and computed 1000 simulations from each day’s posterior distribution. We then conducted pairwise comparisons for each day’s simulations, concluding that a day’s step count was “significantly” larger if 950 of the 1000 simulations were greater than the comparison day. We used a heat map to visualize the pairwise comparisons; this heat map highlighted days with particularly high and low step counts as being “significantly” different from most of the other days. This allowed us to conduct these comparisons without the expensive adjustments that come with lowering our significance threshold for multiple comparisons!<br>  <br><i>Coming Up Next</i></p>
<p>In the next post, we will move away from classical statistics and talk about machine learning techniques. Specifically, we will use a type of deep learning model to automatically generate band names.</p>
<p>Stay tuned!</p>
<p>——-</p>
<p>* Please correct me if I’m wrong! </p>
	<p>
<div>
<div>To <strong>leave a comment</strong> for the author, please follow the link and comment on their blog: <strong><a href="http://methodmatters.blogspot.com/2018/10/multilevel-modeling-solves-multiple.html"> Method Matters</a></strong>.</div>
<hr>
<a rel="nofollow" href="https://www.r-bloggers.com/">R-bloggers.com</a> offers <strong><a rel="nofollow" href="https://feedburner.google.com/fb/a/mailverify?uri=RBloggers">daily e-mail updates</a></strong> about <a rel="nofollow" href="https://www.r-project.org/" title="The R Project for Statistical Computing">R</a> news and <a rel="nofollow" href="https://www.r-bloggers.com/search/tutorial" title="R tutorials">tutorials</a> on topics such as: <a rel="nofollow" href="https://www.r-bloggers.com/search/data%20science" title="Data science">Data science</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Big%20Data" title="Big Data">Big Data, <a rel="nofollow" href="https://www.r-users.com/" title="R jobs">R jobs</a>, visualization (<a rel="nofollow" href="https://www.r-bloggers.com/search/ggplot2" title="ggplot and ggplot2 tutorials">ggplot2</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/boxplot" title="Boxplots using lattice and ggplot2 tutorials">Boxplots</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/map" title="Maps and gis">maps</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/animation" title="Animation in R">animation</a>), programming (<a rel="nofollow" href="https://www.r-bloggers.com/search/RStudio" title="RStudio IDE for R">RStudio</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/sweave" title="Sweave and literate programming">Sweave</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/LaTeX" title="LaTeX in R">LaTeX</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/SQL" title="SQL and databases">SQL</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/eclipse" title="Eclipse IDE for R">Eclipse</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/git" title="git and github, Version Control System">git</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/hadoop" title="Large data in R using Hadoop">hadoop</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/Web+Scraping" title="Web Scraping of google, facebook, yahoo, twitter and more using R">Web Scraping</a>) statistics (<a rel="nofollow" href="https://www.r-bloggers.com/search/regression" title="Regressions and ANOVA analysis tutorials">regression</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/PCA" title="principal component analysis tutorial">PCA</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/time+series" title="Time series">time series</a>, <a rel="nofollow" href="https://www.r-bloggers.com/search/trading" title="finance trading">trading</a>) and more...
<img width="1" alt="" src="http://feeds.feedburner.com/~r/RBloggers/~4/d3UDx_7JCos" height="1"></a></div>