In the last post, I tried to provide a little insight into the chi-square test. In particular, I used simulation to demonstrate the relationship between the Poisson distribution of counts and the chi-squared distribution. The key point in that post was the role conditioning plays in that relationship by reducing variance.
To motivate some of the key issues, I talked a bit about recycling. I asked you to imagine a set of bins placed in different locations to collect glass bottles. I will stick with this scenario, but instead of just glass bottle bins, we now also have cardboard, plastic, and metal bins at each location. In this expanded scenario, we are interested in understanding the relationship between location and material. A key question that we might ask: is the distribution of materials the same across the sites? (Assume we are still just counting items and not considering volume or weight.)
Independence
If we tracked the number of items for a particular day, we could record the data in a contingency table, which in this case would be \(4 \times 3\) array. If we included the row and column totals, it might look like this:
One way to inspect the data would be to calculate row- and column-specific proportions. From this (albeit stylized example), it is apparent that the proportion of each material is constant across locations - 10% of the items are glass, roughly 30% are cardboard, 40% are plastic, and 20% are metal. Likewise, for each material, about 17% are in location 1, 33% in location 2, and 50% in location 3:
This consistency in proportions across rows and columns is the hallmark of independence. In more formal terms, \(P(M = m | L = l) = P(M = m)\) and \(P(L = l|M = m) = P(L = l)\). The conditional probability (what we see in a particular row or column) is equal to the overall probability (what we see in the marginal (total) row or column.
The actual definition of statistical independence with respect to materials and location is
\[ P(M=m \ \&amp; \ L= l) = P(M=m) \times P(L=l) \]
The probability on the left is the cell-specific proportion (the count of the number of items with \(M=m\) and \(L=l\) divided by \(N\), the total number of items in the entire table). The two terms on the right side of the equation are the marginal row and column probabilities respectively. The table of overall proportions gives us an example of data generated from two characteristics that are independent:
There are 116 plastic items in location 3, 19% of the overall items (\(116 \div 600\)). The overall proportion of plastic items is 40%, the overall proportion of items in location 3 is 50%, and \(0.19 \approx 0.4 \times 0.5\). If we inspect all of the cells, the same approximation will hold.
Dependence
In the case where the distributions of materials differ across locations, we no longer have independence. Here is an example, though note that the marginal totals are unchanged:
Looking across the row- and column-specific proportions, it is clear that something unique might be going on at each location:
It is apparent that the formal definition of independence might be violated: \(P(M=m \ \&amp; \ L=l) \ne P(M=m)P(L=l\)). Look again at plastics in location 3: \(0.30 \ne 0.4 \times 0.5\).
The chi-square test of independence
I have been making declarations about independence with my made up contingency tables, just because I was the all-knowing creator who made them up. Of course, when we collect actual data, we don’t have that luxury. That is where the chi-square test of independence helps us.
Here’s the general idea. We start off by making the initial assumption that the rows and columns are indeed independent (this is actually our null hypothesis). We then define a test statistic \(X^2\) as
\[ X^2 = \sum_{m,l} \frac{(O_{ml} - E_{ml})^2}{E_{ml}}.\] This is just a slight modification of the test statistic we saw in part 1, which was presented as a summary of a \(k \times 1\) array. In this context, \(X^2\) is just a summary of the \(M \times \ L\) table. As previously discussed, \(X^2\) has a \(\chi^2\) distribution with a particular parameter specifying the \(k\) degrees freedom.
The question is, how can we calculate \(X^2\)? The observed data \(O_{ml}\) are just the observed data. But, we don’t necessarily know \(E_{ml}\), the expected value of each cell in the contingency table. These expected values can be defined as \(E_{ml} = P(M=m \ \&amp; \ L=l) \times N\). If we assume that \(N\) is fixed, then we are half way there. All that remains is the joint probability of \(M\) and \(L\), \(P(M=m \ \&amp; \ L=l)\). Under independence (which is our starting, or null, assumption) \(P(M=m \ \&amp; \ L=l) = P(M=m)P(L=l)\). If we make the additional assumption that the row and column totals (margins) \(R_m\) and \(C_l\) are fixed, we can calculate \(P(M=m) = R_m/N\) and \(P(L=l) = C_l/N\). So now,
\[E_{ml} = \frac{(R_m * C_l)}{N}.\] Where does that leave us? We calculate the test statistic \(X^2\) and evaluate that statistic in the context of the theoretical sampling distribution suggested by the assumptions of independence and fixed marginal totals. That theoretical sampling distribution is \(\chi^2\) with some degrees of freedom. If the observed \(X^2\) is very large and defies the theoretical distribution (i.e. seems like an outlier), we will reject the notion of independence. (This is just null hypothesis testing using the \(X^2\) statistic.)
Chi-square tests of our two tables
The test statistic from the first table (which I suggest is from a scenario where material and location are independent) is relatively small. We would not conclude that material and location are associated:
## Sum
## 8 23 29 60
## 28 61 91 180
## 39 85 116 240
## 25 31 64 120
## Sum 100 200 300 600
chisq.test(im)
## ## Pearson's Chi-squared test
## ## data: im
## X-squared = 5.0569, df = 6, p-value = 0.5365
In the second case, the test statistic \(X^2\) is quite large, leading us to conclude that material and location are indeed related, which is as we suspected:
## Sum
## 51 5 4 60
## 22 99 59 180
## 21 40 179 240
## 6 56 58 120
## Sum 100 200 300 600
chisq.test(dm)
## ## Pearson's Chi-squared test
## ## data: dm
## X-squared = 314.34, df = 6, p-value __ 2.2e-16
Degrees of freedom
The paramount question is what \(\chi^2\) distribution does \(X^2\) have under the independence assumption? If you look at the results of the chi-square tests above, you can see that, under the null hypothesis of independence and fixed margins, these tables have six degree of freedom, so \(X^2 \sim \chi^2_6\). But, how do we get there? What follows is a series of simulations that start with an unconditional data generation process and ends with the final set of marginal conditions. The idea is to show that by progressively adding stricter conditions to the assumptions, we continuously reduce variability and lower the degrees of freedom.
Unconditional contingency tables
If we start with a data generation process based on the \(4 \times 3\) table that has no conditions on the margins or total number of items, the statistic \(X^2\) is a function of 12 independent Poisson variables. Each cell has an expected value determined by row and column independence. It should follow that \(X^2\) will have 12 degrees of freedom. Simulating a large number of tables under these conditions and evaluating the distribution of the calculated \(X^2\) statistics will likely support this.
The initial (independent) table specified above is our starting point:
addmargins(im)
## Sum
## 8 23 29 60
## 28 61 91 180
## 39 85 116 240
## 25 31 64 120
## Sum 100 200 300 600
row