Recently, a paper was presented at our university that showed a significant effect for a variable of interest but had a relatively small number of observations. One colleague suggested that we should consider the significance of the results with care since the number of observations was fairly small.
This ignited some discussion. Given that the significance test computed exact p-values, why should the significance of the results be less convincing than if we had a larger sample? Independent of the sample size the probability to find a result significant at a 5% level if there is actually no effect should be 5%.
The discussion turned to the question whether small sample sizes could sometimes be problematic because they may magnify possible biases if some data mining takes place. Indeed the dangers of data mining and ‘p-hacking’ are a regular theme in statistics literature and on statistics blogs. To get a better gut feeling of the relationship between sample size and possible biases from data mining, I just have run different Monte-Carlo simulations in R that are shown below.
Freedman’s simulation study
I am currently reading the very nice book Counterfactuals and Causal Inference by Stephen Morgan and Christopher Winship. They cite a classical simulation study by Freedman (1983) that had a result that I found quite surprising and can be related to the question above.
Let us replicate the study with slightly modified numbers in R.
set.seed(123456789)
vars = 100 # Number of explanatory variables
obs = 150 # Number of observations
# Create 100 random iid explanatory variables
x = matrix(rnorm(obs*vars), obs, vars)
colnames(x) = paste0(&quot;x&quot;,1:vars)
# A dependent variable that is independent of all x
y = rnorm(obs)
# Combine to data frame
dat = cbind(data.frame(y=y),x)
# Regress y on all 100 explanatory variables
reg1 = lm(y~., data=dat)
# Collect regression results in a data frame
# and extract p-values
library(broom)
res1 = broom::tidy(reg1)
p = res1$p.value[-1]
# Count the number of significant variables
# at different significance levels
sum(p