Something that never ceases to amaze (depress) me, is how extremely common it is to see casual claims in RCTs, that are not part of the randomization. For instance, the relationship between treatment adherence and outcome, or between alliance and outcome, are often analyzed but seldom experimentally manipulated. This is basically observational research disguised as experimental, but without DAGs, instrumental variables, propensity scores, or any other technique used in observational research. Hence, the effects will almost surely be biased in an unknown direction. This is a post I've been planning to write for a long time, but put off since the theory and methods are so well covered in, for instance, epidemiology. However, when brms added support for multivariate formulas, I figured I could show how some of the techniques can be implemented in R, and their Bayesian counterparts using brms. So, in this post I do three things:
Rant about how quickly clinical psychology researchers forget about casual inference, once they start analyzing non-randomized variables, such as the effect of treatment adherence (or &quot;compliance&quot;) on treatment outcome in a randomized controlled trial. Show that common regression methods either, makes absolutely no sense, or result in badly biased estimates of the casual dose-response effect, when a third variable influences both adherence and outcome, i.e. when the dose-response relationship is confounded. While instrumental variable methods may allow for unbiased estimates in these scenarios. Show how Bayesian intrumental variable regression can be easily fit using brms new multivariate syntax. And also do a small simulation study to compare the results to lavaan, AER::ivreg and biased lm() models. Motivating example
The effect of treatment adherence (psychotherapy dose-response effects)
Let's assume we've performed an RCT and compared some psychotherapy condition to some control condition. Patients in these types of trials seldom complete all scheduled sessions, or adhere 100 % to the treatment. It's rather common for researchers to try to see if there is some relationship between adherence and treatment outcome. Which is a valid question, if there is no added effect of actually &quot;receiving&quot; more of the treatment, then that should be a problem for your theory, or at least some of the components of that theory. However, what usually happens is that researchers somehow forget that the variable &quot;adherence to therapy&quot; is not experimentally manipulated. Unless we have randomized patients to receive 1, 2, ..., 10 sessions, or to do 1, 2, ..., 10, exposure exercises (and somehow ensured adherence was 100% in all conditions)–then we are now longer in RCT country!
Never stop to acknowledge you're doing non-randomized comparisons!
A simple model of confounded adherence effects
A really simple scenario with one know (or unknown) confounder is shown in the DAG below. Each arrow represent casual influence of one variable on another. Y is the observed outcome, adherence is often measured by some proxy, like the number of sessions completed–that we assume is a good indicator of treatment &quot;dose&quot;, it could be any other proxy for treatment adherence. So, what the graph is saying, is that treatment allocation (randomization) influences the outcome (Y) via the amount of treatment received (adherence). Thus, unless something went horribly wrong, randomization should be strongly correlated with S, and only influence Y through the treatment received (S). By definition this makes randomization an instrumental variable. However, the problem is that the effect of S on Y is most likely confounded (U). For instance, baseline functioning might be related both to a better prognosis (better outcome) and treatment adherence. It is easy to come up with scenarios were the relationship could be either positive or negative. Higher baseline functioning could be related to a better prognosis, and to better treatment adherence, or, a patient could be too depressed to actually adhere to the treatment. It is also plausible that worse baseline functioning make you more motivated to adhere to your treatment. Whatever the relationship might be, it is clear that the casual effect of treatment adherence will be confounded.
We will deal with three types of scenarios were a variable \(U\) confounds the estimate of treatment adherence:
Observed confounder: \(U\) is an observed variable that is adjusted for.
Residual confounding: our known confounder \(U\) is measured with error.
Hidden confounding: \(U\) is either unknown or unmeasured, and not adjusted for.
The data generating model
First, let's remind ourselves what is commonly estimates in an RCT is the average treatment effect (ATE), or some related estimate, $$
\Delta = E[Y_i(C) - Y_i(Tx)].
$$
Where \(Y_i(C)\) is the would-be outcome for patient i after participating in the control condition, and \(Y_i(Tx)\) the would be outcome for patient i after participating in the treatment condition. Obviously, we can never observe both \(Y_i(C)\) and \(Y_i(Tx)\) for the same patient. But thanks to randomization when can infer the ATE. We also assume that the individual treatment effect \(Y_i(C) - Y_i(Tx)\) is the same for all individuals, given the same level of adherence. The data generating model (DGM) is, thus, that adherence depends on \(U_i\), the confounder variable,
$$ S_i = 7 - f(U_i) + e_{1i}.$$
Where \(f()\) is just a function that centers and scales the confounding variable, so that the mean adherence is 7, and \(e_i\) is just random variation that influence the relationship between \(U_i\) and \(S_i\). Moreover, the observed measure of adherence could be noisy measure of a patient's actual adherence, \(S_i\), e.g. sessions completed,
$$ \text{Sessions}_i = S_i + e_{2i}.$$
To generate the outcome, we first simulate \(Y_i(C)\), and then calculate \(Y_i(Tx)\),
$$ Y(Tx)_i = Y(C)_i - \beta_S S_i.$$
So that \(\beta_S\) is the casual effect of treatment adherence,
$$ E(\Delta_i | S_i) = \beta_s S_i.$$
Lastly, the outcome is generated as,
$$ Y_i = Y_i(C) [1-R_i(Tx)] + Y_i(Tx) R_i(Tx).$$
Where \(R_i(Tx)\) is 1 for participants allocated to the treatment condition and 0 otherwise.
Different ways to answer the same question
Now, that we have a DGM, let's generate some fake data and fit different models. We'll start with a standard OLS regression, the first model would make sense in the absence of any hidden or residual confounding (extremely unlikely). The second model (&quot;naïve OLS&quot;) makes little sense, since it only analyze the treatment group, and as we'll see, further confounds prognosis with the casual effect of treatment adherence. Sadly, this way of analyzing psychotherapy trials is really common... Anyway, lets generate some fake data. I'll use a two-level random intercept-only model as a starting point, and assume that the confounder (baseline functioning), is measured without error.
library(powerlmm)
# Skeleton to base data on
sp