Hold the Bets! Do Quasi- and True Experimental Evaluations Yield Equally
Valid Impact Results When Effect Generalization is the Goal?
Abstract
Randomized experiments (RCTs) rule out bias from confounded
selection of participants into conditions by design. Quasi-experiments
(QEs) are often considered second-best because they do not share
this benefit. However, when results from RCTs are used to
generalize causal impacts, the benefit from unconfounded selection into
conditions may be offset by confounded selection into locations. In this
work we show that this tradeoff can lead to situations where estimates
from QEs are less-biased from selection than are estimates from
uncompromised RCTs. We establish the conditions theoretically,
demonstrate the idea empirically, and discuss the implications of the
results.