Hold the Bets! Do Quasi- and True Experimental Evaluations Yield Equally Valid Impact Results When Effect Generalization is the Goal?

Randomized experiments (RCTs) rule out bias from confounded selection of participants into conditions by design. Quasi-experiments (QEs) are often considered second-best because they do not share this benefit. However, when results from RCTs are used to generalize causal impacts, the benefit from unconfounded selection into conditions may be offset by confounded selection into locations. In this work we show that this tradeoff can lead to situations where estimates from QEs are less-biased from selection than are estimates from uncompromised RCTs. We establish the conditions theoretically, demonstrate the idea empirically, and discuss the implications of the results.