Hold the Bets! Do Quasi- and True Experimental Evaluations Yield Equally Valid Impact Results When Effect Generalization is the Goal?
Randomized experiments (RCTs) rule out bias from confounded selection of participants into conditions by design. Quasi-experiments (QEs) are often considered second-best because they do not share this benefit. However, when results from RCTs are used to generalize causal impacts, the benefit from unconfounded selection into conditions may be offset by confounded selection into locations. In this work we show that this tradeoff can lead to situations where estimates from QEs are less-biased from selection than are estimates from uncompromised RCTs. We establish the conditions theoretically, demonstrate the idea empirically, and discuss the implications of the results.
No external sources of funding
Declaration of conflicts of interestNo conflict of interest
Corresponding author email@example.com
Lead author country
- United States
Lead author job role
Lead author institutionEmpirical Education Inc.
Ethics statementEmpirical analysis is a secondary analysis of a publicly available dataset
- Yes, I agree to Advance terms