Abstract

Public datasets that summarize schools’ grade-level achievement within and across demographic subgroups protect academic privacy by withholding aggregates for subgroups with few students. Some states disclose achievement for any subgroup (or grade) with at least five students, while others only disclose achievement for those with at least 40 (Jacob et al., 2014). The internal validity of impact estimates from observational studies with school-level propensity score matching, and similarly the external validity of estimates from randomized trials where schools or districts opt into the trial, would benefit from using demographic achievement to fit models of intervention propensity or selection into the trial. The consequences of small-cell suppression when fitting such models have not yet been described, though, nor have there been prescriptions for addressing it. We evaluate propensity score models fit to data where suppressed values have been imputed with mixed-effects model estimates of average achievement within school, grade, and subgroup. We then examine whether covariate balance between matched groups or the proportion of bias removed from the resulting impact estimate suffers compared to propensity score matching applied to data with the true but highly noisy small-cell aggregates, aggregates a researcher could form with access to completely observed student-level data. After making this comparison using simulations, we leverage our unique access to both publicly available school-level and restricted-use student-level datasets of achievement on Texas state tests to evaluate elementary and middle schools that opted to lengthen their academic calendars starting in the 2020-2021 school year.