Proposal Description

States, schools and clinics increasingly track outcomes that our policies seek to influence, but designs available for study of those policies have limitations of internal validity, external validity or both, limitations that are typically magnified when privacy considerations limit data availability. Demand for such studies being likely to continue, it is better to address and explore these limitations than to ignore them. This session comprises four methodological talks and one meta-methodological talk on these themes, each with a motivating application involving private and public administrative data.
The methods: combine data from a controlled trial with a broader health database in order to estimate population-level treatment effects; mitigate selection on observables in studies of bias in administrative decisionmaking, while also structuring debate about selection on unobservables; leverage school demographic and achievement summaries to identify usual-practice comparison schools for adopters of new education programs, even with suppression of prior academic achievement indicators for smaller demographic subgroups; and estimate experimental or quasi-experimental treatment effects despite the outcomes’ being subject to small-cell suppression. The methods are demonstrated with a randomized clinical trial coupled to the United Kingdom’s national health registry, with an observational study of police hiring practices in the United States’s third-largest city; with an observational study of an elementary school program of a type commonly used to address Covid-era learning loss, as implemented on the eve of the pandemic in the U.S.’s second-largest state, using that state’s public-use administrative data; and with an observational study of mortality benefits from the U.S.’s 2014 expansion of health insurance entitlements, using published national health statistics to compare U.S. counties that were and were not affected by the change. The third and fourth talks compare the new methods’ results with those possible using protected data not subject to the privacy restrictions.
A fifth talk compares uses of existing methods rather than presenting new ones, charting the diversity of responses to a large U.S. municipality’s offer of privileged access to administrative data in exchange for quasi-experimental analyses. Participating researchers were asked to estimate impacts on public transit use of an extended fare holiday that had been offered at random to a subset of the population, although neither this subset nor the random assignments were shared with the researchers. The tens of analyses submitted varied notably, with different interpretations of apparently unambiguous research questions and additional consequential variation even within similarly-minded reports. The findings suggest new questions for data stewards and policymakers to pose to potential users of public data, while tangibly demonstrating the positive contribution of having epistemically diverse teams address common empirical effectiveness questions.