Abstract

New Zealand’s 2023 Census tailored its field collection strategies to different areas. Using a cluster-randomised controlled trial, we test response mode, timeliness, and quality after mailing letters with internet access codes (IACs) versus paper forms with IACs. We test whether our measures of paper response and non-response, based on 2018 Census patterns, deployed paper correctly. Note: results too disclosive of unreleased census data will be redacted.
Context: About 70% of private dwellings in the country are in the mailout stream. We mail each dwelling a letter with an IAC, with or without a pack of paper forms. Anyone could request paper forms, and later processes gave a mixture of paper and IACs.
Enablement mode: In mailout, 20% were enabled with paper; 80%, without (IAC-only). To assign meshblock areas to an enablement mode, we developed a “paper response index” and “non-response index”: areas above a cutoff on either scale were assigned to paper enablement.
Randomised enablement: Did paper enablement affect response mode (paper or online), self-response, or response quality? An observational study would lack statistical power. Instead, after ethical analysis, we randomised about 3% of mailout areas, those with scores near the cutoff on one or both indices.
Analysis 1: We use GLMMs to test the effects of paper enablement on responding on paper, self-response received before non-response follow-up, and response quality. We use a Holm-Bonferroni correction for multiplicity.
Analysis 2: We use a multi-rating regression discontinuity design to estimate those three effects while incorporating information from the wider mailout stream. These analyses also show how well the paper response index predicted response on paper and how well the non-response index predicted non-response, controlling for enablement mode.
Complications: A devastating cyclone hit the week before initial mailout. Census was postponed in the hardest hit regions, which are removed from these analyses.

Strand: As this deals with public capacity for online survey response in a post-pandemic world, this may be most appropriate in strand 1 (Challenges and innovations in data production) or 3 (The new normal).

This is joint work from a group at Statistics New Zealand.