Abstract

Web surveys are increasingly used for official statistics, public health research, social and marketing research, and opinion polls. Most general population web surveys are based on online panels maintained by commercial survey agencies. Survey agencies differ in their panel management and sampling strategies, using probability and non-probability sampling. However, web surveys generally lack a suitable sampling frame for the general population. In addition, web surveys suffer from potential undercoverage and high nonresponse rates, which might cause biased estimates. Finally, internet access and use are often correlated with demographic variables. Therefore, potential bias in web surveys might depend on the sampling and panel management strategy. Little is known if these different strategies cause differences in survey estimates. This paper presents results from a systematic validation study specifically designed to analyze the effect of different panel strategies on differences from results based on probability face-to-face surveys or census data.
Five agencies were commissioned simultaneously with the same web survey using an identical standardized questionnaire on 36 factual health items. An additional independent sixth web survey was commissioned six months later. Two of the six web surveys use probability sampling. In total, 26.446 respondents answered. A calibration approach was used to control the effect of standard demographics on the point estimates. The web survey point estimates were compared with the point estimates of three large-scale probability-based face-to-face health surveys of the general population and census data. Results show smaller biases for one of the web surveys using probability sampling; for the other probability-based web survey, results are mixed. Non-probability-based web surveys usually show larger biases.