Abstract

National Statistics Institutes (NSIs) are facing numerous challenges. Although surveys have served as a solid foundation for producing official statistics for more than 50 years, increasing costs, decreasing response rates, and decreasing list frame coverage are raising concerns about the quality of the resulting estimates. At the same time, data users are wanting more official statistics reported at finer temporal and spatial scales. In the era of big data, administrative and other data are becoming increasingly available. Consequently, many NSIs are actively moving to either integrate survey and non-survey data or to use only non-survey data to produce official statistics. In this session, speakers from four different countries will discuss the progress being made and the problems yet to be solved if NSIs are to truly leverage all available data in the production of official statistics. Anders Holmberg, Australian Bureau of Statistics, will focus on ways to find a balance between cost efficient statistics production and the thirst for timely data analytics, while acknowledging the paradoxes and trade-offs. Ron Jarmin, U.S. Census Bureau, will discuss bridging the gap between theory and practice when using all available data, including how building statistics for the retail trade sector from transaction level data can yield fundamental improvements in both price and quantity measurement, the investments in statistical infrastructure needed leverage the value of private sector data, and issues that require solutions to scale the use of these data for official statistics. Ton de Waal, Statistics Netherlands, will explore some of the theoretical challenges when using diverse data, such as measuring the quality of estimates (partly) based on non-probability data with a focus on assessing selection bias in practical situations and on assessing the effect of measurement error on the quality of estimates. Andrea Diniz da Silva, Brazil's Escola Nacional de Ciencias Estatisticas - ENCE, will highlight recent achievements of the Regional Hub for Big Data in Brazil, one of four regional hubs created by the United Nations to provide support to NSIs, for training and fostering the interest of young statisticians on the use of big data and for the practices and challenges regarding the use of big data for statistics in Latin America and the Caribbean Region. Linda J. Young, USDA National Agricultural Statistics Service, will discuss the common issues identified by all four speakers and potential opportunities for collaboration as NSIs move rapidly to improve estimates using all available information.