Abstract

Tightening budgets is an ongoing challenge for national statistical offices and the production of official statistics. In particular, data editing processes have been heavily dependent on manual work – data entries are often inspected by statisticians, which is time consuming and costly. Transparency of the editing process is another aspect to be improved as individuals involved in the process may make different judgements on data and results may differ.

One way to improve cost efficiency is to automate the editing process. We present such almost fully automated data editing process for Producer Price Indices at Statistics Finland. It relies on automatic editing and more efficient editing rules to move the incoming price data forward within the process. The data is processed with clear rules and consistent methods, improving not only the efficiency but the transparency as well. Editing is done by mimicking the actions done by human editors: use all available data, compare price development with other products of the same enterprise and use imputation methods in a similar way as has been used so far for missing observations. During the process, metadata is collected extensively, which ensures the transparency of the process.

A higher level of automation brings its own challenges. It requires balancing efficiency of the production and the quality of the final statistical products. Additionally, automation needs sufficient data to work effectively. Therefore, sample sizes may need to increase, which in turn leads to a higher respondent burden for enterprises. While the number of observations that are modified increases in the automated process, the number of observations needing manual inspection falls over 80 per cent.