Abstract

Research synthesis (i.e. systematic review and meta-analysis) is the integration of existing knowledge and is the cornerstone of evidence-based medicine, exerting major influence in shaping public health policy and clinical practice, as it resolves seemingly contradicting research findings pertinent to a topic. However, some methodological limitations and challenges in research synthesis urgently need to be addressed to enable timely and high-quality evidence-based practice. In this IPS session, we will discuss limitations of current methods and provide robust alternatives.

Limitation 1: We are in the era of information explosion, with over 10,000 publications indexed per day in Scopus. As a result, well-conducted systematic reviews are often time-consuming (6 months to 2 years) and resource-intensive (up to $200,000). Such long periods are often not practical, when policymakers need timely evidence for decisions that could have huge social, economic, and health implications.

Limitation 2: Research synthesis aims to compile all available evidence; however, in practice, researchers frequently face studies that report the same outcome differently, such as a continuous variable (e.g., scores for rating depression) or a binary variable (e.g., counts of patients with depression dichotomized by certain latent, unreported depression scores). Current methods to combine binary and continuous outcomes may be inaccurate when effect sizes are large or cut-off values for dichotomizing binary events are extreme.

Limitation 3: Randomised controlled trials are often not powered to assess safety outcomes and adverse events. Meta-analysis can increase the power; however, data extraction errors can have an impact on the pool estimates. Currently, it is unknown what are the extents, most common mistakes, and the impact of data extraction errors on recommendations of safety outcome meta-analyses.

Limitation 4: Meta-analysis is often used to synthesise proportions including estimates of disease frequency like incidence or prevalence. In order to obtain better approximation of such data to the normal distribution, researchers use variance-stabilizing transformations including logit, arcsine-square-root, or Freeman–Tukey double-arcsine transformations. Despite the increasing usage, researchers have expressed concerns about the performance of the various transformations.

To address these limitations, in this session we will
1. present robust rapid synthesis approaches by limiting the inclusion of studies by year or publication and number of publications, and the trade-off between workload and precision of point estimates of these approaches;
2. present a Bayesian hierarchical model for combining standardized mean differences and odds ratios in the same meta-analysis;
3. report findings of a large reproducibility study identifying the most common types of errors, the impact of these errors in pooled estimates, and provide recommendations to reduce data extraction error;
4. report findings of simulation studies using MCMC method for estimating the performance of the natural estimator, logit transformed estimator, and Freeman-Tukey double arcsine estimator.