Abstract

Digital health interventions deliver health services electronically through formal care when implemented in hospitals or general practices. Digital health interventions can range from electronic medical records used by health care professionals in hospitals and/or general practices to mobile health apps used by consumers. Digital health interventions generally involve complex interactions between users, technology, and the healthcare team, posing challenges for implementation and evaluation. For example, electronic medication management (eMM) systems, containing patients’ medical and clinical data, are expected to improve quality of care delivery in hospitals, including reducing medication errors. However, rigorous evidence demonstrating these effects is relatively limited.
Cluster randomized controlled trials (CRCTs) are increasingly being used to evaluate digital health interventions when it is inappropriate or impossible to use individual randomization. CRCTs commonly use a parallel group design, in which the clusters are randomized to either the intervention or control arm of the study. It is often regarded as unethical to withhold an intervention from a proportion of participants if it is believed that the intervention will do more good than harm. Stepped wedge CRCT design (SW-CRCT), where the intervention is delivered sequentially to all trial clusters over a number of time periods, is an alternative to the traditional parallel groups design. The objectives of this paper are to describe the design and analysis of CRCTs and SW-CRCTs for evaluating the effectiveness of digital health interventions, and to discuss the practical and methodological challenges of these trials. Two specific case studies are drawn on: 1) an evaluation of the effectiveness of a mobile health app to achieve target serum urate concentrations in people with gout using a CRCT, 2) an evaluation of the effectiveness of an eMM system at a pediatric hospital to reduce medication errors using a SW-CRCT. Sample size calculations take into account the estimated between-cluster variance and the design effect associated with the cluster and stepped-wedge design. The main outcome of interest may include changes in specific outcomes such as test results or medication error rate following implementation of the interventions. For each outcome of interest, data collected across all measurement periods and all study steps are used in the analyses comparing intervention status (pre versus post intervention). The analyses consider the correlation within each cluster and include multiple time points for both pre and post intervention. The SW-CRCT design potentially also allows determination of temporal changes in system effectiveness, e.g., to determine if error rates continue to decline over time.