Abstract

Comparing the performance of different measurement systems and quantifying their level of agreement is an important challenge in medical and industrial contexts‎. The probability of agreement (PoA) was recently introduced as an intuitive metric that assesses the agreement between an established measurement system and a new one. The PoA is the probability that the difference between single measurements on the same subject by the two systems falls within the range that is deemed to be clinically or practically acceptable. Defining the PoA in this way makes its interpretation easy, even for non-statisticians, no matter how complicated the underlying model and estimation procedure might be. The PoA may be visualized with the probability of agreement plot which helpfully summarizes the agreement between systems. The current PoA methodology is based on a one-factor random effects model that represents measurement variation from all sources with a single random effect. However, in clinical analysis and quality improvement efforts, a measurement system is often used by multiple operators to collect data. Accounting for their effects is an important part of understanding the measurement variation, and evaluating the operators' effects separately is essential for assessing their contribution to disagreement between systems should it arise. In this presentation we develop the required methodology for applying the PoA when the influence of operators is separately accounted for in the measurement error model (either as fixed or random effects). The developed methodology accounts for unbalanced replicate measurements across operators and systems‎. The methodology is illustrated on a dataset of respiratory rates of Chronic Obstructive Pulmonary Disease (COPD) patients‎. The proposed PoA analysis is used to assess the agreement between a gold standard device and a chest-band.