Abstract

Recently, visualization techniques and effect size measures aimed at maximizing the interpretability and comparability of statistical results are becoming increasingly important, not only in the context of interpreting results from machine learning algorithms but also in empirical research in general as the scientific community shifts its focus in reporting away from p-values. Many such methods have been proposed across disciplines, from individual conditional expectation plots for improved interpretation of black box model results to predicted changes in probability as a more instructive basis for medical decision-making compared to odds ratios. However, most of these quantities are practically motivated and narrowly applicable to a specific setting or field, leading to some inconsistencies and unclarities in the overall methodology. We have developed a formal framework for the consistent derivation of effect size measure definitions and visualization techniques aimed at maximizing the interpretability and comparability of regression results. Specifically, our framework gives a common mathematical setting for such methods and definitions of generalized quantities which may be specified to correspond to, amongst other things, partial dependence plots, marginal effects, adjusted predictions, and predictive comparisons. We achieve this generalization by utilizing probability measures to derive weighted averages over areas of interest for the regressor values. This approach also allows for various different assumptions regarding the dependence structure of the regressors, and, most importantly, for the specification of the generalized quantities to be tailored to each specific research question or meta-analysis. The framework provides a consistent method for deriving point estimates and uncertainty regions for every quantity and may be applied to the results of both frequentist and Bayesian inference. Notably, our generalized version of marginal effects may be specified to separately quantify main and interaction effects. Furthermore, the framework includes a method for comparing the expected distributions of the target variable before and after a change of regressor values, combining estimation and sampling uncertainty in the interest of communicability. In this paper, we first give a brief summary of this theoretical framework and subsequently provide various examples of its practical relevance using real research questions from the fields of interpretable machine learning, multi-analyst studies, and medical decision-making.