Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity‐score matched samples

TLDR

The propensity score represents a subject’s probability of treatment given observed baseline covariates, and when correctly specified, treated and untreated subjects share similar covariate distributions, a property that underlies the validity of propensity‑score matching widely used in medical research. This paper presents methods for assessing correct specification of the propensity‑score model, including standardized differences, variance ratios, higher‑order moment comparisons, five‑number summaries, and various graphical diagnostics. The authors form matched sets of treated and untreated subjects with similar propensity scores and employ statistical and graphical diagnostics—standardized differences, variance ratios, higher‑order moments, five‑number summaries, and plots—to evaluate model specification, including deriving the sampling distribution of standardized differences under the null. The study shows that many existing diagnostics, especially those comparing estimated propensity‑score distributions between groups, are uninformative, underscoring limitations of prior methods for assessing model adequacy.

Abstract

The propensity score is a subject's probability of treatment, conditional on observed baseline covariates. Conditional on the true propensity score, treated and untreated subjects have similar distributions of observed baseline covariates. Propensity-score matching is a popular method of using the propensity score in the medical literature. Using this approach, matched sets of treated and untreated subjects with similar values of the propensity score are formed. Inferences about treatment effect made using propensity-score matching are valid only if, in the matched sample, treated and untreated subjects have similar distributions of measured baseline covariates. In this paper we discuss the following methods for assessing whether the propensity score model has been correctly specified: comparing means and prevalences of baseline characteristics using standardized differences; ratios comparing the variance of continuous covariates between treated and untreated subjects; comparison of higher order moments and interactions; five-number summaries; and graphical methods such as quantile-quantile plots, side-by-side boxplots, and non-parametric density plots for comparing the distribution of baseline covariates between treatment groups. We describe methods to determine the sampling distribution of the standardized difference when the true standardized difference is equal to zero, thereby allowing one to determine the range of standardized differences that are plausible with the propensity score model having been correctly specified. We highlight the limitations of some previously used methods for assessing the adequacy of the specification of the propensity-score model. In particular, methods based on comparing the distribution of the estimated propensity score between treated and untreated subjects are uninformative.