A spreadsheet for deriving a confidence interval, mechanistic inference and clinical inference from a P value

TLDR

The null‑hypothesis significance test based solely on a p‑value can mislead inference about the true effect magnitude, yet most statistical packages do not provide uncertainty‑based, magnitude‑oriented conclusions. This paper introduces a spreadsheet that, using the p‑value, observed effect, and minimal substantial effect thresholds, generates mechanistic and clinical magnitude‑based inferences. The spreadsheet evaluates mechanistic uncertainty by checking whether the confidence interval overlaps substantial positive or negative values, and clinical uncertainty by assessing whether the probability of benefit meets a promising threshold while harm remains acceptable, allowing users to set confidence levels and benefit/harm cutoffs and applying the method to common effect statistics under normal or t‑distribution assumptions.

Abstract

The null-hypothesis significance test based only on a p value can be a misleading approach to making an inference about the true (population or largesample) value of an effect statistic. Inferences based directly on the uncertainty in the true magnitude of the statistic are more comprehensible and practical but are not provided by statistical packages. I present here a spreadsheet that uses the p value, the observed value of the effect and smallest substantial values for the effect to make two kinds of magnitude-based inference: mechanistic and clinical. For a mechanistic inference the spreadsheet shows the effect as unclear if the confidence interval, which represents uncertainty about the true value, overlaps values that are substantial in a positive and negative sense; the effect is otherwise characterized with a statement about the chance that it is trivial, positive or negative. For a clinical inference the effect is shown as unclear if its chance of benefit is at least promising but its risk of harm is unacceptable; the effect is otherwise characterized with a statement about the chance that it is trivial, beneficial or harmful. The spreadsheet allows the researcher to choose the level of confidence (default, 90%) for mechanistic inferences and the threshold chances of benefit (default, 25%) and harm (default, 0.5%) for clinical inferences. The spreadsheet can be used for the most common effect statistics: raw, percent and factor differences in means; ratios of rates, risks, odds, and standard deviations; and correlations. The calculations are based on the same assumption of a normal or t sampling distribution that underlies the calculation of the p value for these statistics.