The Behavior of the P-Value When the Alternative Hypothesis is True

TLDR

The P‑value, a random variable derived from the test statistic distribution, is uniformly distributed under the null but varies with sample size and true parameter values under the alternative, offering insight into its behavior across different scenarios. The study explores how the alternative‑hypothesis P‑value distribution can inform the design, analysis, and interpretation of clinical trials.

Abstract

The P-value is a random variable derived from the distribution of the test statistic used to analyze a data set and to test a null hypothesis. Under the null hypothesis, the P-value based on a continuous test statistic has a uniform distribution over the interval [0, 1], regardless of the sample size of the experiment. In contrast, the distribution of the P-value under the alternative hypothesis is a function of both sample size and the true value or range of true values of the tested parameter. The characteristics, such as mean and percentiles, of the P-value distribution can give valuable insight into how the P-value behaves for a variety of parameter values and sample sizes. Potential applications of the P-value distribution under the alternative hypothesis to the design, analysis, and interpretation of results of clinical trials are considered.