The Importance of Beta, the Type II Error and Sample Size in the Design and Interpretation of the Randomized Control Trial

TLDR

Many therapies labeled “no different from control” in trials with inadequate samples have not received a fair test, and the risk of missing important therapeutic improvements due to small sample sizes warrants greater attention in trial planning. The study re‑examined 71 negative RCTs to assess whether their sample sizes were sufficient to achieve >90 % power to detect 25 % or 50 % therapeutic improvements. The authors re‑analyzed each trial’s data, calculating the probability of detecting the specified effect sizes given the reported sample sizes. They found that 67 trials had >10 % risk of missing a 25 % improvement, 50 trials had the same risk for a 50 % improvement, and 57 trials could still have a 25 % improvement while 34 could still have a 50 % improvement within 90 % confidence intervals.

Abstract

Seventy-one "negative" randomized control trials were re-examined to determine if the investigators had studied large enough samples to give a high probability (greater than 0.90) of detecting a 25 per cent and 50 per cent therapeutic improvement in the response. Sixty-seven of the trials had a greater than 10 per cent risk of missing a true 25 per cent therapeutic improvement, and with the same risk, 50 of the trials could have missed a 50 per cent improvement. Estimates of 90 per cent confidence intervals for the true improvement in each trial showed that in 57 of these "negative" trials, a potential 25 per cent improvement was possible, and 34 of the trials showed a potential 50 per cent improvement. Many of the therapies labeled as "no different from control" in trials using inadequate samples have not received a fair test. Concern for the probability of missing an important therapeutic improvement because of small sample sizes deserves more attention in the planning of clinical trials.