The American Statistician · 2019 · 334 citations · 48 references
Public HealthBiostatisticsExperimental DesignMethod ValidationPsychometricsHypothesis TestingMeasurement ErrorSample SizeStatistical SignificanceTest ValidityValidity EvidenceEffect SizeEffect Size InterpretationP Value InterpretationP-valuesProbability MisinterpretationStatistical ErrorStatistical Significance TestingStatistical Validity
The present note explores sources of misplaced criticisms of P-values, such as conflicting definitions of “significance levels” and “P-values” in authoritative sources, and the consequent misinterpretation of P-values as error probabilities. It then discusses several properties of P-values that have been presented as fatal flaws: That P-values exhibit extreme variation across samples (and thus are “unreliable”), confound effect size with sample size, are sensitive to sample size, and depend on investigator sampling intentions. These properties are often criticized from a likelihood or Bayesian framework, yet they are exactly the properties P-values should exhibit when they are constructed and interpreted correctly within their originating framework. Other common criticisms are that P-values force users to focus on irrelevant hypotheses and overstate evidence against those hypotheses. These problems are not however properties of P-values but are faults of researchers who focus on null hypotheses and overstate evidence based on misperceptions that p = 0.05 represents enough evidence to reject hypotheses. Those problems are easily seen without use of Bayesian concepts by translating the observed P-value p into the Shannon information (S-value or surprisal) –log2(p).
48
A Mathematical Theory of Communication
Claude E. Shannon · Bell System Technical Journal · 1948 · 80.7K citations
null · Philosophy study · 2016 · 28.3K citations · Full text
Statistical Methods for Research Workers
American Journal of Clinical Pathology · 1945 · 11.1K citations
Information theory, inference, and learning algorithms
Choice Reviews Online · 2004 · 6.6K citations