Why P-Values Aren't Enough: The Case for Effect Sizes in Psychology
Direct Summary (TL;DR)
A p-value only tells you whether an effect is unlikely to be zero under the null hypothesis—it never tells you how large, important, or replicable that effect actually is.
Because p-values conflate sample size (N) with effect size (d), relying solely on statistical significance leads to two dangerous research errors:
- Trivial effects crowned as major discoveries in massive samples (e.g., N = 50,000, p < .001, d = 0.04).
- Potentially transformative interventions discarded in small, underpowered samples (e.g., N = 16, p = .09, d = 0.75).
To provide true scientific insight, the American Psychological Association (APA 7th Edition) and the Open Science movement mandate reporting standardized effect sizes (like Cohen's d) alongside exact p-values and confidence intervals.
What a P-Value Actually Measures (and What It Doesn't)
A p-value is the probability of observing data as extreme as (or more extreme than) what was measured, assuming the null hypothesis (H₀) is strictly true.
Common P-Value Fallacies
| Common Misinterpretation | The Accurate Scientific Reality |
|---|---|
| ❌ "p = .03 means the hypothesis is 97% likely to be true." | False: P-values assume the null is true; they do not calculate the probability of the alternative hypothesis. |
| ❌ "p = .03 means there is a 3% probability the finding is a fluke." | False: A p-value is a conditional probability of the data, not an error probability of the finding. |
| ❌ "A smaller p-value reflects a larger, more impactful effect." | False: A tiny p-value can result entirely from an enormous sample size with a negligible effect. |
| ❌ "p > .05 proves that there is no difference between groups." | False: Failing to reject the null hypothesis is not proof of the absence of an effect (absence of evidence ≠ evidence of absence). |
The Sample Size Sensitivity Problem
The fundamental limitation of null hypothesis significance testing (NHST) is illustrated below. Consider a fixed treatment difference of 2 points on an anxiety inventory (SD = 10):
| Study Scenario | Sample Size per Group (n) | True Effect Size (Cohen's d) | Calculated t-statistic | p-value | NHST Verdict (α = .05) |
|---|---|---|---|---|---|
| Study A | 20 | 0.20 (Small) | 0.89 | .376 | ❌ "Not Significant" |
| Study B | 100 | 0.20 (Small) | 2.00 | .047 | ✅ "Statistically Significant" |
| Study C | 500 | 0.20 (Small) | 4.47 | < .0001 | ✅ "Highly Significant" |
| Study D | 5,000 | 0.20 (Small) | 14.14 | < .00001 | ✅ "Extremely Significant" |
Analysis of the Paradox
In all four scenarios, the true biological and psychological reality is identical (d = 0.20). Yet in Study A, the researcher may abandon the line of inquiry, while Study D presents the exact same modest effect as an earth-shattering statistical milestone.
Effect sizes isolate the signal from sample-size noise.
Statistical Significance vs. Practical Significance
┌──────────────────────────────────────────┐
│ Effect Magnitude (Cohen's d) │
│ Negligible Substantial │
┌───────────────┼────────────────────┬─────────────────────┤
│ p < .05 │ Type I Risk: │ Optimal Outcome: │
│ (Significant) │ Statistical Noise │ Meaningful Finding │
│ │ (Large Sample Bias)│ (High Power + Effect│
├───────────────┼────────────────────┼─────────────────────┤
│ p ≥ .05 │ Correct Rejection: │ Type II Risk: │
│ (Non-Signif.) │ True Null │ Underpowered Study │
│ │ │ (Real Effect Missed)│
└───────────────┴────────────────────┴─────────────────────┘
Real-World Scenario: Clinical vs. Statistical Significance
Imagine a pharmaceutical trial testing a new sleep aid against placebo (N = 20,000):
- Result: Participants fell asleep 1.8 minutes faster (p < .0001, d = 0.05).
- Conclusion: Statistically undeniable, but clinically meaningless to an insomniac. Without reporting effect size, this drug could be marketed as "highly significant."
The "New Statistics" Framework (Cumming, 2014)
In response to the replication crisis in psychology, modern quantitative methodology emphasizes the New Statistics:
- Shift from Dichotomous Thinking to Estimation: Move away from black-and-white "p < .05" verdicts. Focus on how much change occurred.
- Report Point Estimates and Confidence Intervals: Always accompany effect sizes with 95% confidence intervals to illustrate measurement precision.
- Pre-Registration and Power Planning: Determine required sample sizes before data collection using formal sample size power calculations.
- Meta-Analytic Perspective: Evaluate individual studies as cumulative contributions to meta-analyses rather than standalone final words.
APA 7th Edition Statistical Reporting Standard
The APA 7th edition manual strictly requires effect sizes for all quantitative findings. Here is a model results paragraph:
An independent-samples t-test was conducted to examine the effect of cognitive reappraisal training on stress reactivity. Participants in the reappraisal condition (n = 28, M = 19.40, SD = 4.20) exhibited significantly lower salivary cortisol levels following the stressor compared to control participants (n = 28, M = 24.10, SD = 4.80), t(54) = 3.90, p < .001, d = 1.04, 95% CI [0.48, 1.60]. The effect size was large according to Cohen's (1988) benchmarks, indicating that cognitive reappraisal produced a substantial attenuation of physiological stress.
Which Effect Size Should You Report?
| Research Design / Analysis | Primary Effect Size Metric | Alternative / Corrected Metric |
|---|---|---|
| Two Independent Group Means | Cohen's d | Hedges' g (for N < 40) |
| Paired / Repeated Measures | Cohen's dz or drm | Repeated-measures g |
| Linear Association (2 Continuous Variables) | Pearson's correlation (r) | Spearman's rank (rho) |
| ANOVA (3+ Groups, Factorial) | Partial Eta-Squared (ηp²) | Omega-Squared (ω²) |
| Multiple Linear Regression | Coefficient of Determination (R²) | Cohen's f² |
| 2x2 Categorical Contingency | Odds Ratio (OR) / Relative Risk | Phi coefficient (φ) / Cramér's V |
Key Takeaways
- A p-value answers "Could this be random noise?", while an effect size answers "How much does it matter in reality?".
- Large samples can generate statistically significant p-values for imperceptible differences.
- Small samples with non-significant p-values may harbor large, promising effects that simply require replication with adequate power.
- Always report effect sizes with confidence intervals alongside exact p-values in compliance with APA 7th edition standards.