Testing a simple null against a simple alternative

How α, β and power depend on the critical value

Beta version. This explorer has not yet been fully tested. Please check important results independently, and report problems on the issues page.

Educational use only. This explorer is a teaching tool for building statistical literacy. It is not intended for determining test results in situations where real decisions depend on them.

For a simple null hypothesis H₀: θ = θ₀ against a simple alternative H₁: θ = θ₁, suppose that the test statistic T is normally distributed with mean μ₀ under H₀ and μ₁ under H₁, and the same variance Var(T) under both. Change the means, the variance or the critical value, and watch the two error probabilities and the power respond. You can also drag the critical value in the plot.

Densities of T under H₀ and H₁ with the critical value c
Model for T

Leave this off for any other normally distributed test statistic, and enter its variance directly.

Test

Keeps α at most 0.15 and zooms the trade-off plot to that range. Switch off to explore every α.

When μ₀, μ₁ or σ² change, keep fixed:

Significance level αP(Type I error) = P(Reject H₀ | H₀ true)
βP(Type II error) = P(Accept H₀ | H₀ false)
Power 1 − βP(Reject H₀ | H₀ false)

The trade-off between α and power

Power 1 − β plotted against α for every choice of c An increasing curve from (0, 0) to (1, 1) above the diagonal: lowering α also lowers the power. The current choice of c is marked as a point.

Each point on the curve is one critical value c. Moving c buys power only at the price of a larger α. A larger standardised distance lifts the whole curve towards the corner (0, 1), where α is small and the power is high. The dashed line is a test that ignores the data, where the power equals α.

What the explorer assumes

Under H₀, T ~ N(μ₀, Var(T)); under H₁, T ~ N(μ₁, Var(T)). If T is the sample mean X̄ of n independent observations with variance σ², then E(X̄) = μ and Var(X̄) = σ²/n, so only the variance of T changes with n. With equal variances, the most powerful test (Neyman–Pearson lemma) uses one critical value: reject H₀ when t > c if μ₁ > μ₀, and when t < c if μ₁ < μ₀.