Inference & experiments

Hypothesis Testing

After a checkout redesign, the average order value seems to have increased. Did the new design make a difference, or was it just a lucky week?

That is what statistical significance tells us. Hypothesis testing checks whether a difference we observe is strong enough evidence of a real effect, or whether random variation could explain it. It builds on Confidence Intervals: they estimate a range of plausible values, while a test evaluates evidence against a specific assumption.

We’ll start with the core concepts, walk through the checkout example step by step, and then cover one- and two-tailed tests, t-tests, one- and two-sample comparisons, and power. To compare two versions of something, see A/B testing.

Null and alternative hypothesis

Every hypothesis test starts with two competing statements about the population.

We assume H₀ is true unless the data provide strong evidence against it. The alternative hypothesis describes the effect we want to investigate.

The symbols depend on the research question:

• Change: H₀: μ = μ₀, H_a: μ ≠ μ₀
• Increase: H₀: μ ≤ μ₀, H_a: μ > μ₀
• Decrease: H₀: μ ≥ μ₀, H_a: μ < μ₀

Think of a courtroom: a defendant is presumed innocent until evidence proves otherwise. Similarly, we reject H₀ when the evidence is sufficiently strong. Otherwise, we fail to reject H₀, but that doesn't prove it is true.

Significance level and p-value

The significance level (α) is the threshold you set before testing. It defines the acceptable risk of a Type I error: rejecting H₀ when it is actually true. A common choice is 0.05 (5%).

The p-value is your data’s surprise meter: how likely you would be to observe results at least as extreme as yours if H₀ were true. A small p-value means the results are surprising under that assumption; a large one means they are less surprising.

The decision rule is simple:

• If p < α, reject H₀.
• If p ≥ α, fail to reject H₀.

Remember: α is your threshold; the p-value is calculated from your data. Neither tells you the probability that H₀ is true or whether an effect is practically meaningful.

The rejection region

If H₀ is true, the test statistic (here a z-score) follows a bell curve centered on 0. Most results land near the middle, and extreme results are rare. The significance level α decides how much of the curve counts as “too extreme”. With α = 0.05, the most extreme 5% of results fall in the rejection region. In a two-tailed test, that 5% is split between the two tails, 2.5% each, and the cut-off points are called critical values.

• Result in the middle (blue). Fail to reject H₀: there isn't enough evidence.
• Result in a red tail. Reject H₀: the result is too extreme to blame on chance, so it counts as evidence for H_a.

For α = 0.05 and a two-tailed test, the critical values are −1.96 and +1.96. These are the same z-scores behind a 95% confidence interval.

Steps of hypothesis testing

Now let's run the full test on the checkout redesign. Before the change, orders in the shop averaged €60, and from past data we know the standard deviation is €20. After the redesign, we look at 100 orders and the average is €64. Is that €4 difference real, or could it be chance?

1. Start with the null hypothesis. This is the idea that the redesign changed nothing: the average order value is still €60 (H₀: μ = 60).
2. Set the alternative hypothesis. This says the redesign did change something: the average is no longer €60 (H_a: μ ≠ 60).
3. Decide how sure you want to be. Before you say the redesign made a difference, you fix the significance level. We use the usual 5% (α = 0.05).
4. Calculate the p-value. This is the probability of getting results like ours if the redesign really changed nothing. With 100 orders, the standard error is 20 / √100 = 2, so z = (64 − 60) / 2 = 2.0: our average is two standard errors above €60 (the dot in Chart 1). That gives a two-tailed p-value of about 0.046.
5. Make the decision. If this probability is very low, below your significance level, you decide the redesign probably did change something. Here 0.046 is below 0.05, so we reject H₀: the data give statistical evidence that the average order value has changed. If it had been above 0.05, we would say we don't have enough evidence.

z = x̄ − μ₀σ√nx̄ is the sample mean, μ₀ the value under H₀, σ the standard deviation and n the sample size.

Notice how close the call is: p = 0.046 against a threshold of 0.05. With a slightly smaller difference we would have ended up at “fail to reject”. The test gives you a clear rule for deciding, but it is not a guarantee.

One-tailed or two-tailed?

The alternative hypothesis decides where the rejection region sits.

• One-tailed test. H_a says the parameter is either greater than or less than the null value (μ > μ₀ or μ < μ₀). It looks in one direction only, so all of α goes into one tail: the right tail or the left tail. For α = 0.05, the critical value is 1.645.
• Two-tailed test. H_a says the parameter is not equal to the null value (μ ≠ μ₀). It looks for a change in both directions, so α is split between the two tails. For α = 0.05, the critical values are ±1.96.

In our example, asking “did the average order value increase?” is a right-tailed test, and the p-value comes from the right tail only. Asking “is it different from €60?” is a two-tailed test, and the p-value comes from both tails. With z = 2.0, that gives a one-tailed p-value of about 0.023 and a two-tailed p-value of about 0.046.

How do you choose?

• One-tailed: more powerful for detecting an effect in one direction, but it can never detect an effect in the opposite direction.
• Two-tailed: the better choice when a change in either direction matters. If a redesign that lowers order value would be just as important to know about as one that raises it, use two tails.

When σ is unknown: the t-test

The z-test assumes you know the population standard deviation σ. In practice you almost never do, so you estimate it with the sample standard deviation s. The test statistic then becomes a t-statistic. Because s is itself uncertain, t follows the t-distribution, which has heavier tails than the normal curve. How heavy depends on the degrees of freedom (df = n − 1 for one sample): the smaller the sample, the heavier the tails.

t = x̄ − μ₀s√nSame shape as z, but with the sample standard deviation s instead of σ. You compare t with a critical value from the t-distribution with n − 1 degrees of freedom.

Here is why this matters. Imagine a small pilot of the new checkout: 25 orders with an average of €64 and a sample standard deviation of €10, against the historical average of €60. The standard error is 10 / √25 = 2, so t = (64 − 60) / 2 = 2.0, the same value as before. But with 24 degrees of freedom the critical value is 2.064, so t = 2.0 does not reach it. We fail to reject (two-tailed p ≈ 0.057). A z-test on the same numbers would have rejected (p ≈ 0.046). A small sample carries less information, and the t-test takes that into account.

With large samples the two tests give almost the same answer, which is why z works as an approximation when n is large.

Two ways to be wrong

A test is a decision under uncertainty, so it can go wrong in two ways. Think of a smoke alarm: it can go off because you burned the toast, or stay silent during a real fire.

• Type I error (false alarm). You reject H₀ although it is true. Its probability is α. Like the alarm going off for burnt toast.
• Type II error (missed effect). You fail to reject H₀ although it is false. Its probability is β. Like the alarm staying silent in a real fire.

Back to the checkout redesign, the four possible outcomes look like this:

A wasted rollout costs the development budget. A missed winner costs about €4 on every future order. Which mistake hurts more should guide how you choose α.

Lowering α reduces false alarms, but it makes missed effects more likely, unless you also collect more data.

One-sample and two-sample tests

So far we have compared one sample with a known value. That is a one-sample test: you collect one group of data and check whether its mean is equal to a specific number. Think of the average delivery time of a food service, claimed to be 30 minutes. You time a sample of deliveries and ask whether the average is close enough to 30, or different enough to say the claim is wrong.

A two-sample test compares two groups with each other instead of with a fixed number. It asks whether the two population means are equal. This is exactly what happens in an A/B test: customers are randomly split into a control group (version A) and a treatment group (version B), and the two-sample test tells you whether the difference in your metric is statistically significant. H₀ says the two versions perform the same, μ_A = μ_B. H_a says they don't.

The two-sample t-test compares the means of two independent groups. It assumes independent observations and approximately normal populations, or sufficiently large samples. Welch’s t-test also allows the groups to have different variances.

The test statistic measures the difference between the sample means relative to its standard error:

t = x̄_A − x̄_B√( s_A²n_A + s_B²n_B )The larger the difference relative to its uncertainty, the stronger the evidence against H₀.

The testing procedure remains the same: state the hypotheses, choose α, calculate the p-value, and make a decision.

Example: Comparing two checkouts

An A/B test produces the following results:

• Old checkout (A): 100 orders, average value €60
• New checkout (B): 100 orders, average value €64
• Both groups: standard deviation €20

The standard error is approximately €2.83, giving t ≈ 1.41 and a two-sided p-value of approximately 0.16.

Since p > 0.05, we fail to reject H₀. The observed €4 difference is not statistically significant at the 5% level.

Why? Unlike a test against a fixed benchmark, an A/B test estimates both group means from samples. Each estimate carries uncertainty, making the difference harder to establish confidently.

This brings us to the next question: How many observations do we need to detect a real effect? That is where statistical power and sample size come in.

Power and sample size

α directly sets the probability of a Type I error. The probability of a Type II error depends on how you design the study. The chance of detecting a real effect is called the power of a test: power = 1 − β. A common target is 80%.

Suppose the redesign really does raise the average order value from €60 to €64, with σ = €20 and n = 100. That true effect is 2 standard errors away from the null, so the test statistic centers on 2.0 instead of 0. Only the part beyond 1.96 leads to a rejection, which happens about 52% of the time: power ≈ 0.52 and β ≈ 0.48. So even though the effect is real, this test would miss it almost half the time. That is why “fail to reject” never proves “no effect”.

Four things drive power:

• Effect size. A larger true difference is easier to detect.
• Sample size. A larger n shrinks the standard error and pulls the two curves apart.
• Variability. A smaller σ has the same effect as a larger sample.
• Significance level. A higher α raises power, but also the risk of false alarms.

You can also turn this around and use it before collecting data to plan the sample size for a two-tailed z-test.

n = ( (z_α/2 + z_β) × σδ )²δ is the smallest difference worth detecting. For α = 0.05, z_α/2 = 1.96, and for 80% power, z_β = 0.84.

For our example, n = ((1.96 + 0.84) × 20 / 4)² ≈ 196. So the shop needs roughly 200 orders, not 100, to have an 80% chance of detecting a €4 increase. The same logic carries over to comparing two groups in A/B Testing.

Common mistakes

• “We accept H₀.” Failing to reject doesn't prove H₀ is true; it only means the evidence wasn't strong enough.
• Confusing significant with important. With enough data, even a tiny, useless difference can be statistically significant.
• Choosing α or the tail direction after seeing the data. Decide both beforehand, otherwise the test is biased.
• Using z when σ is estimated from a small sample. Use the t-test instead, because z makes the evidence look stronger than it is.
• Ignoring power. A non-significant result from a small sample often means the test was too weak, not that there is no effect.
• Reporting only the test that worked. At α = 0.05, about 1 in 20 tests on pure noise looks significant.

How to explain it to stakeholders