Inference & experiments

A/B Testing

A/B testing is a controlled experiment that compares two versions of a website, app or marketing message. The versions differ in a single feature, so any difference in how users behave can be traced back to that one change.

In a business, A/B testing helps stakeholders optimize websites and apps to improve marketing results, revenue or customer experience. Instead of relying on opinions, a team tests a change on a share of its users, for example a new checkout button, a different headline or a redesigned onboarding flow, and decides based on evidence.

The results are judged with metrics such as average revenue per user, session duration, click rate and conversion rate.

Why do we need A/B testing?

Observed differences between two versions may simply be due to random chance. Statistical inference helps determine whether a difference is likely to be genuine before a decision is made.

Imagine an online shop changes its checkout button to green. In the following week, the conversion rate rises from 5.0% to 5.6%. Did the button cause this? Not necessarily. Conversion rates move up and down from week to week by chance, and that week might also have included a sale or a holiday.

The two problems A/B testing is exceptionally good at solving are:

• Confusing correlation with causation (spurious correlation). The conversion rate rose after the button change, but the sale may be the real reason. In an A/B test, both versions run at the same time, so the sale affects both groups equally.
• Selection bias. If customers could choose the new button themselves, the ones who choose it might be the keenest buyers anyway. In an A/B test, users are assigned to a version randomly, so neither group is favored from the start.

How to design an A/B test

This is a common interview question. A clean answer walks through five steps: variables, hypothesis, metrics, assigning users to groups, and analysis of the results. The running example below is a messaging app that wants to test group video calling.

Step 1: Define your variables

Start by naming the cause you want to test and the effect you want to measure. This keeps the experiment focused on one relationship.

• Independent variable. The cause, the thing you deliberately change. It is not influenced by anything else in the experiment. Here: whether a user has access to group video calling.
• Dependent variable. The effect, the outcome you measure. Its value is expected to change because of the independent variable. Here: the number of calls per user.

A more complex test might compare several versions of the feature, or several settings of the same feature. The logic stays the same: you change the independent variable and watch what happens to the dependent variable.

Step 2: Formulate your hypothesis

A good experiment starts with a clear, testable hypothesis: a statement that connects the feature to an expected outcome. A strong hypothesis has three parts:

• The change. What are you introducing?
• The expected impact. What do you think will happen?
• The metric. How will you measure it?

For the statistical test later on, the hypothesis is written as a pair of competing statements:

• Null hypothesis (H0). The change has no effect. Group video calling does not change the number of calls per user.
• Alternative hypothesis (Ha). The change does have an effect. Group video calling increases the number of calls per user.

The test then asks whether the data give enough evidence to reject the null hypothesis.

Step 3: Choose metrics

The goal is not to track activity, but to measure whether the feature is delivering real value.

• Primary metric. Directly reflects the success of your hypothesis. For example, average time spent per user or number of calls per user when the hypothesis is about increasing engagement.
• Supporting metrics. Provide context and help explain why the primary metric changed. Examples: average order value, bounce rate, or session duration (if conversion rate is measured).
• Guardrail metrics. Ensure you don't improve one aspect of the product while harming another.

Step 4: Assign users to treatment and control groups

An A/B test is a controlled experiment. The treatment is the change being tested, and users are split into two groups:

• Control group continues with the existing experience (1:1 video calling).
• Treatment group gets access to the new feature (group video calling).

The difference in the metric between the two groups is the measured effect of the treatment. For example, if the treatment group averages 4.4 calls per user and the control group 4.0, the feature added 0.4 calls per user. This only works if exposure to the feature is the only meaningful difference between the groups. If the treatment group happened to contain mostly power users, you could not tell whether the feature or the user mix caused the gap.

An important decision is who to include in the experiment.

Key principle: randomization

Randomization is important because it:

• Creates comparable groups, so the only systematic difference is the tested change.
• Reduces selection bias by giving each participant an equal chance of being assigned to either group.
• Balances known and unknown factors (such as age, device, or user behavior) across groups on average.
• Supports causal conclusions, allowing you to attribute differences in outcomes to the treatment rather than other variables.
• Makes statistical tests valid, since most inferential methods assume random assignment.

Why it matters

Without randomization, it's difficult to know whether the change caused the result or whether some other factor did. Such a factor is called a confounder, and confounders can lead to incorrect conclusions about cause and effect.

Two ways to randomize

• Completely randomized design. Every user is assigned to control or treatment by a random process, for example a random number generated for each user ID. It is simple and works well when the sample is large.
• Randomized block design. You first sort users into blocks of similar users, then randomize within each block. It is used to handle known nuisance factors.

A nuisance factor is something that affects the outcome but is not what you are testing. Suppose mobile users make far fewer calls than desktop users. With a small sample, pure randomization could by chance put more desktop users in one group and distort the comparison. To prevent this, block by device: split users into mobile and desktop, then randomize within each block. If a difference appears inside a block, you can be more confident that the feature caused it and not the device.

Also think about how long to run the experiment. The duration should be long enough to capture typical user behavior, but not so long that it delays decision-making.

Checklist for reducing bias

Before launching an A/B test, ask yourself:

• ✅ Have I clearly defined who is eligible?
• ✅ Are participants assigned randomly?
• ✅ Are the control and treatment groups similar in important characteristics?
• ✅ Are both groups exposed to the same conditions at the same time?
• ✅ Can participants influence which group they're assigned to? (If yes, redesign the assignment.)
• ✅ Have I verified that the allocation matches the intended split?

Step 5: Analyse results and draw conclusions

The question to ask: "Did B perform better because of our change, is the improvement large enough to matter, and did it avoid causing other problems?"

Effect size: absolute vs relative

• Absolute improvement. The difference between the groups.
• Relative improvement. The percentage increase compared with the original.

Statistical significance and the p-value

If there were actually no difference between A and B, how likely is it to observe a difference this large? If that is unlikely (commonly p < 0.05), the difference is considered statistically significant.

Confidence intervals

A confidence interval shows the range of plausible effects. It tells us, for example: based on this experiment, the true improvement is plausibly between 0.40% and 0.80%.