Sampling

Central Limit Theorem

In Sampling Methods we saw how to pick a sample. This post answers the next question: what can a sample tell you about the whole population? The answer comes from the sampling distribution of the sample mean, its standard error and the central limit theorem, which builds on the bell curve from Normal Distribution.

Sampling distribution of the sample mean

A sampling distribution is the probability distribution of a sample statistic, such as the mean, based on repeated random samples from a population. Because of sampling variability, the mean changes from sample to sample, and the distribution of these means is the sampling distribution of the mean.

Imagine you want to know the average time visitors spend on your website or app per session. You do not analyze every session, you analyze a random sample. The sample mean will be close to the population mean, but not exactly equal to it. Take another sample and you get a slightly different mean. The chart shows 16 sample means for samples of 10 sessions and for samples of 100 sessions. The means scatter around the true mean, and larger samples scatter much less. That is how you get an accurate estimate without analyzing every session.

Standard error

The standard error measures the variability of the sample means. It is the standard deviation of the sampling distribution, and it tells you how precise the sample mean is as an estimate of the population mean. In practice you calculate it from your sample:

standard error = s√ns is the sample standard deviation and n is the sample size.

The smaller the standard error, the more precise the estimate. Because n sits under a square root, you need four times as many observations to cut the standard error in half. The table uses the session-time example, with a standard deviation of about 2.6 minutes.

The standard error is also what turns a single sample mean into a range of plausible values, which is the idea behind confidence intervals.

What the central limit theorem says

The central limit theorem describes the shape of that sampling distribution. It states that the sampling distribution of the mean approaches a normal distribution as the sample size increases. In other words, as your sample size grows, the distribution of sample means takes the shape of a bell curve. And as you sample more observations from a population, the sample mean gets closer to the population mean.

Imagine you want to estimate the average session time on a website with millions of sessions per month. Instead of analyzing every session, you collect data on a representative sample. If the sample is large enough, its mean session time will be roughly equal to the population mean.

Example: session time

Imagine you run a web shop with millions of sessions per month, and you want to know how long visitors stay. Analyzing every session is slow and costly, so you work with repeated random samples of 100 sessions. The mean session time of the first sample is 4.2 minutes, of the second 3.9 minutes and of the third 4.1 minutes. The means differ slightly from sample to sample because of sampling variability.

Session times are skewed: most visitors leave quickly, and a few stay very long. The chart shows what happens when you look at the means of many samples from such data. With a sample size of 1, the means are just the session times themselves, and the distribution is skewed. As the sample size grows, the distribution of the means becomes more symmetric and closer to a bell curve, and it centers on the population mean.

With a standard deviation of about 2.6 minutes, a sample of 100 sessions gives a standard error of 2.6 / √100 ≈ 0.26 minutes. In practice, data professionals usually take a single sample. The sample size depends on factors like budget, time, resources and the desired level of confidence in the estimate.

Conditions

To apply the central limit theorem, these conditions must be met:

• Randomization. The sample must be the result of random selection, so every member of the population has an equal chance of being chosen.
• Independence. The values in the sample must be independent: one observation does not affect another. If the sample was selected randomly, you can usually assume independence.
• 10%. To help ensure independence, the sample should be no larger than 10% of the population when you sample without replacement, which is the usual case.
• Sample size. The sample has to be sufficiently large.

There is no exact rule for how large that is. It depends on two things: how precise you need to be (the larger the sample, the closer the distribution is to normal and the more precise the estimate) and the shape of the population. If the population is already roughly bell-shaped, even a small sample works. Many data professionals consider a sample size of 30 sufficient in that case. If the population is extremely skewed or has many outliers, a larger sample is usually preferred, and exploratory data analysis helps you decide how large.

How to explain it to stakeholders