Probability

Normal Distribution

Most people have seen the bell curve before. Knowing what it looks like is not the same as knowing what it tells you. This post starts with how any continuous distribution works, then looks at the normal distribution and the empirical rule, and ends with a simple way to spot outliers. It builds on Random Variables & Probability Distributions and Four Common Discrete Distributions.

Probability Density

A probability function gives the probabilities for the possible outcomes of a random variable. There are two types:

• Probability mass function (PMF). Describes discrete random variables.
• Probability density function (PDF). Describes continuous random variables.

The graph of a PDF is a curve. On it, the x-axis shows the value of the variable you measure, and the y-axis shows the probability density. Probability density is not the same as probability. For continuous variables, probability comes from the area under the curve.

As an example, take a random sample of cherry trees. Their heights are approximately normally distributed with a mean of 15 feet and a standard deviation of 2 feet. What is the probability that a randomly chosen tree measures exactly 15.1 feet? Essentially zero. A continuous variable has infinitely many possible values: 15.1, 15.175, 15.175245 and so on, so no single value carries a meaningful probability.

For continuous distributions it only makes sense to talk about intervals, such as the interval between 14.5 and 15.5 feet. For this interval the area is about 0.20, so there is roughly a 20% chance that a tree falls between 14.5 and 15.5 feet.

The Normal Distribution

The normal distribution is a continuous probability distribution that is symmetric about the mean and bell-shaped. It is also called the Gaussian distribution, after the German mathematician Carl Gauss, who first described its formula. It is the most common distribution in statistics because so many datasets follow a bell shape: height, weight, blood pressure, shoe size, test scores and more.

All normal distributions share these features:

• Mean at the center. The mean is also the peak: the most probable outcome. For the trees, that is 15 feet.
• Symmetry. 50% of the data lies above the mean and 50% below.
• Spread. The standard deviation controls how wide or narrow the curve is.
• Less likely further out. The further a value is from the mean, the lower its probability. The least probable outcomes are the very short and very tall trees.
• Total area equal to 1. The area under the curve covers 100% of the possible outcomes.

The 68–95–99.7 Rule (Empirical Rule)

The values on a normal curve are spread in a regular pattern, based on their distance from the mean. This is the empirical rule, also known as the 68-95-99.7 rule.

In other words, most trees (68%) measure between 13 and 17 feet, and almost all of them (99.7%) measure between 9 and 21 feet. The rule gives you a quick estimate of how the values in a large dataset are distributed, without any calculation. That saves time and helps you understand your data faster.

Z-Scores

The empirical rule counts distance in standard deviations. A z-score gives you that distance for any single value: how far is this observation from the mean, measured in standard deviations.

z = x − μσx is the value, μ is the mean and σ is the standard deviation.

A positive z-score means the value is above the mean, a negative one means it is below, and a z-score of 0 means it is exactly the mean. A z-score between −1 and 1 is inside the 68% range, between −2 and 2 inside the 95% range, and between −3 and 3 inside the 99.7% range.

Z-scores also let you compare values that are measured on different scales. Imagine a score of 85 on a test with a mean of 70 and a standard deviation of 10, and a score of 650 on another test with a mean of 500 and a standard deviation of 100. Both have a z-score of 1.5, so both results are equally far above their group's average.

When Does a Value Become Unusual?

Knowing where a value sits on a normal distribution helps you spot values that differ significantly from the rest of the data. Data professionals often use 3 standard deviations as a screening rule: a z-score above 3 or below −3 flags a value for review.

A 40-foot cherry tree has a z-score of (40 − 15) / 2 = 12.5. That is far outside the 9 to 21 foot range, so it is more likely a typo than a record-breaking tree. Some extreme values come from errors in data collection or processing, and these false values can skew your results. That does not automatically mean the data is wrong. It means the observation deserves investigation.

Z-Scores in Machine Learning

Z-scores are not only a tool for describing data. In machine learning, converting a feature to z-scores is a standard preprocessing step called standardization. Every feature is rescaled to a mean of 0 and a standard deviation of 1, so features measured on very different scales become comparable.

Imagine a model that predicts customer churn from two features: age (roughly 18 to 80) and yearly spend (roughly 0 to 50,000). Without scaling, the spend values are hundreds of times larger than the age values, and many models would treat spend as far more important simply because its numbers are bigger. After standardization, a value of 1.5 means "1.5 standard deviations above average" for both features, and neither one dominates because of its unit.

The first group compares points by distance, penalizes coefficients or learns by gradient descent, so the scale of a feature changes the result. Tree-based models split the data on thresholds within one feature at a time, so scaling does not change their results.

Keep in mind

• Calculate the mean and standard deviation on the training data only. Then apply those same two numbers to the validation and test data. If you use the whole dataset, information from the test set leaks into training and your evaluation looks better than it really is.
• Standardizing does not make data normal. It only shifts and rescales it, and the shape stays the same.
• Outliers can distort the mean and standard deviation. Check strong outliers first. In some cases robust scaling is the better choice.

How to explain it to stakeholders

Stakeholders rarely need the formula. They need to know what the curve means for their decision. Think of a delivery-time example:

Here is how it could sound in a meeting:

Key Takeaway

• CENTER → Mean
• SPREAD → Standard deviation
• DISTANCE → Z-score
• UNUSUAL → Tails
• ML → Standardization