Regression analysis

OLS: How Does Linear Regression Find the Best-Fit Line?

Imagine placing a ruler across a scatterplot of data points. You can tilt or move the ruler to create many different lines, and several might look like a good fit. But how do we decide which line fits the data best?

Linear regression needs a way to measure how well each line fits. For every data point, we compare the actual value with the value predicted by the line. These differences are called residuals. Ordinary Least Squares (OLS) squares each residual and adds them together. The best-fit line is the one with the smallest sum of squared residuals.

In this post, we’ll explore how this works using the five months of NordHome data introduced in Simple Linear Regression.

Error Measurement and Residuals

A regression line rarely passes through every data point. The vertical distance between an observed value and the value predicted by the line is called a residual.

eᵢ = yᵢ − ŷᵢ Here, yᵢ is the actual value and ŷᵢ is the predicted value.

• A positive residual means the actual value is above the line.
• A negative residual means the actual value is below the line.
• A zero residual means the prediction matches the actual value exactly.

For NordHome, consider the month with €7k in advertising spend. The regression line predicts sales of €55k, while actual sales are €53k. The residual is therefore:

eᵢ = 53 − 55 = −2 The model overestimated sales by €2k.

Why Can't We Simply Add the Residuals?

In an ordinary least squares regression with an intercept, the residuals sum to zero. For NordHome, the five residuals are −3, +2, +4, −2 and −1 (Chart 1):

−3 + 2 + 4 − 2 − 1 = 0

Positive and negative errors cancel each other out. As a result, simply adding the residuals does not tell us how well the line fits the data.

The Solution: Square the Residuals

OLS squares every residual before adding them together. Squaring removes negative signs and gives larger errors more weight. The resulting measure is called the sum of squared residuals (SSR):

SSR = Σᵢ₌₁ⁿ (yᵢ − ŷᵢ)²

For NordHome's fitted line:

SSR = 9 + 4 + 16 + 4 + 1 = 34 A smaller SSR indicates a closer fit for the same dataset. OLS finds the intercept and slope that minimize this sum.

Ordinary Least Squares (OLS) Estimation

A straight regression line is defined by two coefficients:

• Intercept (β̂₀): the predicted value of y when x = 0.
• Slope (β̂₁): the expected change in predicted y for a one-unit increase in x.

The fitted regression equation is:

ŷ = β̂₀ + β̂₁x

The hats indicate that these are estimates calculated from the observed data, rather than the unknown true population parameters.

OLS chooses the intercept and slope that produce the smallest possible SSR. This is how it identifies the best-fit line for the given data.

Practical example: trying different lines

Before using a formula, test it by hand. Here are three lines for NordHome's data:

The first line sits below every point, so all five residuals are positive and the SSR is large. The second line's residuals nearly cancel out, with a sum of only +1, yet its SSR is still 57. That is exactly why we square the residuals before adding them. The OLS line balances the errors and has the smallest SSR of any straight line: 34. You could keep adjusting the slope and the intercept and watch the SSR change, but you would never get below 34. OLS does that search for you, with a formula instead of trial and error.

Estimating the Coefficients

OLS finds the line with the smallest sum of squared residuals. But how does it determine the best slope and intercept?

Rather than testing countless possible lines, OLS uses mathematical formulas to calculate both coefficients directly from the observed data.

Step 1: Calculate the slope

The slope tells us how much the model's predicted sales change when advertising spend increases by one unit. OLS calculates it using the deviations of advertising spend and sales from their respective means.

β̂₁ = Σ(xᵢ − x̄)(yᵢ − ȳ)Σ(xᵢ − x̄)² The numerator measures how x and y vary together. The denominator measures how much x varies around its mean.

For NordHome, the calculation gives:

β̂₁ = 11422.8 = 5

This means that for every additional €1k spent on advertising, the model predicts an increase of €5k in sales. This is the relationship estimated from our five-month sample, not necessarily a causal effect.

Step 2: Calculate the intercept

Once we know the slope, we can calculate the intercept:

β̂₀ = ȳ − β̂₁x̄ The intercept ensures that the fitted line passes through the point defined by the mean advertising spend and mean sales.

For NordHome, the mean advertising spend is €5.2k and the mean sales are €46k:

β̂₀ = 46 − (5 × 5.2) = 20

The estimated intercept is therefore 20, meaning the line predicts sales of €20k when advertising spend is zero. Since zero is outside the observed advertising-spend range in this example, that interpretation should be treated with caution.

The resulting regression equation

Combining the slope and intercept gives us the fitted line:

ŷ = 20 + 5x

This is the line used in our NordHome charts. OLS calculates these coefficients to minimize the sum of squared residuals for the observed data.

In practice, statistical software performs these calculations almost instantly. Understanding the formulas helps us interpret the coefficients and understand how the model arrives at its predictions.

A quick note on correlation

Once we have the fitted line, it helps to ask how closely the two variables move together. The correlation coefficient (r) measures the strength and direction of their linear relationship, ranging from −1 to +1.

• Positive correlation: the variables tend to increase together.
• Negative correlation: one tends to decrease as the other increases.
• Zero correlation: there is no linear relationship.

For NordHome, r ≈ 0.97 indicates a strong positive linear relationship between advertising spend and sales in our five observations. However, correlation alone does not tell us the exact slope or prove that advertising causes sales to increase.

What Comes Next?

We’ve seen how ordinary least squares finds the best-fitting line by minimising the sum of squared residuals. But a line that fits our data best is not necessarily a model we can fully trust.

How do we know whether the relationship is reliable? And how can we tell whether the model captures a meaningful pattern rather than just fitting a small sample?

Next, we’ll explore The Four Assumptions of Simple Linear Regression and Evaluating and Interpreting a Regression Model.

Related posts

• Simple Linear Regression What the line predicts and what its slope and intercept mean.
• The Four Assumptions of Simple Linear Regression When you can trust a fitted line, and how to check it.
• Evaluating and Interpreting a Regression Model How to judge whether a model is suitable for drawing conclusions.
• Generalized Linear Models How regression extends to yes/no and count outcomes.
• Descriptive Statistics Means and spread, the building blocks of the OLS formulas.
• Exploratory Data Analysis How to look at relationships in retail data before you model them.