Regression analysis

Evaluating and Interpreting a Regression Model

A regression model can produce a convincing line. But does it actually fit the data well? How certain can we be about its estimates? And what can we conclude from the results?

In The Four Assumptions of Simple Linear Regression, we explored when a fitted line can be trusted. Now, we take the next step: evaluating model fit, understanding uncertainty, and interpreting the results in a business context.

Because a model that fits the data well does not automatically provide reliable predictions or prove cause and effect.

Throughout this post, we use one example: NordHome Retail fitted a simple linear regression on 24 months of data to predict monthly sales from advertising spend, both in thousands of euros (€k). The fitted line is sales = 20 + 5 × advertising spend.

Interpret Uncertainty: How Reliable Are the Estimates?

Statistical software reports a fitted regression in one summary table. Here is the summary for NordHome’s model of monthly sales on advertising spend, in the format of Python’s statsmodels. Four columns matter most: coef shows the estimated relationship, std err shows how precise it is, P>|t| gives the result of the hypothesis test, and the last two columns give the 95% confidence interval. These are the purple columns, and the sections below explain them one by one. The blue R-squared values at the top measure the model’s fit and are explained in Model Evaluation Metrics.

OLS Regression Results

Dep. Variable:	sales_k	R-squared:	0.850
Model:	OLS	Adj. R-squared:	0.843
Method:	Least Squares	F-statistic:	124.7
No. Observations:	24	Prob (F-statistic):	1.57e-10
Df Residuals:	22	AIC:	153.7
Df Model:	1	BIC:	156.1

	coef	std err	t	P>\|t\|	[0.025	0.975]
Intercept	20.000	2.930	6.827	0.000	13.924	26.076
advertising_spend_k	5.000	0.448	11.165	0.000	4.071	5.929

Summary Regression summary for NordHome, in the format of Python’s statsmodels. Sales and advertising spend are both in €k. Illustrative data.

Coefficient: How much do sales change?

NordHome’s estimated slope is 5.0. Since both advertising spend and sales are measured in thousands of euros (€k), this means that every additional €1,000 in advertising spend is associated with an estimated €5,000 increase in monthly sales.

The intercept is 20, meaning the model predicts €20,000 in sales when advertising spend is zero. This interpretation is useful only if zero advertising spend is meaningful in the business context.

These are estimates of a relationship, not proof that advertising causes sales to increase.

Standard Error: How precise is the estimate?

The slope is estimated at 5.0, with a standard error of 0.448. Think of the standard error as an indicator of how much the estimated slope would tend to vary across comparable samples.

Here, the standard error is relatively small compared with the estimated slope. This suggests that the relationship has been estimated with reasonable precision, although the standard error alone does not tell us whether the model is appropriate or whether the relationship is causal.

P-value: Is there evidence of a relationship?

Regression asks a yes-or-no question about the slope: is there really a relationship between advertising and sales? To answer it, we set up two competing hypotheses.

H₀ (null hypothesis): the true slope is 0. Advertising has no relationship with sales.
H₁ (alternative hypothesis): the true slope is not 0. Advertising and sales are related.

We start by assuming H₀ is true. The p-value then tells us how likely it is to see a slope as far from 0 as ours (5.0), or further, just by chance.

In the summary table, the p-value (P>|t|) is 0.000, meaning below 0.001. If advertising really had no relationship with sales, a result like this would almost never happen by chance. So we reject H₀ and conclude that the relationship in NordHome’s data is very unlikely to be a coincidence (read Hypothesis Testing for more).

The p-value comes from a t-test on the slope, which is why the summary table also shows a t value.

Confidence Interval: How large might the slope be?

The 95% confidence interval for the slope is approximately [4.071, 5.929].

In business terms, the estimated increase in monthly sales is between approximately €4,071 and €5,929 for each additional €1,000 in advertising spend.

The interval excludes zero, which is consistent with the statistically significant result. It also communicates the uncertainty around the estimated slope: the point estimate is 5.0, but the true population slope is not known exactly.

A 95% confidence interval is produced by a method that would contain the true slope in approximately 95% of repeated samples under the model assumptions. It does not mean there is a 95% probability that the true slope lies within this particular interval.

Visualizing Uncertainty

A regression line gives us an estimate, but how much confidence should we place in it? And how much might actual sales vary from month to month?

A 95% confidence band and a 95% prediction interval answer two different questions.

The shaded band: How certain are we about average sales? Suppose NordHome spends €8,000 on advertising. The regression line estimates the average sales level for months with this level of spend. The shaded band shows the uncertainty around that estimate. It is generally narrowest near the average advertising spend and widens towards the extremes, where the estimate is less precise.
The dashed lines: How much might sales vary in one month? Even if the model estimates average sales of €60,000, actual sales in a particular month could be higher or lower. The prediction interval accounts for this month-to-month variation as well as uncertainty in the estimated average. That is why it is wider than the confidence band.

Scatter plot of advertising spend and sales with a regression line, a shaded 95 percent confidence band and wider dashed prediction interval lines. — Chart 1 The blue shaded band shows uncertainty around the estimated average sales at each advertising spend level. The dashed lines show a wider prediction interval for an individual month’s sales. Both intervals are 95% intervals. Illustrative data.

In short: the confidence band describes uncertainty about the average relationship; the prediction interval describes uncertainty about an individual month’s sales.

Model Evaluation Metrics: How Well Does the Model Fit?

A regression model estimates a relationship between a predictor and an outcome. But how closely do its predictions match the observed data?

R² — Coefficient of Determination

R² measures how much of the variation in observed sales the fitted model accounts for.

For NordHome, an R² of 0.85 means the model accounts for approximately 85% of the variation in monthly sales across the 24 months used to fit it. The remaining variation is not accounted for by the model.

Two scatter plots of sales against advertising spend: the left shows errors around the average line, the right shows much smaller errors around the fitted line, with R squared of 0.85. — Chart 2 Left: prediction errors when we always predict average sales. Right: prediction errors around the fitted regression line. R² measures the reduction in the sum of squared errors compared with the mean-only baseline. Illustrative data.

The regression model reduces the sum of squared errors by approximately 85% compared with always predicting average sales.

R² = 1 − Σ(yᵢ − ŷᵢ)²Σ(yᵢ − ȳ)²yᵢ is observed sales, ŷᵢ is predicted sales, and ȳ is average sales. R² compares the model’s squared prediction errors with those of a baseline that always predicts average sales.

A high R² does not guarantee accurate predictions on new data or prove that advertising causes sales to increase.

MAE and MSE

MAE — Mean Absolute Error. MAE measures the average absolute difference between predicted and actual sales.

For NordHome, an MAE of €4k means the model’s predictions differ from actual monthly sales by €4,000 on average in absolute terms. Because MAE uses the original sales units, it is relatively easy to interpret.

MAE = Σ|yᵢ − ŷᵢ|nn is the number of months, yᵢ is observed sales, and ŷᵢ is predicted sales. MAE averages the absolute prediction errors.

MSE — Mean Squared Error. MSE measures the average squared prediction error. Squaring the errors gives larger mistakes disproportionately more influence.

For NordHome, an illustrative MSE of 30 (€k)² means the average squared prediction error is 30 in squared-thousand-euro units. A month with a €15k prediction error contributes 225 to the sum of squared errors, compared with just 16 for a €4k error.

MSE = Σ(yᵢ − ŷᵢ)²nMSE averages the squared prediction errors, so its unit is the square of the sales unit, here (€k)².

MSE is useful when large prediction errors are particularly costly, although its squared units make it harder to interpret directly.

Two bar charts of 24 months of prediction errors: absolute errors with an average of 4 thousand euros, and squared errors with an average of 30, where one large miss dominates. — Chart 3 NordHome’s 24 monthly prediction errors, measured in two ways. The €15k error contributes 15 to the MAE sum but 225 to the MSE sum because the error is squared. Illustrative data.

Adjusted R²: Does Adding More Predictors Improve the Model?

Imagine NordHome Retail wants to predict monthly sales. It starts with a simple regression model using advertising spend as its only predictor. The model has an R² of 0.85, meaning it accounts for 85% of the variation in historical monthly sales.

But what happens if the analyst adds six more predictors, such as website traffic, discount rate and store footfall? The model’s R² increases to 0.87. At first glance, this looks like an improvement. However, adding predictors can help a model fit historical data even when some of those variables contribute little useful information. They may capture random patterns that will not hold in new data.

This is where adjusted R² becomes useful. It is used in multiple linear regression, where a model has more than one predictor. Unlike ordinary R², it accounts for the number of predictors in the model. It increases when the improvement in fit is sufficient to justify the additional complexity, and decreases when it is not.

Adjusted R² = 1 − (1 − R²)(n − 1)n − k − 1n is the number of observations and k is the number of predictors. The more predictors a model has, the smaller the denominator becomes and the larger the penalty. When you compare models with different numbers of predictors, adjusted R² is a fairer yardstick than R². It does not replace evaluating predictions on unseen data.

Consider NordHome’s two models:

Metric	Model A	Model B
Predictors	1	7
R²	0.85	0.87
Adjusted R²	0.843	0.810

Both models use the same 24 months of data. Although Model B has a higher R², its adjusted R² is lower. The additional predictors improve the fit, but not enough to compensate for the added complexity.

Line chart of R squared rising and adjusted R squared falling as unhelpful predictors are added. — Chart 4 Adding predictors to an ordinary least-squares model with an intercept cannot decrease R² on the same dataset. Adjusted R² accounts for the number of predictors and can decrease when an additional predictor does not improve the fit enough. Illustrative data.

Adjusted R² does not prove that the extra predictors are useless. It signals that the improvement in fit may not justify including them.

Do not rely on R² alone

A high R² does not guarantee accurate predictions on new data, confirm that the model assumptions hold, or prove that advertising causes sales.

R² measures explained variation, while MAE and MSE measure prediction errors. AIC and BIC can help compare candidate models fitted to the same response data; lower values are generally preferred.

Avoid Misinterpretation: Does Advertising Cause Sales?

For NordHome, an R² of 0.85 corresponds to a positive correlation of approximately 0.92: months with higher advertising spend tend to have higher sales.

Does this mean that advertising causes the increase? Not necessarily. Seasonal demand, for example, may raise both advertising spend and sales, so part of the increase may reflect the season rather than advertising.

To test whether advertising actually drives sales, NordHome would need a randomized controlled experiment. Its key parts are:

Treatment group. The group that receives the change. For NordHome, stores with increased advertising spend.
Control group. The group that does not receive the change. Stores with unchanged advertising spend.
Random assignment. Chance decides which store goes into which group, so the groups are comparable.
Control of other factors. Seasonality, pricing and promotions must be the same for both groups.
Measurable difference. The difference in sales between the groups can be observed and measured.

Setting up such an experiment is laborious and often impractical in a real business. This is why regression results are usually reported as associations (“is associated with”), not as proof that one thing causes the other.

What Can Go Wrong?

Patterns in the residuals. A curve may indicate that a straight line misses part of the relationship. A funnel shape may indicate that prediction errors become more variable at higher sales or advertising levels. See The Four Assumptions of Simple Linear Regression.
Influential months. One unusual month can substantially change the fitted slope, especially if its advertising spend is far from the others.
Extrapolation. NordHome’s model was fitted using advertising spend between €5k and €20k. Predictions at €100k go far beyond the observed range and may be unreliable.
Association versus causation. The model shows that advertising spend and sales move together. It does not establish that higher advertising spend causes sales to increase.

A good fit does not guarantee reliable predictions, so check residuals, investigate influential observations and validate the model on data it has not seen.

How to explain it to stakeholders

“In our historical data, every additional €1,000 in advertising was associated with about €5,000 more in sales. The estimate is reasonably precise: the plausible range is €4,100 to €5,900 per extra €1,000. The model accounts for 85% of the variation in monthly sales. That does not prove advertising caused the increase, and we should validate the model before using it for forecasts.”

Key takeaway

Evaluating a regression model requires more than checking whether the line fits the data.

Model fit tells us how well the model describes historical observations. Statistical uncertainty tells us how precisely the relationship has been estimated. Business interpretation helps us understand what the evidence supports — and where caution is needed.

A useful regression analysis combines fit metrics, uncertainty estimates and an understanding of the model’s limitations. A high R² alone is never enough.