Regression analysis
The Four Assumptions of Simple Linear Regression
A regression model can find the best-fit line, but how do we know whether we can trust its results? The four assumptions of simple linear regression help us answer this question. They allow us to check whether a linear model is appropriate for our data and whether its statistical conclusions are reliable. In this article, we will explore why each assumption matters, what can go wrong when it is violated, and how to check it using visualizations and Python.
The Four Assumptions at a Glance
The four assumptions of simple linear regression help us identify potential problems before we interpret the results:
- Linearity: Is a straight line an appropriate way to describe the relationship?
- Independence: Can we treat the observations as independent?
- Normality: Are the errors approximately normally distributed, as required for conventional statistical inference?
- Homoscedasticity: Does the error variance remain approximately constant?
1. Linearity: Is a Straight Line Appropriate?
Simple linear regression assumes that the relationship between the independent variable (X, advertising spend) and the dependent variable (Y, sales) can be reasonably represented by a straight line.
As advertising spend increases, sales might also increase. But how do they increase? If sales tend to rise at a roughly constant rate, a straight line may describe the relationship well. If sales rise rapidly at first and then level off, the relationship is curved instead.
Why does it matter? If the actual relationship is curved but we fit a straight line, the model may systematically overestimate or underestimate values in different parts of the data. Its predictions and interpretation may therefore be misleading.
How can we check it? Start with a scatterplot, with X on the horizontal axis and Y on the vertical axis. Look for an approximately linear pattern rather than a clear curve. The points do not need to lie exactly on a line; what matters is that the average of Y changes in a straight-line way as X increases. Chart 1 shows the difference.
This check should happen before building the regression model. In Python, it takes a few lines:
import seaborn as sns
import matplotlib.pyplot as plt
sns.scatterplot(data=df, x="advertising_spend", y="sales")
plt.show()When the data is very scattered, a curve can be hard to see in the raw scatterplot. A second view helps: after fitting the line, plot the residuals against the fitted values. If the relationship is linear, the residuals scatter randomly around zero. If it is curved, they form a systematic arch, because the line is too low in some regions and too high in others.
Points above the red zero line mean the model predicted too little, and points below it mean it predicted too much. A random scatter means the model is not missing anything systematic. A curved shape means part of the relationship is missing from the straight-line model.
If a clear curve appears, possible remedies are to transform a variable, for example by taking its logarithm, or to add a squared term, and then check the plots again.
With several predictors
A single scatterplot is not enough when the model has more than one X. A scatterplot matrix plots every variable against every other, so you can check linearity for each predictor at once and also see whether predictors overlap. We cover this in Multiple Linear Regression.
2. Independence: Are the Observations Independent?
The independence assumption means that the error for one observation should not depend on the error for another. After accounting for the relationship described by our regression model, the remaining errors should not show a systematic connection.
Why does it matter? When errors are correlated, the model may underestimate uncertainty. Standard errors can become too small and confidence intervals too narrow, making a relationship appear more statistically convincing than it really is.
How can we check it? Start by understanding how the data were collected. For monthly observations, plotting residuals over time can reveal patterns that suggest autocorrelation.
Consider NordHome's monthly advertising spend and sales. A major campaign in March might continue to influence sales in April. If the model underestimates sales in March, it might also underestimate sales in April because both months are affected by the same underlying factor. Their errors may therefore be related.
Visualizing Independence
To investigate this, compare each month's residual with the previous month's residual. The horizontal axis shows the previous month's residual, while the vertical axis shows the current month's residual.
When errors are independent, the points should form a scattered cloud without a clear pattern. When positive residuals tend to follow positive residuals, and negative residuals tend to follow negative residuals, the points may form an upward pattern. This suggests positive autocorrelation.
How to Read the Two Plots
Left: Independent errors
The dots form a scattered cloud with no clear direction, and the dashed line is flat.
- A high residual can be followed by a high or a low one.
- A low residual can be followed by a high or a low one.
- Knowing last month's residual tells us almost nothing about this month's.
This is consistent with independence.
Right: Dependent errors
The dots rise from bottom left to top right, and the dashed line slopes upward.
- A positive residual tends to be followed by a positive one.
- A negative residual tends to be followed by a negative one.
- Knowing last month's residual gives us a useful hint about this month's.
This suggests positive autocorrelation.
The pattern is a warning sign rather than definitive proof of dependence. A formal test, such as the Durbin–Watson test, can help assess autocorrelation in regression residuals.
What is autocorrelation?
Autocorrelation means that a series is correlated with its own past values. Here, a month's residual is related to the residual of the month before. If the model underestimates March, it tends to underestimate April too. Positive autocorrelation, where high follows high, is the common case. It matters because the months partly repeat each other, so the model has less real information than it thinks.
3. Normality: Are the Errors Normally Distributed?
The normality assumption concerns the errors in the regression model. To understand it, we first need to distinguish between errors and residuals.
- Errors are the differences between the true underlying relationship and the observed values that the model aims to describe. They represent the model's natural, unobserved noise.
- Residuals are the differences between the observed values and the values predicted by the fitted regression model, like the −3, +2, +4, −2 and −1 in OLS: How Does Linear Regression Find the Best-Fit Line?.
We cannot directly observe the true errors, so we use residuals to assess whether the normality assumption is reasonable.
Why does it matter? Normality is particularly important when we want to make statistical inferences, such as constructing confidence intervals or conducting hypothesis tests under the usual regression assumptions. A regression model can still calculate a line when residuals are not normally distributed. However, the usual inferential procedures may not be reliable, particularly in smaller samples.
How can we check it? Two common visual tools are a histogram and a quantile-quantile plot (Q-Q plot).
- A histogram shows the distribution of the residuals. We look for a roughly bell-shaped pattern centered around zero.
Errors are balanced and typical. Normality looks fine.
Too many extreme errors, both too high and too low.
A few months where actual sales are far above the prediction. The model underpredicts badly.
- A Q-Q plot compares the observed residual quantiles with the quantiles expected under a normal distribution. If the residuals are approximately normally distributed, the points should follow a roughly straight line.
Errors behave as a normal distribution would. Normality looks fine.
Too many extreme errors, both too high and too low.
A few months where actual sales are far above the prediction. The model underpredicts badly.
The Q-Q plot is often particularly useful because deviations from a straight line can reveal departures from normality. With only five months, neither plot tells us much, so in practice you would run these checks on a longer history. In Python, model.resid retrieves the residuals from a fitted regression model:
import statsmodels.api as sm
import statsmodels.formula.api as smf
import matplotlib.pyplot as plt
model = smf.ols("sales ~ advertising_spend", data=df).fit()
residuals = model.resid
sm.qqplot(residuals, line="s")
plt.show()If the residuals deviate substantially from the expected pattern, investigate the distribution and consider whether a transformation, such as taking the logarithm of a right-skewed outcome variable, is appropriate. Then refit the model and check again.
4. Homoscedasticity: Is the Error Spread Consistent?
Homoscedasticity means that the errors have approximately the same variance across the range of predicted values. In simpler terms, the model's prediction errors should have a similar spread, whether predicted sales are low or high.
Imagine NordHome's model predicts monthly sales. In some months, actual sales will be above the prediction; in others, they will be below it. That is normal. But if the errors are small when predicted sales are low and become much larger when predicted sales are high, the error variance may not be constant. This is called heteroscedasticity.
Why does it matter? When error variance changes across predictions, the usual standard errors may be unreliable if we ignore the problem. As a result, confidence intervals and hypothesis tests may be misleading, even though the regression line can still be fitted.
How can we check it? Plot the residuals against the fitted (predicted) values. The horizontal axis shows predicted sales, and the vertical axis shows residuals—the differences between actual and predicted sales.
Ideally, the points form a random cloud around zero with a similar vertical spread across the plot. If the cloud widens or narrows as predicted sales increase, creating a funnel-like shape, this may indicate heteroscedasticity.
How to Read the Two Plots
Left: Constant variance
The points form an even band around the zero line.
- The spread is similar for low and high predicted sales.
- The model's errors are about equally large in every month.
- Standard errors and confidence intervals can be trusted more.
This is consistent with homoscedasticity.
Right: Changing variance
The points fan out as predicted sales increase, forming a funnel.
- Errors are small when predicted sales are low.
- Errors grow much larger when predicted sales are high.
- Standard errors and confidence intervals may be misleading.
This suggests heteroscedasticity.
The plot takes a few lines, using the model fitted above:
import seaborn as sns
import matplotlib.pyplot as plt
fitted_values = model.fittedvalues
residuals = model.resid
sns.scatterplot(x=fitted_values, y=residuals)
plt.axhline(0)
plt.xlabel("Fitted Values")
plt.ylabel("Residuals")
plt.show()If the assumption is violated, possible approaches include transforming the outcome variable or using statistical methods that account for non-constant variance.
What if an assumption is violated?
A violation does not automatically mean that the entire analysis is useless. It means that we need to understand the problem before deciding how to proceed. Depending on the situation, possible next steps include:
- Non-linearity. Transform a variable or consider a different model form.
- Non-normal residuals. Investigate unusual observations or skewness and consider an appropriate transformation.
- Dependent observations. Review the sampling process or use a model that accounts for dependence.
- Non-constant variance. Consider transforming the outcome or using methods that account for heteroscedasticity.
There is no single solution that works for every dataset. Transformations also change how we interpret the variables and coefficients, so we should always reassess the model after making changes. If the assumptions remain seriously violated, a different modeling approach may be more appropriate.
Key takeaway
Fitting a line is only the first step: the four assumptions (linearity, independent observations, normal errors and constant variance) tell us whether the model's results can be interpreted with confidence. Check linearity and independence before fitting, and use residual plots for the other two afterward. The goal is not to make the data perfect, but to understand the limitations of the model and decide how to use it.
Related posts
- OLS: How Does Linear Regression Find the Best-Fit Line? How the best-fit line, residuals and SSR work.
- Evaluating and Interpreting a Regression Model How to judge whether a model is suitable for drawing conclusions.
- Exploratory Data Analysis Look at your data before you model it.