Regression analysis

Overfitting and Underfitting

Imagine you want to predict net revenue per customer. One model barely improves on a simple guess. Another fits its training data almost perfectly—but performs poorly on new customers. Both models are unreliable, but for opposite reasons.

In Evaluating and Interpreting a Regression Model, we looked at how well a model fits the data it was given. Now we take the next step: will it still work on customers it has never seen?

This is the problem of generalization. A model that is too simple may miss important patterns (underfitting). A model that learns the training data too closely may fail to generalize (overfitting). Throughout this post, we use customer revenue as an illustrative example. All data is illustrative.

Training Data and Test Data

Before building a machine learning model, data scientists typically split their dataset into training and test data. This is a standard step in the machine learning workflow: the model needs data to learn from, but we also need separate data to check whether it can make reliable predictions on new observations.

We divide the data into two parts:

• Training data is used to teach the model. It contains the predictors and the actual outcomes the model learns from.
• Test data is held back during training. Once the model is built, we use it to evaluate how well it predicts outcomes it has not seen before.

A practical example

Imagine we have data from 100 customers and want to predict their net revenue. We randomly assign 80 customers to the training set and reserve the other 20 for the test set.

The model learns from the 80 training customers. It then predicts revenue for the 20 test customers without having used their actual revenue values during training. We compare those predictions with the actual values to see how well the model performs on unseen data.

This matters because a model can perform very well on its training data but poorly on new customers. Comparing its performance on both datasets helps us identify potential overfitting and assess whether the model has learned patterns that generalize beyond the data it was trained on.

Splitting the Data with Python

from sklearn.model_selection import train_test_split

# Separate predictors from the target
X = df.drop(columns="net_revenue")
y = df["net_revenue"]

# Split the data: 80% training, 20% testing
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.2,
    random_state=42
)

print("Training customers:", len(X_train))
print("Test customers:", len(X_test))

Training customers: 80
Test customers: 20

What the code does:

• X contains the predictors used to estimate revenue.
• y contains the actual revenue values we want to predict.
• test_size=0.2 reserves 20% of the rows for testing.
• random_state=42 makes the random split reproducible, so you get the same split when you run the code again on the same data.

The result is 80 training observations and 20 test observations, assuming the dataset contains 100 rows. The code assumes df is your prepared pandas DataFrame and that net_revenue is your target column.

Comparing performance on both datasets shows why we need both:

This is the central lesson: training performance alone cannot tell us whether a model will generalize well.

Two Ways a Model Can Fail

Underfitting

An underfitting model is too simple to capture the underlying pattern in the data. It performs poorly even on the data it was trained on, often resulting in a low R².

Imagine predicting net revenue per customer using only the weekday of their first order. Monday and Saturday customers may spend very different amounts, but the weekday alone might explain very little of that variation. The model ends up predicting roughly the same revenue for everyone.

In this example, the model has not learned enough about what drives customer spending. Its R² is 0.03, meaning it explains only 3% of the variation in revenue in the data used to fit the model.

Typical causes of underfitting:

• Model too simple. A straight line misses a curved relationship between customer characteristics and revenue.
• Missing useful predictors. The model ignores important factors, such as order frequency or average basket size.
• Too much regularization. The model is restricted so strongly that it cannot learn the underlying relationship.

Because the model fails to capture the pattern in its training data, it is also unlikely to make reliable predictions for new customers.

Overfitting

Overfitting is the opposite problem. An overfitting model learns the training data too closely. It performs very well on the data it was trained on, but its predictions become less reliable when it encounters new data.

Imagine predicting net revenue per customer using 40 predictors but having data from only 60 customers. The model has many variables to work with relative to the number of observations. It may start treating random quirks as meaningful patterns. For example, one customer who happened to make a large purchase during a flash sale might disproportionately influence what the model learns. That pattern may not hold for other customers.

To understand why this happens, we need to distinguish between two things in the data:

• Signal: The underlying relationships that help explain customer revenue.
• Noise: Random variation that does not reflect a reliable, generalizable relationship.

An overfitting model learns both the signal and some of the noise. As a result, it fits its training data closely but struggles to generalize to the wider population.

How to Spot Underfitting and Overfitting

Comparing the error on the training data with the error on the test data tells the three situations apart.

Chart 4 shows the same 24 customers fitted three ways: a straight line (degree 1), a smooth curve (degree 2) and a very flexible curve (degree 12).

The errors behind these three fits, measured on the 24 training customers and on 200 new customers, tell the same story in numbers:

Underfitting shows up as a high error on both sets. Overfitting shows up as a low training error but a much higher test error. Errors are RMSE (root mean squared error) in euros per customer.

Adding predictors can reduce training error even when the new variables contribute little useful information. For predictive modeling, cross-validation helps compare models on data not used to fit them. Adjusted R² is another useful measure when comparing linear regression models, but it does not replace evaluating predictive performance on unseen data. See Evaluating and Interpreting a Regression Model for adjusted R².

Bias and Variance

Underfitting and overfitting describe what goes wrong with a model. Bias and variance help explain why.

• Bias describes how much a model’s assumptions prevent it from capturing the real relationship in the data. A model with high bias is often too simple and consistently misses important patterns. This is commonly associated with underfitting.
• Variance describes how sensitive a model is to the particular training data it receives. A model with high variance may learn patterns that are specific to one sample rather than patterns that hold more generally. This is commonly associated with overfitting.

The Bias–Variance Trade-off

A very simple model may miss important relationships, even when trained on different samples. A very complex model may fit one training sample extremely well but produce very different results when trained on another.

The goal is to find a balance: a model complex enough to capture meaningful patterns, but not so complex that it starts learning random noise.

The chart below shows how the two errors trade off as the model becomes more complex.

Strategies to Reduce Underfitting and Overfitting

If the model underfits

• Choose a more flexible model. A straight-line model may be unable to capture a curved relationship.
• Add relevant predictors. Include variables that contain useful information about the outcome.
• Improve feature engineering. Transform existing variables or create new features that better represent the underlying relationships.

If the model overfits

• Reduce model complexity. Remove unnecessary predictors or choose a simpler model.
• Use regularization. Penalize excessive model complexity to reduce sensitivity to the training data.
• Use cross-validation. Compare candidate models on different training/validation splits to find a model that generalizes well.

Several of these strategies come down to the same question: which predictors belong in the model? Too few useful predictors can leave important patterns unexplained, while unnecessary predictors can increase overfitting risk. The right choice depends on the data, model complexity, and how the model is evaluated. Deciding which ones to keep is called variable selection, and it is covered in Multiple Linear Regression.

How do we know if the changes work?

Evaluate the selected model on a separate test set that was not used for model selection. This provides an estimate of how well the model predicts unseen observations.

The goal is not to build the most complicated model or the model with the smallest training error. It is to find a model that captures meaningful patterns and performs reliably on new data.

How to explain it to stakeholders

Related posts

• Evaluating and Interpreting a Regression Model How R², MAE and MSE measure fit, and how confidence intervals describe uncertainty.
• Simple Linear Regression The fitted line and what its slope and intercept mean.
• The Four Assumptions of Simple Linear Regression When a fitted line can be trusted.