Mitigating overfitting, repeated data use, and model bias

Introduction: Building Models That Actually Work

Hello there! Welcome to one of the most critical chapters in your Exam ATPA journey. Up until now, you’ve likely learned how to build complex models that can fit data with incredible precision. But here is the secret: anybody can build a model that fits old data perfectly. The real skill is building a model that predicts the future accurately.

In this chapter, we are going to learn how to stop our models from "memorizing" the past (overfitting), how to avoid the pitfalls of repeated data use and data leakage, and how to ensure our models are fair, compliant, and free from discriminatory impact (mitigating model bias and proxy discrimination). Think of this as the "Quality Control" department of predictive analytics. Let's dive in!

1. Overfitting: The "Memorization" Trap

What is Overfitting?
Imagine you are studying for a math exam. Instead of learning the formulas, you simply memorize the answers to the 50 practice questions provided. When you take the actual exam, the questions are slightly different. You fail because you didn't learn the logic; you only learned the specific examples. That is overfitting.

In predictive modeling, overfitting happens when a model is so complex that it starts to treat "noise" (random fluctuations) as if it were a real pattern.

Starting with Too Many Variables

A primary driver of overfitting is starting with an excessive number of predictor variables relative to the sample size. When a model includes dozens or hundreds of predictors, it easily picks up spurious correlations that exist purely by chance in the training sample. This inflates model variance and degrades performance on new data. Actuaries must actively prune the feature space through domain expertise, screening, and dimension reduction.

The Bias-Variance Tradeoff

To understand overfitting, we have to look at two competing forces:
• Bias: This is the error from overly simple assumptions. If you try to fit a straight line to data that clearly curves, you have High Bias (Underfitting).
• Variance: This is the error from being too sensitive to small fluctuations. If your model changes drastically every time you add one new data point, you have High Variance (Overfitting).

Quick Review: We want the "Goldilocks" zone—a model that is complex enough to capture the signal but simple enough to ignore the noise.

How to Detect Overfitting

The easiest way to spot overfitting is to compare your model’s performance on two sets of data:
• Training Set: The data the model learns from.
• Test Set: The "unseen" data used to grade the model.
The Red Flag: If your training error is very low but your test error is substantially higher, the model has overfitted.

2. Mitigating Overfitting: Regularization

If our model is too complex or starts with too many variables, we need to constrain it. This is called Regularization. It works by adding a penalty to the objective function for having coefficients that are too large or too numerous.

Ridge Regression (L2 Regularization)

Ridge regression adds a penalty term proportional to the sum of squared coefficients:
\( \text{Penalty} = \lambda \sum_{j=1}^{p} \beta_j^2 \)
• How it works: It shrinks the coefficients toward zero, reducing their variance, but it never sets them exactly to zero. It retains all predictors while curbing excessive sensitivity to collinear or noisy features.

Lasso Regression (L1 Regularization)

Lasso (Least Absolute Shrinkage and Selection Operator) adds a penalty proportional to the sum of absolute values of the coefficients:
\( \text{Penalty} = \lambda \sum_{j=1}^{p} |\beta_j| \)
• The Magic of Lasso: Because of the geometric shape of the L1 constraint, Lasso forces some coefficients to become exactly zero when \( \lambda \) is sufficiently large. This means Lasso automatically performs variable selection, mitigating the danger of starting with too many variables!

Memory Aid: Lasso = Less variables (it cuts some out). Ridge = Reduces coefficients (but keeps them all).

Key Takeaway

Regularization uses a tuning parameter \( \lambda \). When \( \lambda = 0 \), you have an unpenalized standard model. As \( \lambda \) increases, the penalty grows stronger, shrinking coefficients and simplifying the model (reducing variance at the expense of a slight increase in bias).

3. Repeated Data Use and Data Leakage

The Danger of Repeated Data Use

Repeated data use occurs when an analyst uses the same validation or test data repeatedly to make modeling decisions—such as screening predictors, tuning hyperparameters, choosing functional forms, and comparing candidate algorithms. While the model may not train directly on the holdout observations, the analyst implicitly incorporates information from the holdout set into the modeling workflow.

• Holdout Overfitting: If you try 100 different model configurations and choose the one with the lowest validation error, part of that top score is simply luck on that specific holdout slice. The validation error becomes an overly optimistic, biased estimate of true generalization error.
• Mitigation: To mitigate repeated data use, implement a disciplined data-splitting protocol:
1. Three-Way Split: Partition the data into Training (to fit parameters), Validation (to compare candidate models and tune hyperparameters), and Test (to provide an unbiased final audit).
2. Lock the Test Set: Keep the test set locked in a "vault." Evaluate it only once at the conclusion of the project. If a model fails on the test set, you cannot tweak the model on that test set without corrupting its independence.
3. Nested Cross-Validation: When data is limited, use nested cross-validation, where an inner loop tunes hyperparameters and an outer loop estimates generalization performance without data reuse bias.

Data Leakage

Data Leakage occurs when information from outside the training environment is inadvertently introduced into model fitting, making the model look unrealistically accurate during development.

• Target Leakage: Including a predictor that contains information about the target that would not be available at the actual moment of prediction. Example: Using "Claim Approval Date" to predict whether an insurance claim is fraudulent. Approval happens only after the investigation concludes.
• Train-Test Contamination: Calculating preprocessing parameters (such as imputation medians, standard scaling factors, or target encoding) on the full dataset before splitting. The training set is contaminated with summary statistics from the test set.
• The Rule of Prevention: Always partition your data first. Compute all transformations, scalers, and imputations strictly on the training partition, then apply those fixed transformations to the validation and test sets.

4. Model Bias, Fairness, and Proxy Discrimination

Actuaries must ensure that predictive models behave ethically, comply with legal and professional standards, and do not unfairly disadvantage protected demographic groups.

Understanding Fairness Concepts

Bias in predictive analytics does not just mean mathematical error; it refers to systematic unfairness across groups defined by sensitive attributes (such as race, religion, gender, or age).
• Demographic Parity (Independence): The acceptance or prediction rate is identical across all protected groups: \( P(\hat{Y}=1 | A=0) = P(\hat{Y}=1 | A=1) \).
• Equalized Odds (Separation): The model has equal true positive rates and equal false positive rates across groups.
• Predictive Parity (Sufficiency): The positive predictive value (precision) is equal across groups: \( P(Y=1 | \hat{Y}=1, A=0) = P(Y=1 | \hat{Y}=1, A=1) \).
Note: Mathematical fairness criteria often conflict with one another; satisfying demographic parity may require sacrificing predictive parity when underlying base rates differ.

Proxy Discrimination

Simply removing a protected characteristic from the model is rarely enough. Proxy discrimination occurs when an algorithm uses neutral predictors that correlate strongly with a protected attribute, effectively reconstructing the excluded characteristic.
• Example: Omitting race from a property insurance pricing model, but including five-digit ZIP codes, credit scores, or neighborhood tenure that mirror historical segregation patterns.
• Actuaries must actively test whether non-protected features serve as proxies and verify that model predictions do not produce disparate impact.

Mitigating Model Bias

1. Pre-processing: Re-balancing, re-weighting, or transforming the training sample so that representation and historical imbalances are corrected before model training.
2. In-processing: Incorporating fairness constraints or fairness penalties directly into the objective function during model fitting.
3. Post-processing: Adjusting classification decision thresholds or calibrations separately across groups to ensure equitable decision outcomes.

5. Cross-Validation: The Resampling Safeguard

To guard against single-split variance and verify that regularization and feature reduction have controlled overfitting, actuaries rely on \( K \)-Fold Cross-Validation.

Step-by-Step Process:
1. Randomly split the training data into \( k \) mutually exclusive folds of approximately equal size (typically \( k = 5 \) or \( k = 10 \)).
2. Train the candidate model on \( k - 1 \) folds combined.
3. Compute the validation metric on the single remaining fold.
4. Repeat this procedure \( k \) times, allowing every fold to serve as the holdout assessment exactly once.
5. Average the \( k \) performance metrics to yield a stable estimate of out-of-sample performance.

Why do we do this? Cross-validation maximizes data efficiency, avoids the risk of an unrepresentative single split, and provides an objective benchmark for hyperparameter tuning.

Chapter Summary Review

Quick Review Box:
• Overfitting: Occurs when a model captures random noise rather than underlying signal, exacerbated by starting with too many variables.
• Regularization: Penalizes model complexity. Lasso (L1) performs variable selection by driving coefficients to zero; Ridge (L2) shrinks coefficients to reduce variance.
• Repeated Data Use: Repeatedly evaluating models on the same validation/test set causes holdout overfitting and inflated performance metrics. Mitigate by using a strict three-way split and locking the test set until final sign-off.
• Data Leakage: Unintended flow of future or holdout information into model training. Prevent by splitting before any preprocessing.
• Fairness and Proxy Discrimination: Excluding sensitive attributes does not prevent discrimination if correlated proxies remain. Mitigate via pre-processing, in-processing, or post-processing fairness adjustments.
• Cross-Validation: Rotates \( k \) data folds to deliver robust, low-variance performance evaluations.