Welcome to Regression Diagnostics: The "Health Checkup" for Your Models
In your FRM journey so far, you have learned how to estimate linear regressions using Ordinary Least Squares (OLS). But how do we know if a model is reliable for risk management? A flawed model can produce misleading risk forecasts, flawed hedge ratios, and catastrophic risk management decisions. Regression Diagnostics is the process of testing whether classical regression assumptions hold and identifying potential model defects.
1. Conditions for OLS to Be BLUE
According to the Gauss-Markov Theorem, the OLS estimator is the Best Linear Unbiased Estimator (BLUE)—meaning it has the smallest variance among all linear unbiased estimators—provided the following conditions are met:
- Linearity in Parameters: The dependent variable is a linear function of the population parameters and an error term.
- Random Sampling / Strict Exogeneity: The conditional expectation of the error term given the regressors is zero, \( E[\epsilon | X] = 0 \). This implies regressors are uncorrelated with the error term.
- No Perfect Collinearity: No independent variable is an exact linear combination of other independent variables.
- Spherical Errors:
1. Homoskedasticity: The error term has a constant conditional variance, \( \text{Var}(\epsilon_i | X) = \sigma^2 \).
2. No Autocorrelation: The error terms are uncorrelated across observations, \( \text{Cov}(\epsilon_i, \epsilon_j | X) = 0 \) for \( i \neq j \).
If these conditions hold, OLS is BLUE. If the error terms are additionally normally distributed, OLS estimators are the minimum variance unbiased estimators among all unbiased estimators (BUE), not just linear ones.
2. Heteroskedasticity: Non-Constant Error Variance
What is it?
Heteroskedasticity occurs when the variance of the error terms is not constant across observations.
Analogy: Imagine an archer whose shots are tightly clustered when standing close to the target, but scatter widely at greater distances. The shot variance expands with distance. That is heteroskedasticity.
Types of Heteroskedasticity
1. Unconditional Heteroskedasticity: The variance changes, but not in a way related to the values of the independent variables. This typically does not compromise statistical inference.
2. Conditional Heteroskedasticity: The error variance is linked to the levels of the independent variables. This is the primary concern in the FRM exam because it distorts standard hypothesis tests.
Consequences, Detection, and Remedies
Consequences: OLS coefficient estimates (\( \hat{\beta} \)) remain unbiased and consistent. However, OLS standard errors are biased (usually underestimated), leading to inflated \( t \)-statistics and incorrect conclusions regarding statistical significance.
How to spot it:
- Residual Plots: Scatter plots of residuals against fitted values or regressors showing a distinct "fan" or "funnel" pattern.
- Breusch-Pagan Test: A formal test that regresses squared residuals on the independent variables to test for linear variance dependence.
- White Test: A test that regresses squared residuals on the regressors, their squares, and cross-products, detecting both linear and non-linear heteroskedasticity without assuming a specific functional form.
How to fix it: Calculate White's Heteroskedasticity-Consistent Standard Errors (robust standard errors). These adjust standard errors and \( t \)-statistics without altering coefficient estimates.
3. Multicollinearity: Imperfect vs. Perfect Collinearity
A crucial distinction exists between perfect collinearity and imperfect multicollinearity:
- Perfect Collinearity: An exact linear relationship exists among two or more independent variables (e.g., \( X_2 = 2X_1 \), or falling into the "dummy variable trap" by including all categorical categories alongside an intercept). Under perfect collinearity, the data matrix cannot be inverted, and OLS estimation is mathematically impossible.
- Multicollinearity (Imperfect): Two or more explanatory variables are highly, but not perfectly, correlated with one another.
Consequences, Detection, and Remedies
Consequences: OLS estimators remain unbiased and consistent. However, standard errors become inflated, rendering individual \( t \)-statistics small and insignificant, even while overall goodness-of-fit measures remain strong. Coefficients also become sensitive to small changes in data.
How to spot it:
- High \( R^2 \) with Low \( t \)-Statistics: The classic diagnostic symptom where the overall \( F \)-statistic is significant, but individual regressors appear insignificant.
- Correlation Matrix: High pairwise correlations between regressors (e.g., \( |r| > 0.8 \)).
- Variance Inflation Factor (VIF): Quantifies how much the variance of an estimated coefficient is inflated due to collinearity. A \( \text{VIF} > 5 \) to \( 10 \) indicates serious multicollinearity.
How to fix it: Drop one of the redundant collinear variables, collect additional data, or transform the correlated variables (e.g., taking differences or constructing principal components).
4. Visualizing Residuals and Identifying Outliers
Residual diagnostics provide visual and numerical tools to verify regression adequacy:
- Visualizing Residuals: Plotting residuals (\( e_i \)) against fitted values (\( \hat{Y}_i \)) or independent variables (\( X_i \)) allows quick detection of non-linearity (curved patterns), heteroskedasticity (funnel patterns), or structural breaks.
- Outliers: An observation with a residual substantially larger than others, indicating the model fits that point poorly.
- High Leverage Points: Observations with extreme values in the explanatory variables (far from the sample mean of \( X \)). High leverage means the observation has the potential to disproportionately affect the regression line.
- Influential Observations: An observation that is both an outlier and possesses high leverage. Removing an influential observation noticeably shifts estimated regression coefficients. Influential points are commonly assessed via Cook's distance and studentized residuals.
5. Model Specification: Omitted Variables vs. Irrelevant Regressors
Specification errors arise from choosing an incorrect set of variables or wrong functional form:
- Omitted Variable Bias: Occurs when a variable that genuinely influences \( Y \) and correlates with one or more included regressors is left out. Consequence: OLS estimates of the included variables become biased and inconsistent.
- Including Irrelevant Regressors: Occurs when variables that have no relationship to \( Y \) are included. Consequence: OLS estimates remain unbiased and consistent, but estimation variance increases (efficiency is lost), leading to larger standard errors.
- Functional Form Misspecification: Fitting a linear equation to a non-linear relationship (evaluated via tests such as Ramsey's RESET test).
6. Model Selection and the Bias-Variance Trade-Off
When selecting regressors, risk modelers face the bias-variance trade-off:
- Underfitting (High Bias, Low Variance): Omitting variables keeps parameter estimation variance low across samples, but introduces substantial bias if relevant regressors are omitted.
- Overfitting (Low Bias, High Variance): Including many regressors reduces in-sample bias and inflates \( R^2 \), but inflates parameter estimation variance, causing the model to fit sample noise and predict poorly out-of-sample.
Model Selection Criteria
Standard \( R^2 \) never decreases when regressors are added. To prevent overfitting, penalized criteria are used:
- Adjusted \( R^2 \): Adjusts for the number of estimated parameters \( k \) relative to sample size \( n \). It penalizes unnecessary variables, but the penalty is modest.
- Akaike Information Criterion (AIC): Balances likelihood against parameter count, penalizing model complexity. Frequently used when the primary goal is out-of-sample prediction.
- Bayesian (Schwarz) Information Criterion (BIC): Imposes a penalty proportional to \( \ln(n) \), which penalizes extra parameters more severely than AIC for large sample sizes, favoring more parsimonious models.
- Mallows' \( C_p \): Compares the precision of a subset model to the full model, where a \( C_p \approx k \) indicates an unbiased model.
7. Serial Correlation and Residual Normality
- Serial Correlation (Autocorrelation): Residuals from one period correlate with adjacent periods, common in time series. While coefficients remain unbiased, standard errors are underestimated. Detected using the Durbin-Watson (DW) test (where \( DW \approx 2 \) means no serial correlation, \( DW < 2 \) indicates positive serial correlation). Corrected using Newey-West standard errors (HAC standard errors), which correct for both heteroskedasticity and autocorrelation.
- Normality of Residuals: Evaluated using the Jarque-Bera (JB) test, which evaluates skewness (expected 0 under normality) and kurtosis (expected 3). A significant JB statistic rejects normality.
8. Diagnostic Summary Table
- Heteroskedasticity: Detect with Breusch-Pagan / White test. Remedy with White robust standard errors.
- Perfect Collinearity: Exact linear dependence. Estimation fails; drop redundant regressor.
- Multicollinearity: Detect via high \( R^2 \) with low \( t \)-stats, high VIF. Remedy by dropping or combining variables.
- Omitted Variable: Causes bias and inconsistency. Remedy by including the omitted variable.
- Irrelevant Regressor: Preserves unbiasedness but inflates variance. Remedy by pruning model using AIC/BIC.
- Outliers & Influential Points: Inspect residual plots, leverage values, and Cook's distance.