Welcome to the World of Hypothesis Testing!
In the FRM curriculum, Hypothesis Testing is like being a detective in the world of data. Up until now, you’ve learned how to calculate sample means and variances. Now, we learn how to evaluate whether observed patterns represent a genuine statistical relationship or merely the artifact of random sampling variation.
Whether you are testing whether an active investment strategy generates alpha or verifying that a value-at-risk (VaR) model is properly calibrated, hypothesis testing is your primary quantitative framework.
1. The Starting Line: Null and Alternative Hypotheses
Every hypothesis test begins with two mutually exclusive and exhaustive statements:
The Null Hypothesis ( (H_0 )): This is the default assertion or status quo. It typically posits no effect, no difference, or that a parameter equals a specific baseline value ( ( heta = heta_0 )). In statistical testing, we maintain (H_0 ) as true unless the sample evidence contradicts it beyond a reasonable threshold.
The Alternative Hypothesis ( (H_a ) or (H_1 )): This is the research statement you seek to substantiate. We reject (H_0 ) in favor of (H_a ) only when the empirical evidence is sufficiently strong.
One-Sided vs. Two-Sided Tests
The structure of the test depends directly on the research question:
Two-Sided (Two-Tailed) Test: Used when testing for any deviation from the hypothesized parameter in either direction.
Example: Is portfolio expected return ( mu ) different from 5%? ( (H_0: mu = 5 % ) vs. (H_a: mu neq 5 % ))
One-Sided (One-Tailed) Test: Used when testing for a specific directional deviation (greater than or less than).
Example: Is portfolio expected return greater than 5%? ( (H_0: mu leq 5 % ) vs. (H_a: mu > 5 % ))
Key Convention:
The Null Hypothesis ( (H_0 )) always contains the equality sign ( (= ), ( leq ), or ( geq )). The Alternative Hypothesis ( (H_a )) represents the strict inequality ( ( neq ), (> ), or (< )).
2. Type I and Type II Errors, Test Size, and Power
Because statistical decisions rely on sample estimates rather than full population censuses, two potential errors can occur:
Type I Error: Rejecting the Null Hypothesis when it is actually true (a false positive).
The probability of committing a Type I error is the Significance Level ( ( alpha )), also known as the size of the test: (P( text{Reject } H_0 mid H_0 text{ is true}) = alpha ).
Type II Error: Failing to reject the Null Hypothesis when it is actually false (a false negative).
The probability of committing a Type II error is denoted by ( beta ): (P( text{Fail to reject } H_0 mid H_0 text{ is false}) = beta ).
Power of the Test
The Power of a Test is the probability of correctly rejecting a false Null Hypothesis: ( text{Power} = 1 - beta ). Power increases with larger sample size ( (n )), larger true effect size (distance between true parameter and hypothesized value), and higher chosen significance level ( ( alpha )).
3. Decision Framework: Test Statistics, P-Values, and Confidence Intervals
A standardized Test Statistic measures the distance between the sample estimate and the hypothesized value under (H_0 ), scaled by the standard error ( (SE )):
( text{Test Statistic} = frac{ text{Sample Statistic} - text{Hypothesized Value}}{SE( text{Sample Statistic})} )
The Critical Value Approach
At a chosen significance level ( alpha ), compare the calculated test statistic to the critical value from the reference distribution:
- For a two-sided test: Reject (H_0 ) if ( vert text{Test Statistic} vert > vert text{Critical Value}_{ alpha/2} vert ).
- For a one-sided test: Reject (H_0 ) if the test statistic falls strictly into the rejection tail defined by ( alpha ).
The P-Value Approach
The p-value is the smallest significance level at which (H_0 ) would be rejected. Equivalently, it is the probability of observing a test statistic at least as extreme as the calculated sample value, assuming (H_0 ) is true.
- If ( text{p-value} < alpha ), Reject (H_0 ) (statistically significant).
- If ( text{p-value} geq alpha ), Fail to reject (H_0 ).
Duality with Confidence Intervals
There is an exact correspondence between a two-sided hypothesis test and a Confidence Interval (CI) at confidence level (1 - alpha ):
(CI = text{Sample Estimate} pm ( text{Critical Value} times SE) )
A hypothesized value ( mu_0 ) is rejected at significance level ( alpha ) if and only if ( mu_0 ) lies strictly outside the ((1 - alpha) ) confidence interval. If ( mu_0 ) lies within the interval, we fail to reject (H_0 ) at level ( alpha ).
4. Hypothesis Tests for Single Populations
Testing a Population Mean ( ( mu )):
- Z-test: Used when population variance ( sigma^2 ) is known: (Z = frac{ bar{x} - mu_0}{ sigma / sqrt{n}} ).
- t-test: Used when population variance is unknown and estimated by sample variance (s^2 ), with (n - 1 ) degrees of freedom: (t = frac{ bar{x} - mu_0}{s / sqrt{n}} ).
Testing a Single Population Variance ( ( sigma^2 )):
- Chi-Square ( ( chi^2 )) Test: For a normally distributed population, testing (H_0: sigma^2 = sigma_0^2 ) uses the test statistic ( chi^2 = frac{(n - 1)s^2}{ sigma_0^2} ) with (n - 1 ) degrees of freedom.
Comparing Two Population Variances:
- F-test: Tests (H_0: sigma_1^2 = sigma_2^2 ) using independent samples: (F = frac{s_1^2}{s_2^2} ) with (n_1 - 1 ) numerator and (n_2 - 1 ) denominator degrees of freedom.
5. Testing the Difference Between Two Population Means
Financial risk management frequently requires comparing returns or risk measures between two portfolios, assets, or time regimes. The appropriate test depends on whether the two samples are independent or paired:
A. Independent Samples
When testing (H_0: mu_1 - mu_2 = d_0 ) (frequently (d_0 = 0 )) across two distinct populations:
(t = frac{( bar{x}_1 - bar{x}_2) - d_0}{SE( bar{x}_1 - bar{x}_2)} )
- Equal Variances Assumed (Pooled Variance): When ( sigma_1^2 = sigma_2^2 ), compute pooled variance (s_p^2 = frac{(n_1 - 1)s_1^2 + (n_2 - 1)s_2^2}{n_1 + n_2 - 2} ) with (SE = s_p sqrt{ frac{1}{n_1} + frac{1}{n_2}} ) and (n_1 + n_2 - 2 ) degrees of freedom.
- Unequal Variances (Unpooled): When variances cannot be assumed equal, (SE = sqrt{ frac{s_1^2}{n_1} + frac{s_2^2}{n_2}} ), with degrees of freedom adjusted via Welch's formula.
B. Paired (Dependent) Samples
When observations in both samples are naturally paired (e.g., performance of the same fund before and after a policy shift), compute pairwise differences (d_i = x_{1,i} - x_{2,i} ). The hypothesis (H_0: mu_d = d_0 ) reduces to a single-sample t-test on the mean difference ( bar{d} ):
(t = frac{ bar{d} - d_0}{s_d / sqrt{n}} )
where (s_d ) is the sample standard deviation of the pairwise differences, evaluated with (n - 1 ) degrees of freedom.
6. The Multiple-Testing Problem
In quantitative modeling and backtesting, analysts often test dozens or hundreds of candidate variables, trading strategies, or risk factors simultaneously. This creates the multiple-testing problem (also known as data snooping or p-hacking).
If you conduct (m ) independent tests, each at significance level ( alpha ), the probability of making at least one Type I error across the entire family of tests—called the Family-Wise Error Rate (FWER)—is:
(FWER = 1 - (1 - alpha)^m )
Example: If you test 20 independent strategies at ( alpha = 0.05 ), the probability of finding at least one false positive is (1 - (1 - 0.05)^{20} approx 64.15 % ).
Remedies for Multiple Testing
Bonferroni Correction: To maintain an overall family-wise error rate of ( alpha ), each individual test is conducted at a adjusted significance level:
( alpha^* = frac{ alpha}{m} )
Equivalently, individual p-values are multiplied by (m ). While simple and conservative, the Bonferroni correction can reduce test power when (m ) is large.
False Discovery Rate (FDR): Modern alternatives control the expected proportion of false rejections among all rejected hypotheses rather than suppressing any single Type I error.
7. Step-by-Step Guide to Hypothesis Testing
Step 1: Formulate the Null ( (H_0 )) and Alternative ( (H_a )) hypotheses.
Step 2: Select the significance level ( ( alpha )), such as 0.05 or 0.01.
Step 3: Select the appropriate test statistic (Z, t, Chi-square, or F) and specify its distribution under (H_0 ).
Step 4: Compute the sample test statistic and corresponding p-value or critical value.
Step 5: Apply the decision rule: Reject (H_0 ) if the test statistic exceeds the critical value (or ( text{p-value} < alpha )); otherwise, fail to reject (H_0 ).
8. Common Pitfalls to Avoid
1. 'Accepting' the Null: We never say 'accept (H_0 ).' We only 'fail to reject (H_0 ).' Failing to reject merely indicates that sample evidence is insufficient to contradict the null hypothesis, not that the null is proven true.
2. Misallocating Alpha across Tails: For a two-sided test at ( alpha = 5 % ), critical regions allocate (2.5 % ) ( ( alpha/2 )) into each tail. For a one-sided test, the entire (5 % ) is concentrated in one tail, yielding a smaller critical value threshold.
3. Confusing Statistical and Economic Significance: A tiny effect size can produce a statistically significant p-value if sample size (n ) is extremely large, but transaction costs or market frictions may render it economically negligible.