Empirical Bayesian estimation: nonparametric and semiparametric cases

Q: What is the difference between nonparametric and semiparametric Empirical Bayesian estimation?

Nonparametric estimation assumes no distributions for either the conditional loss or \(\Theta\), whereas semiparametric estimation assumes a specific parametric form for the conditional loss given \(\Theta\) while leaving the prior of \(\Theta\) unspecified.

Q: In the equal sample size nonparametric model, why is \(\frac{\hat{v}}{n}\) subtracted when estimating the variance of hypothetical means \(a\)?

Because the total variance among sample means reflects both true risk heterogeneity and sample process noise: \(Var(\bar{X}_i) = a + \frac{v}{n}\), so subtracting \(\frac{\hat{v}}{n}\) isolates the unbiased estimate of \(a\).

Q: What are the nonparametric estimators for \(v\) and \(a\) under the Bñhlmann-Straub model with unequal exposures?

\(\hat{v} = \frac{\sum_{i=1}^{k} \sum_{j=1}^{n_i} m_{ij} (X_{ij} - \bar{X}_i)^2}{\sum_{i=1}^{k} (n_i - 1)}\) and \(\hat{a} = \frac{\sum_{i=1}^{k} m_i (\bar{X}_i - \bar{X})^2 - (k - 1)\hat{v}}{m - \frac{1}{m} \sum_{i=1}^{k} m_i^2}\)

Q: In the Poisson semiparametric Empirical Bayesian model, why does \(v = \mu\)?

Because for a Poisson claim count variable, the conditional mean equals the conditional variance: \(v(\Theta_i) = \mu(\Theta_i)\), which implies \(v = E[v(\Theta)] = E[\mu(\Theta)] = \mu\).

Q: Why can \(\hat{a}\) evaluate to a negative number, and what action must be taken if it does?

\(\hat{a}\) can be negative due to sample variance when process variance exceeds between-risk variation; in this case, we apply the truncation rule \(\hat{a}^* = \max(0, \hat{a}) = 0\), setting credibility \(Z = 0\).

Q: Once the Empirical Bayesian parameters \(\hat{\mu}\), \(\hat{v}\), and \(\hat{a}^*\) are determined, how is the credibility premium \(P_i\) computed for risk \(i\)?

\(P_i = Z_i \bar{X}_i + (1 - Z_i)\hat{\mu}\), where \(Z_i = \frac{m_i}{m_i + \hat{v}/\hat{a}^*}\) (or \(Z = \frac{n}{n + \hat{v}/\hat{a}^*}\) when exposures are equal).

Exam ASTAM – Advanced Short-Term Actuarial Mathematics · SOA (Society of Actuaries) · 4 min read · Updated: 9 Nov 2026

Welcome to Empirical Bayesian Estimation!

In classical Bayesian credibility, we specify both a conditional distribution for the risk given the risk parameter \(\Theta\) and a prior distribution for \(\Theta\). However, in practice, the prior distribution of \(\Theta\) is almost never known. This is where Empirical Bayesian Estimation steps in: rather than assuming a prior distribution, we estimate the structural parameters—the overall mean \(\mu\), the expected value of the process variance (EPV) \(v\), and the variance of the hypothetical means (VHM) \(a\)—directly from past data across multiple risks.

The method is divided into two primary cases based on the level of distributional assumptions made:

- Nonparametric estimation: No parametric distribution is assumed for either the conditional loss distribution or the prior parameter distribution. Estimation relies entirely on sample moments and variances.

- Semiparametric estimation: A parametric conditional distribution is assumed for the loss given \(\Theta\) (for example, Poisson or exponential), which establishes a functional relationship between the conditional mean and the conditional variance. However, no distribution is assumed for \(\Theta\).

1. The Nonparametric Case

In the nonparametric setting, we assume we have data on \(k\) risks. We consider two standard models: equal sample sizes and unequal exposures.

Model 1: Equal Sample Sizes (Bühlmann Model)

Suppose each of the \(k\) policyholders is observed for \(n\) periods, with observed claim amount \(X_{ij}\) for risk \(i\) in period \(j\).

Step 1: Individual Risk Means
\(\bar{X}_i = \frac{1}{n} \sum_{j=1}^{n} X_{ij}\)

Step 2: Grand Mean (\(\hat{\mu}\))
\(\hat{\mu} = \bar{X} = \frac{1}{k} \sum_{i=1}^{k} \bar{X}_i\)

Step 3: Expected Value of Process Variance (\(\hat{v}\))
We average the unbiased sample variances across all \(k\) policyholders:
\(\hat{v} = \frac{1}{k(n - 1)} \sum_{i=1}^{k} \sum_{j=1}^{n} (X_{ij} - \bar{X}_i)^2\)

Step 4: Variance of Hypothetical Means (\(\hat{a}\))
The sample variance of the \(\bar{X}_i\)'s estimates \(Var(\bar{X}_i) = a + \frac{v}{n}\). Therefore, we obtain an unbiased estimator of \(a\) by subtracting the estimated process variance component:
\(\hat{a} = \frac{1}{k - 1} \sum_{i=1}^{k} (\bar{X}_i - \bar{X})^2 - \frac{\hat{v}}{n}\)

Model 2: Unequal Exposures (Bühlmann-Straub Nonparametric Model)

When risks have varying exposures \(m_{ij}\) across periods \(j = 1, \dots, n_i\), let \(m_i = \sum_{j=1}^{n_i} m_{ij}\) and \(m = \sum_{i=1}^{k} m_i\). The risk mean is \(\bar{X}_i = \frac{1}{m_i} \sum_{j=1}^{n_i} m_{ij} X_{ij}\), and the grand mean is \(\bar{X} = \frac{1}{m} \sum_{i=1}^{k} m_i \bar{X}_i\).

Estimating \(v\):
\(\hat{v} = \frac{1}{\sum_{i=1}^{k} (n_i - 1)} \sum_{i=1}^{k} \sum_{j=1}^{n_i} m_{ij} (X_{ij} - \bar{X}_i)^2\)

Estimating \(a\):
\(\hat{a} = \frac{\sum_{i=1}^{k} m_i (\bar{X}_i - \bar{X})^2 - (k - 1)\hat{v}}{m - \frac{1}{m} \sum_{i=1}^{k} m_i^2}\)

2. The Semiparametric Case

In semiparametric empirical Bayesian estimation, we specify the form of the conditional distribution \(X_{ij} | \Theta_i\), but leave the prior distribution of \(\Theta\) unspecified. Because the conditional model specifies a mathematical relationship between the conditional mean \(\mu(\Theta)\) and the conditional variance \(v(\Theta)\), we estimate \(v\) directly from this structural link rather than using a pure sample variance.

Poisson Frequency Model

The most common semiparametric model occurs when claim counts \(N_{ij}\) given \(\Theta_i\) follow a Poisson distribution with mean \(m_{ij} \lambda(\Theta_i)\). Define \(X_{ij} = \frac{N_{ij}}{m_{ij}}\) as the claim rate per unit of exposure.

For a Poisson variable, the conditional variance equals the conditional mean:
\(E[X_{ij} | \Theta_i] = \mu(\Theta_i) = \lambda(\Theta_i)\)
\(Var(X_{ij} | \Theta_i) = \frac{\lambda(\Theta_i)}{m_{ij}} = \frac{\mu(\Theta_i)}{m_{ij}}\)

Thus, the process variance parameter is \(v(\Theta_i) = \mu(\Theta_i)\), which implies:
\(v = E[v(\Theta)] = E[\mu(\Theta)] = \mu\)

Parameter Estimation for Poisson Semiparametric Model:
1. Overall Mean: \(\hat{\mu} = \bar{X} = \frac{1}{m} \sum_{i=1}^{k} \sum_{j=1}^{n_i} m_{ij} X_{ij} = \frac{\text{Total Claims}}{\text{Total Exposure}}\)
2. Process Variance: Since \(v = \mu\), our estimate is immediately:
\(\hat{v} = \hat{\mu}\)
3. Variance of Hypothetical Means: We substitute \(\hat{v} = \hat{\mu}\) into the Bühlmann-Straub numerator:
\(\hat{a} = \frac{\sum_{i=1}^{k} m_i (\bar{X}_i - \bar{X})^2 - (k - 1)\hat{\mu}}{m - \frac{1}{m} \sum_{i=1}^{k} m_i^2}\)

Note on equal exposures: When \(n_i = n\) and \(m_{ij} = 1\), this simplifies to \(\hat{v} = \hat{\mu} = \bar{X}\) and \(\hat{a} = \frac{1}{k - 1} \sum_{i=1}^{k} (\bar{X}_i - \bar{X})^2 - \frac{\hat{\mu}}{n}\).

3. The Truncation Rule for \(\hat{a}\)

Because \(\hat{a}\) is estimated by subtracting an estimated process variance component from observed between-group variation, it is possible for sample data to yield \(\hat{a} < 0\).

Since the variance of the hypothetical means cannot be negative, we use the modified estimator:
\(\hat{a}^* = \max(0, \hat{a})\)

When \(\hat{a} \le 0\), we set \(\hat{a}^* = 0\). This implies that individual experience exhibits no more variation than expected by pure chance (process variance), leading to a credibility factor of \(Z = 0\) and assigning 100% weight to the overall mean \(\hat{\mu}\).

Summary of Nonparametric vs. Semiparametric Estimation

1. Nonparametric: No conditional distribution assumed. \(\hat{v}\) is estimated using sample variances within risks across observation periods.
2. Semiparametric: Conditional distribution is assumed (e.g., Poisson, where \(v = \mu\)). We do not need sample variances of individual periods to estimate \(v\); we use \(\hat{v} = \hat{\mu}\).
3. Credibility Calculation: Once \(\hat{\mu}\), \(\hat{v}\), and \(\hat{a}\) are estimated, the credibility premium for risk \(i\) is calculated using standard formulas: \(P_i = Z_i \bar{X}_i + (1 - Z_i)\hat{\mu}\), where \(Z_i = \frac{m_i}{m_i + \hat{v}/\hat{a}}\) (or \(Z = \frac{n}{n + \hat{v}/\hat{a}}\) under equal exposures).

References

SOA Exam ASTAM Syllabus - Fall 2026 — Society of Actuaries
SOA Exam ASTAM Syllabus - Spring 2026 — Society of Actuaries

Quick check

Can you answer these now?

Open each question to check the key ideas from this chapter.

What is the difference between nonparametric and semiparametric Empirical Bayesian estimation?

Nonparametric estimation assumes no distributions for either the conditional loss or \(\Theta\), whereas semiparametric estimation assumes a specific parametric form for the conditional loss given \(\Theta\) while leaving the prior of \(\Theta\) unspecified.

In the equal sample size nonparametric model (Bñhlmann), how is the expected value of the process variance \(v\) estimated?

\(\hat{v} = \frac{1}{k(n - 1)} \sum_{i=1}^{k} \sum_{j=1}^{n} (X_{ij} - \bar{X}_i)^2\)

In the equal sample size nonparametric model, why is \(\frac{\hat{v}}{n}\) subtracted when estimating the variance of hypothetical means \(a\)?

Because the total variance among sample means reflects both true risk heterogeneity and sample process noise: \(Var(\bar{X}_i) = a + \frac{v}{n}\), so subtracting \(\frac{\hat{v}}{n}\) isolates the unbiased estimate of \(a\).

What are the nonparametric estimators for \(v\) and \(a\) under the Bñhlmann-Straub model with unequal exposures?

\(\hat{v} = \frac{\sum_{i=1}^{k} \sum_{j=1}^{n_i} m_{ij} (X_{ij} - \bar{X}_i)^2}{\sum_{i=1}^{k} (n_i - 1)}\) and \(\hat{a} = \frac{\sum_{i=1}^{k} m_i (\bar{X}_i - \bar{X})^2 - (k - 1)\hat{v}}{m - \frac{1}{m} \sum_{i=1}^{k} m_i^2}\)

In the Poisson semiparametric Empirical Bayesian model, why does \(v = \mu\)?

Because for a Poisson claim count variable, the conditional mean equals the conditional variance: \(v(\Theta_i) = \mu(\Theta_i)\), which implies \(v = E[v(\Theta)] = E[\mu(\Theta)] = \mu\).

How is \(\hat{a}\) calculated in a Poisson semiparametric model with unequal exposures?

\(\hat{a} = \frac{\sum_{i=1}^{k} m_i (\bar{X}_i - \bar{X})^2 - (k - 1)\hat{\mu}}{m - \frac{1}{m}\sum_{i=1}^{k} m_i^2}\), where \(\hat{\mu} = \frac{\text{Total Claims}}{\text{Total Exposure}}\).

Why can \(\hat{a}\) evaluate to a negative number, and what action must be taken if it does?

\(\hat{a}\) can be negative due to sample variance when process variance exceeds between-risk variation; in this case, we apply the truncation rule \(\hat{a}^* = \max(0, \hat{a}) = 0\), setting credibility \(Z = 0\).

Once the Empirical Bayesian parameters \(\hat{\mu}\), \(\hat{v}\), and \(\hat{a}^*\) are determined, how is the credibility premium \(P_i\) computed for risk \(i\)?

\(P_i = Z_i \bar{X}_i + (1 - Z_i)\hat{\mu}\), where \(Z_i = \frac{m_i}{m_i + \hat{v}/\hat{a}^*}\) (or \(Z = \frac{n}{n + \hat{v}/\hat{a}^*}\) when exposures are equal).

Ready to test yourself?

Turn these notes into exam-style practice. Get unlimited AI questions on this topic with instant marking and explanations.

Practise This Topic

More Exam ASTAM – Advanced Short-Term Actuarial Mathematics chapters

* The content provided by thinka is generated by AI and may not always be accurate or up-to-date. Please use it as a supplementary resource and verify with official materials.

Put These Notes into Practice

Reading the notes is a great start. Now practise with unlimited AI-generated questions and get instant feedback. 100,000+ students are already improving their grades.

Start Practising Now View Pricing

Done reading? Test yourself with AI practice questions

Practise This Topic Now