Define and apply a Markov process

Welcome to the World of Markov Processes!

Hello there! Welcome to a core chapter in your CS2 journey. In discrete-time Markov chains, systems evolve in fixed steps (like annual reviews of a No Claims Discount scheme). However, the real world often moves continuously: an insured individual can fall ill, recover, or pass away at any instant in time.

A continuous-time stochastic process possessing the memoryless property is called a Markov process (specifically, a Markov jump process when the state space is discrete). In this chapter, we define the essential mechanics of Markov processes, derive their governing differential equations, and apply them to standard actuarial models.

1. Definition and the Continuous-Time Markov Property

Let \( \{X_t : t \ge 0\} \) be a continuous-time stochastic process taking values in a discrete state space \( S \). The process satisfies the Markov property if the probability distribution of any future state depends only on the present state, and not on past history.

Formally, for any times \( 0 \le t_0 < t_1 < \dots < s < s+t \) and states \( x_0, x_1, \dots, i, j \in S \):

\( P(X_{s+t} = j \mid X_s = i, X_{t_n} = x_n, \dots, X_{t_0} = x_0) = P(X_{s+t} = j \mid X_s = i) \)

In terms of filtrations, letting \( \mathcal{F}_s \) denote the history of the process up to time \( s \):

\( P(X_{s+t} = j \mid \mathcal{F}_s) = P(X_{s+t} = j \mid X_s) \)

We denote the transition probability of moving from state \( i \) at time \( s \) to state \( j \) at time \( s+t \) by \( p_{ij}(s, s+t) \). If this probability depends only on the elapsed duration \( t \) and not on the start time \( s \), the process is time-homogeneous, written simply as \( p_{ij}(t) \).

2. Transition Intensities and the Generator Matrix

In discrete time, transitions are governed by probabilities over fixed steps. In continuous time, we describe instantaneous movement using transition intensities (also called hazard rates or forces of transition).

Definition of Transition Intensity

For \( i \ne j \), the transition intensity \( \mu_{ij}(t) \) at time \( t \) is defined as:

\( \mu_{ij}(t) = \lim_{h \to 0^+} \frac{p_{ij}(t, t+h)}{h} \)

This implies that over a tiny time interval \( h \):

• \( p_{ij}(t, t+h) = \mu_{ij}(t) h + o(h) \) for \( j \ne i \)
• \( p_{ii}(t, t+h) = 1 - \sum_{j \ne i} \mu_{ij}(t) h + o(h) = 1 + \mu_{ii}(t) h + o(h) \)

where \( o(h) \) denotes terms that vanish faster than \( h \) as \( h \to 0 \).

We define the diagonal entries by setting \( \mu_{ii}(t) = -\sum_{j \ne i} \mu_{ij}(t) \), so that the row sums of the intensity matrix are always zero.

The Generator Matrix (\( Q \))

When the process is time-homogeneous, the intensities are constant: \( \mu_{ij}(t) = \mu_{ij} \). We arrange them into a generator matrix \( Q \):

\( Q = \begin{pmatrix} \mu_{11} & \mu_{12} & \mu_{13} \\ \mu_{21} & \mu_{22} & \mu_{23} \\ \mu_{31} & \mu_{32} & \mu_{33} \end{pmatrix} \)

Essential Rules for \( Q \):

• Off-diagonal entries are non-negative: \( \mu_{ij} \ge 0 \) for all \( i \ne j \).
• Diagonal entries are non-positive: \( \mu_{ii} \le 0 \).
• Every row sums to zero: \( \sum_{j} \mu_{ij} = 0 \).

3. The Kolmogorov Differential Equations

The continuous-time analogues of the Chapman-Kolmogorov equations relate transition probabilities to transition intensities via differential equations.

Time-Homogeneous Case

For a time-homogeneous Markov jump process, we have two equivalent sets of equations:

1. Kolmogorov Forward Equations:
\( \frac{d}{dt} p_{ij}(t) = \sum_{k} p_{ik}(t) \mu_{kj} = \sum_{k \ne j} p_{ik}(t) \mu_{kj} + p_{ij}(t) \mu_{jj} \)
In matrix form: \( \frac{d}{dt} P(t) = P(t) Q \), with initial condition \( P(0) = I \).

2. Kolmogorov Backward Equations:
\( \frac{d}{dt} p_{ij}(t) = \sum_{k} \mu_{ik} p_{kj}(t) = \sum_{k \ne i} \mu_{ik} p_{kj}(t) + \mu_{ii} p_{ij}(t) \)
In matrix form: \( \frac{d}{dt} P(t) = Q P(t) \).

Time-Inhomogeneous Case

When intensities vary with time or age \( t \):

Forward Equation: \( \frac{\partial}{\partial t} p_{ij}(s, t) = \sum_{k} p_{ik}(s, t) \mu_{kj}(t) \)

Backward Equation: \( \frac{\partial}{\partial s} p_{ij}(s, t) = -\sum_{k} \mu_{ik}(s) p_{kj}(s, t) \)

Solving the Kolmogorov Equations in Simple Cases

Consider a simple two-state mortality model: State 0 (Alive) and State 1 (Dead), with constant transition intensity \( \mu_{01} = \mu \) and absorbing State 1 (\( \mu_{10} = 0 \)).

The forward equation for remaining in State 0 is:
\( \frac{d}{dt} p_{00}(t) = p_{00}(t) \mu_{00} = -\mu p_{00}(t) \)

Separating variables and using \( p_{00}(0) = 1 \):
\( p_{00}(t) = e^{-\mu t} \)

Since the probabilities must sum to 1:
\( p_{01}(t) = 1 - p_{00}(t) = 1 - e^{-\mu t} \)

4. Holding Times, Jump Chains, and Simulation

In this section, consider a time-homogeneous Markov jump process with constant transition intensities. The exponential holding-time and jump-probability formulas below apply in states with positive total exit rate.

Distribution of Holding Times

The time \( T_i \) spent in state \( i \) before jumping to any other state is called the holding time (or waiting time). Due to the Markov property, \( T_i \) is memoryless, meaning it follows an exponential distribution:

\( T_i \sim \text{Exponential}(\lambda_i) \), where \( \lambda_i = -\mu_{ii} = \sum_{j \ne i} \mu_{ij} \)

The expected holding time in state \( i \) is \( E[T_i] = \frac{1}{\lambda_i} = \frac{1}{-\mu_{ii}} \).

The Jump Chain

When the process leaves state \( i \), the probability that it jumps specifically to state \( j \) (for \( j \ne i \)) is given by the embedded jump chain probability:

\( q_{ij} = \frac{\mu_{ij}}{\sum_{k \ne i} \mu_{ik}} = \frac{\mu_{ij}}{-\mu_{ii}} \)

Simulating a Markov Jump Process

For a time-homogeneous Markov jump process, simulate the sample path step-by-step:

• Step 1: Start in initial state \( X_0 = i \) at time \( t = 0 \).
• Step 2: If \(-\mu_{ii}=0\), the state is absorbing: remain there for the rest of the simulation and stop. Otherwise, generate the holding time \( T \sim \text{Exponential}(-\mu_{ii}) \) using a uniform pseudo-random number \( U_1 \sim U(0, 1) \): \( T = -\frac{\ln(U_1)}{-\mu_{ii}} \).
• Step 3: Update current time to \( t + T \). If this exceeds the maximum simulation horizon, stop.
• Step 4: Select the next state \( j \ne i \) according to discrete jump probabilities \( q_{ij} \) using a second independent uniform random variable \( U_2 \).
• Step 5: Set \( i = j \) and return to Step 2.

5. Actuarial Applications

1. The Poisson Process

A homogeneous Poisson process with rate \( \lambda \) is a continuous-time Markov process on state space \( \{0, 1, 2, \dots\} \) counting the number of events \( N_t \) up to time \( t \). Its generator has \( \mu_{i, i+1} = \lambda \), \( \mu_{ii} = -\lambda \), and all other \( \mu_{ij} = 0 \).

• Number of events: \( P(N_t = k) = \frac{(\lambda t)^k e^{-\lambda t}}{k!} \) for \( k = 0, 1, 2, \dots \)
• Inter-event times: Independent and identically distributed \( \text{Exponential}(\lambda) \) variables.

2. Sickness-Death and Marriage Models

Markov jump processes are standard models for multi-state insurance and pension benefits:

• Healthy-Sick-Dead Model: States \( \{H, S, D\} \). Transitions exist from Healthy to Sick (\( \sigma \)), Sick to Healthy (\( \rho \)), Healthy to Dead (\( \mu \)), and Sick to Dead (\( \nu \)). State \( D \) is absorbing.
• Marriage Model: States \( \left\{\begin{gathered}\text{Single},\ \text{Married},\ \text{Widowed},\\ \text{Divorced},\ \text{Dead}\end{gathered}\right\} \).

6. Duration-Dependent (Semi-Markov) Models

In standard Markov processes, intensities depend only on current age or calendar time \( t \). However, in many health contracts, recovery rates depend directly on how long an individual has already been sick.

When transition intensities depend on duration of stay \( z \) in the current state (i.e. \( \mu_{ij}(t, z) \)), the process is duration-dependent (a semi-Markov process). The classical Markov property with respect to state alone no longer holds unless duration is tracked as part of an expanded state variable \( (X_t, Z_t) \).

7. Summary Checklist

1. Markov Process: Memoryless continuous-time process where future behaviour given the present does not depend on the past.
2. Intensities: \( \mu_{ij} \ge 0 \) for \( j \ne i \), and \( \mu_{ii} = -\sum_{j \ne i} \mu_{ij} \) so each row of \( Q \) sums to 0.
3. Kolmogorov Forward Equations: \( \frac{d}{dt} P(t) = P(t) Q \).
4. Holding Times: For constant positive exit rates, exponentially distributed with mean \( 1 / (-\mu_{ii}) \).
5. Simulation: For a time-homogeneous process, alternate exponential holding times and jump-chain destination states; stop when an absorbing state is reached.