Cramer Rao Lower Bound

Given a statistical model \(X P_\theta\) with a fixed true parameter \(\theta\), the Cramér–Rao lower bound (CRLB) provides a lower bound on the variance of an estimator T(X). The CRLB is useful because if an unbiased estimator achieves the CRLB, it must be a uniformly minimum–variance unbiased estimator , which achieves 2 points,

  1. first gain the relation of \(MSE = Var(\theta(Y)) + (Bias(\theta))^2\)

  2. The current goal of parameteraztion becomes minimizing the above two terms

because it is unbiased by construction and has minimum variance by the CRLB.

Definition

CRLB: Let \(X=\left(X_1, \ldots, X_N\right) \in \mathbb{R}^N\) be a random vector with joint density \(f(X ; \theta)\) where \(\theta \in\) \(\Theta \subseteq \mathbb{R}\). Let \(T(X)\) be a biased estimator of \(\theta\). Assume the Fisher information is always defined and that the operations of integration with respect to \(X\) and differention with respect to \(\theta\) can be interchanged. Then \[ \mathbb{V}[T(X)] \geq \frac{\left(\frac{d}{d \theta} \mathbb{E}[T(X)]\right)^2}{\mathbb{E}\left[\left(\frac{d}{d \theta} \log f(X ; \theta)\right)^2\right]} \equiv \operatorname{CRLB}(\theta) \]

The denominator of the CRLB is the Fisher information. If the estimator is unbiased, then the numerator is one since \(\mathbb{E}[T(X)]=\theta\).

Proof

To prove the CRLB, let \(W\) and \(Y\) be two random variables. In general, \(\mathbb{E}[W] \neq 0\) and \(\mathbb{E}[Y] \neq 0\). However, assume \(\mathbb{E}[Y]=0\). A property of covariance is \[ (\operatorname{Cov}[W, Y])^2 \leq \mathbb{V}[W] \mathbb{V}[Y] . \]

This can be derived by applying Cauchy-Schwarz to random variables (see the Appendix). Now set \(W\) and \(Y\) to \[ W=T(X), \quad Y=\frac{\partial}{\partial \theta} \log f(X ; \theta) \]

We know that the expectation of the score is zero. Therefore, \(\mathbb{E}[Y]=0\) as desired. Then Equation 2 can be rewritten as \[ \begin{aligned} \mathbb{V}[T(X)] \mathbb{V}\left[\frac{\partial}{\partial \theta} \log f(X ; \theta)\right] & \geq\left(\operatorname{Cov}\left[T(X), \frac{\partial}{\partial \theta} \log f(X ; \theta)\right]\right)^2 \\ \mathbb{V}[T(X)] & \geq \frac{\left(\operatorname{Cov}\left[T(X), \frac{\partial}{\partial \theta} \log f(X ; \theta)\right]\right)^2}{\mathbb{V}\left[\frac{\partial}{\partial \theta} \log f(X ; \theta)\right]} \end{aligned} \]

The numerator is our desired quantity. Ignoring the square, we have \[ \begin{aligned} & \operatorname{Cov}\left[T(X), \frac{\partial}{\partial \theta} \log f(X ; \theta)\right] \\ & =\mathbb{E}\left[T(X) \frac{\partial}{\partial \theta} \log f(X ; \theta)\right]-\overbrace{\mathbb{E}[T(X)] \mathbb{E}\left[\frac{\partial}{\partial \theta} \log f(X ; \theta)\right]}^{=0} \\ & =\int T(X) \frac{\partial}{\partial \theta} \log f(X ; \theta) f(X ; \theta) \mathrm{d} \mu(X) \\ & \triangleq \int T(X) \frac{\partial}{\partial \theta} f(X ; \theta) \mathrm{d} \mu(X) \\ & \stackrel{\dagger}{=} \frac{\partial}{\partial \theta} \int T(X) f(X ; \theta) \mathrm{d} \mu(X) \\ & =\frac{\partial}{\partial \theta} \mathbb{E}[T(X)] . \end{aligned} \]

In step \(\star\), we use the fact that if \(g(x)=\log h(x)\), then \(g^{\prime}(x)=h^{\prime}(x) / h(x)\). In step \(\dagger\), we use our assumption that we can interchange integration and differention.

The denominator is the desired quantity because \[ \begin{aligned} \mathbb{V}\left[\frac{\partial}{\partial \theta} \log f(X ; \theta)\right] & =\mathbb{E}\left[\left(\frac{\partial}{\partial \theta} \log f(X ; \theta)\right)^2\right]-\mathbb{E}\left[\frac{\partial}{\partial \theta} \log f(X ; \theta)\right]^2 \\ & =\mathbb{E}\left[\left(\frac{\partial}{\partial \theta} \log f(X ; \theta)\right)^2\right] \end{aligned} \]

Putting these results together in Equation 4, we have \[ \mathbb{V}[T(X)] \geq \frac{\left(\frac{\partial}{\partial \theta} \mathbb{E}[T(X)]\right)^2}{\mathbb{E}\left[\left(\frac{\partial}{\partial \theta} \log f(X ; \theta)\right)^2\right]} \] as desired.