Lehmann–Scheffé theorem

Proof

In the following, the dependence of an estimator on the data ${\displaystyle X}$ will not be written out explicitly, i.e., we write ${\displaystyle S}$ instead of ${\displaystyle S(X)}$.

First of all, if there is no unbiased estimator for ${\displaystyle g(\theta )}$, then there is obviously no UMVUE, and if all unbiased estimator are not square-integrable, then their variances are infinity and the statement is trivial. Thus, we focus on the case where a square-integrable unbiased estimator exists.

Uniqueness of ${\displaystyle h}$: Let ${\displaystyle S=h_{1}(T)}$ and ${\displaystyle R=h_{2}(T)}$ be unbiased estimators of ${\displaystyle g(\theta )}$ for some measurable functions ${\displaystyle h_{1}}$ and ${\displaystyle h_{2}}$. For the expectation of the difference it holds$${\displaystyle \mathbb {E} _{\theta }[h_{1}(T)-h_{2}(T)]=g(\theta )-g(\theta )=0.}$$Since ${\displaystyle T}$ is complete, this implies ${\displaystyle h_{1}-h_{2}=0}$. Thus, ${\displaystyle T^{\ast }=h_{1}(T)=:h(T)}$ is the unique unbiased estimator that is a function of ${\displaystyle T}$.^[3]

${\displaystyle h(T)}$ is the UMVUE: Let ${\displaystyle S}$ be any square-integrable unbiased estimator and ${\displaystyle S^{\ast }=\mathbb {E} _{\theta }[S\mid T]}$. By the factorization lemma there exists measurable function ${\displaystyle {\tilde {h}}}$ such that ${\displaystyle S^{\ast }={\tilde {h}}(T)}$ and since ${\displaystyle S^{\ast }}$ is unbiased as well, by the above, it must hold ${\displaystyle S^{\ast }={\tilde {h}}(T)=h(T)=T^{\ast }}$. Thus, by the Rao–Blackwell theorem, it follows$${\displaystyle \mathrm {Var} _{\theta }(T^{\ast })=\mathrm {Var} _{\theta }(S^{\ast })=\mathrm {Var} _{\theta }(\mathbb {E} _{\theta }[S\mid T])\leq \mathrm {Var} _{\theta }(S),\quad \forall \theta \in \Theta .}$$In other words, ${\displaystyle T^{\ast }=h(T)}$ is an UMVUE and according to the first part it is unique.^[4]

Bernoulli model

Let ${\displaystyle X_{1},\dots ,X_{n}}$ be Bernoulli-distributed with probability ${\displaystyle p\in (0,1)}$. The joint probability mass function ${\displaystyle f_{p}}$ is of the form$${\displaystyle f_{p}(x_{1},\dots ,x_{n})=\prod _{i=1}^{n}p^{x_{i}}(1-p)^{1-x_{i}}=\left({\frac {p}{1-p}}\right)^{\sum _{i=1}^{n}x_{i}}(1-p)^{n},\quad x_{i}\in \{0,1\},}$$thus, by the Fisher–Neyman factorization theorem, ${\displaystyle T(x)=\sum _{i=1}^{n}x_{i}}$ is a sufficient statistic. It is also complete: Let ${\displaystyle \varphi }$ be any measurable function such that ${\displaystyle \mathbb {E} _{\theta }[\varphi (T(X))]=0}$. Since ${\displaystyle T}$ is ${\displaystyle \mathrm {Ber} (n,p)}$-distributed, this means$${\displaystyle 0=\mathbb {E} _{\theta }[\varphi (T(X))]=\sum _{k=0}^{n}\varphi (k){\binom {n}{k}}p^{k}(1-p)^{n-k}=(1-p)^{n}\sum _{k=0}^{n}\varphi (k){\binom {n}{k}}r^{k},}$$with ${\displaystyle r=p/(1-p)}$. Since the right hand side is a polynomial in ${\displaystyle r>0}$ that is equal to zero, each coefficient must be zero as well, implying ${\displaystyle \varphi (k)=0}$ for all ${\displaystyle k}$.

For the estimation of the parameter ${\displaystyle p}$, it is now easy to see that the UMVUE is the sample mean$${\displaystyle {\overline {X}}={\frac {1}{n}}\sum _{i=1}^{n}X_{i}={\frac {1}{n}}T(X),}$$since it is unbiased and a function of ${\displaystyle T(X)}$.

Finding the UMVUE for the parameter ${\displaystyle p(1-p)}$ (that is, the variance of the distribution) is less obvious. According to method 1, we seek a function ${\displaystyle h}$ such that $${\displaystyle p(1-p)=\mathbb {E} _{p}[h(T(X))]=\sum _{k=0}^{n}h(k){\binom {n}{k}}p^{k}(1-p)^{n-k}.}$$By defining ${\displaystyle \rho =p/(1-p)}$, the above rewrites as$${\displaystyle \sum _{k=0}^{n}h(k){\binom {n}{k}}\rho ^{k}=\rho (1+\rho )^{n-2}=\sum _{k=1}^{n-1}{\binom {n-1}{k-1}}\rho ^{k}.}$$Comparing the coefficients shows that ${\displaystyle h(k)=k(n-k)/(n(n-1))}$, thus, the UMVUE is given by^[5]$${\displaystyle h(T(X))={\frac {n}{n-1}}{\overline {X}}(1-{\overline {X}}).}$$Noting that in the Bernoulli model ${\displaystyle X_{i}=X_{i}^{2}}$ and that ${\displaystyle \sum _{i=1}^{n}(X_{i}-{\overline {X}})^{2}=\sum _{i=1}^{n}X_{i}^{2}-n{\overline {X}}^{2}}$, the UMVUE is, in fact, the unbiased sample variance ${\displaystyle S_{n}^{2}}$.

Counterexample

An example of an improvable Rao–Blackwell improvement, when using a minimal sufficient statistic that is not complete, was provided by Galili and Meilijson in 2016.^[7] Let ${\displaystyle X_{1},\ldots ,X_{n}}$ be a random sample from a scale-uniform distribution ${\displaystyle X\sim U((1-k)\theta ,(1+k)\theta ),}$ with unknown mean ${\displaystyle \operatorname {E} [X]=\theta }$ and known design parameter ${\displaystyle k\in (0,1)}$. In the search for "best" possible unbiased estimators for ${\displaystyle \theta }$, it is natural to consider ${\displaystyle X_{1}}$ as an initial (crude) unbiased estimator for ${\displaystyle \theta }$ and then try to improve it. Since ${\displaystyle X_{1}}$ is not a function of ${\displaystyle T=\left(X_{(1)},X_{(n)}\right)}$, the minimal sufficient statistic for ${\displaystyle \theta }$ (where ${\displaystyle X_{(1)}=\min _{i}X_{i}}$ and ${\displaystyle X_{(n)}=\max _{i}X_{i}}$), it may be improved using the Rao–Blackwell theorem as follows:

${\displaystyle {\hat {\theta }}_{RB}=\operatorname {E} _{\theta }[X_{1}\mid X_{(1)},X_{(n)}]={\frac {X_{(1)}+X_{(n)}}{2}}.}$

However, the following unbiased estimator can be shown to have lower variance:

${\displaystyle {\hat {\theta }}_{LV}={\frac {1}{k^{2}{\frac {n-1}{n+1}}+1}}\cdot {\frac {(1-k)X_{(1)}+(1+k)X_{(n)}}{2}}.}$

And in fact, it could be even further improved when using the following estimator:

${\displaystyle {\hat {\theta }}_{\text{BAYES}}={\frac {n+1}{n}}\left[1-{\frac {{\frac {X_{(1)}(1+k)}{X_{(n)}(1-k)}}-1}{\left({\frac {X_{(1)}(1+k)}{X_{(n)}(1-k)}}\right)^{n+1}-1}}\right]{\frac {X_{(n)}}{1+k}}}$

The model is a scale model. Optimal equivariant estimators can then be derived for loss functions that are invariant.^[8]