Statistics [10]: Evaluation of Point Estimation

4 minute read

Published: January 10, 2021

Properties of point estimation, mean squre error and minimum variance unbiased estimation.

Properties of Point Estimation

Consistency

Assume $\theta\in\Theta$ is an unknown parameter, $\hat{\theta}_n=\hat{\theta}_n(x_1,x_2,...,x_n)$ is an estimation obtained from $n$ samples. If for any $\varepsilon > 0$ , there exists $n$ that satisfies $\lim_{n\to\infty}P(|\hat{\theta}-\theta| > \varepsilon) = 0$ . Then $\hat{\theta}$ is called a consistent estimation of $\theta$ .

Bias

Assume $\theta\in\Theta$ is an unknown parameter, $\hat{\theta}_n=\hat{\theta}_n(x_1,x_2,...,x_n)$ is an estimation obtained from $n$ samples. If for any $\theta\in\Theta$ there is $E(\hat{\theta})=\theta$ . Then $\hat{\theta}$ is called an unbiased estimation of $\theta$ .

Efficiency

Assume $\hat{\theta}_1$ and $\hat{\theta}_2$ are two unbiased estimation. if for any $\theta\in\Theta$ there is $var(\hat{\theta}_1) \leq var(\hat{\theta}_2)$ , and there exists at least one $\theta\in\Theta$ so that $var(\hat{\theta}_1) < var(\hat{\theta}_2)$ . Then $var(\hat{\theta}_1)$ is more efficient than $var(\hat{\theta}_2)$ .

Example 1

Assume $X_1,X_2,...,X_n$ are samples from uniform distribution $U(0,\theta)$ , the moment estimation is $\hat{\theta}=2\bar{X}$ and the maximum likelihood estimation is $\tilde{\theta}={\displaystyle \max_{1\leq k\leq n} X_k}$ , consider the unbiasedness of $\hat{\theta}$ and $\tilde{\theta}$ .

Solution. Fisrt, $E(\hat{\theta}) = E(2\bar{X}) = 2\times\dfrac{\theta}{2} = \theta$ , hence, $\hat{\theta}$ is an unbiased estimation.

As for $\tilde{\theta}$ , when $0\leq y \leq \theta$ , the distribution function of $y$ is

$F_{\tilde{\theta}}(y) = P(\tilde{\theta}\leq y) = {\displaystyle P( \max_{1\leq k\leq n} X_k\leq y) = \prod_{k=1}^n P(X_k\leq y) = \left(\dfrac{y}{\theta}\right)^n}$

Then, the probability function would be

$f_{\tilde{\theta}}(y) = {\displaystyle \dfrac{n}{\theta^n}y^{n-1}}$

Hence,

$E(\tilde{\theta}) = {\displaystyle \int_{-\infty}^\infty yf_{\tilde{\theta}}(y)dy = \int_{0}^yy\dfrac{n}{\theta^n}y^{n-1}dy} = \dfrac{n}{n+1}\theta$

Hence, $\tilde{\theta}$ is a biased estimation.

Example 2

Assume $X_1,X_2,...,X_n$ are samples from uniform distribution $U(0,\theta)$ , $\hat{\theta}_1=2\bar{X}$ and $\hat{\theta}_2={\displaystyle \dfrac{n+1}{n} \max_{1\leq k\leq n} X_k}$ , compare their efficiency.

Solution. For $\hat{\theta}_1$ , we have

$var(\hat{\theta}_1) = var(2\bar{X}) = 4var(\bar{X}) = 4\cdot\dfrac{\theta^2}{12n} = \dfrac{\theta^2}{3n}$

For $\hat{\theta}_2$ , firstly, from example 1, we have

$E(\tilde{\theta}^2) = {\displaystyle \int_{0}^{\theta}y^2f_{\tilde{\theta}}(y)dy = \int_{0}^\theta y^2\dfrac{n}{\theta^n}y^{n-1}dy = \dfrac{n}{n+2}\theta^2}$

Then, we have

$var(\tilde{\theta}) = E(\tilde{\theta}^2) - E^2(\tilde{\theta}) = \dfrac{n}{n+2}\theta^2-\dfrac{n^2}{(n+ 1)^2}\theta^2 = \dfrac{n}{(n+1)^2(n+ 2)}\theta^2$

Therefore,

$var(\hat{\theta}_2) = \dfrac{(n+1)^2}{n^2}var(\tilde{\theta}) = \dfrac{1}{n(n+2)}\theta^2$

Mean Square Error (MSE)

$E((\hat{\theta}-\theta)^2) = E\left([\hat{\theta} - E(\hat{\theta})] + [E(\hat{\theta}) - \theta]^2\right) = var(\hat{\theta}) + (E(\hat{\theta}) - \theta)^2$

Example 3

Assume $X1, X_2, ..., X_n$ are samples from $N(\mu,\sigma^2)$ , calculate MSE of moment estimation and maximum likelihood estimation of $\mu$ and $\sigma^2$ .

Solution. Fisrtly, moment estimation of $\mu$ and $\sigma^2$ are $\bar{\mu}=\bar{X}$ and $\bar{\sigma}^2=S^2$ , and maximum likelihood estimation of $\mu$ and $\sigma^2$ are $\hat{\mu} = \bar{X}$ and $\hat{\sigma}^2={\displaystyle \dfrac{1}{n}\sum_{i=1}^n(X_i-\bar{X})^2}$ .

For $\bar{\mu}$ and $\bar{\sigma}^2$ ,

$E((\bar{\mu} - \mu)^2) = E((\bar{X} - \mu)^2) = var(\bar{X}) = \dfrac{\sigma^2}{n}$

$E((\bar{\sigma}^2 - \sigma^2)^2) = E((S^2 - \sigma^2)^2) = var(S^2)$

To obtain $var(S^2)$ , we can use the fact that $\dfrac{(n-1)s^2}{\sigma^2}\sim \chi^2(n-1)$ and $var(\chi^2(n-1))=2(n-1)$ , so that

$var(S^2) = \dfrac{\sigma^4}{(n-1)^2}\cdot 2(n-1) = \dfrac{2\sigma^4}{n-1}$

For $\hat{\mu}$ and $\hat{\sigma}^2=\dfrac{n-1}{n}S^2$ ,

$E((\hat{\mu} - \mu)^2) = E((\bar{X} - \mu)^2) = var(\bar{X}) = \dfrac{\sigma^2}{n}$

$E((\hat{\sigma}^2 - \sigma^2)^2) = E((\dfrac{n-1}{n}S^2 - \sigma^2)^2) = E\left(\left(\dfrac{n-1}{n}(S^2-\sigma^2)-\dfrac{1}{n}\sigma^2\right)\right)$

$=\left(\dfrac{n-1}{n}\right)^2E\left((S^2-\sigma^2)^2\right) - 2E\left(\dfrac{n-1}{n}(S^2-\sigma^2)\cdot \dfrac{1}{n}\sigma^2\right) + \dfrac{1}{n^2}\sigma^4$

$=\left(\dfrac{n-1}{n}\right)^2 var(S^2) + \dfrac{1}{n^2}\sigma^4 = \left(\dfrac{n-1}{n}\right)^2 \dfrac{2\sigma^4}{n-1} + \dfrac{1}{n^2}\sigma^4 = \dfrac{2n-1}{n^2}\sigma^4$

Therefore,

$E((\hat{\sigma}^2 - \sigma^2)^2) = \dfrac{2n-1}{n^2}\sigma^4 \leq \dfrac{2\sigma^4}{n-1} = E((\bar{\sigma}^2 - \sigma^2)^2)$

Minimum Variance Unbiased (MVU)

Assume $\hat{\theta}$ is an unbiased estimation of $g(\theta)$ , if for any unbiased estimation $\hat{\theta}_1$ , $var_{\theta}(\hat{\theta})\leq var_{\theta}(\hat{\theta}_1)$ holds for any $\theta$ , then $\hat{\theta}$ is called minimum variance unbiased estimation of $g(\theta)$ .

Cramer-Rao Inequality

Suppose $T = T(x_1,x_2,...,x_n)$ is an unbiased estimator of $g(\theta)$ , the variance of any unbiased estimator is then bounded by

$var(T) \geq \dfrac{(g\text{'}(\theta))^2}{nI(\theta)}$

where

$I(\theta) = E\left(\left(\dfrac{\partial}{\partial\theta}\ln p(x;\theta)\right)^2\right)$

is called fisher information, $n$ is the number of the samples and $p(x;\theta)$ is the probability density function.

Proof. Firstly,

$\int_{-\infty}^\infty p(x_k ; \theta)dx_k = 1 \Rightarrow \dfrac{\partial}{\partial\theta}\int_{-\infty}^\infty p(x_k ; \theta)dx_k = \int_{-\infty}^\infty \dfrac{\partial}{\partial\theta} p(x_k ; \theta)dx_k = 0$

$\int_{-\infty}^\infty \dfrac{\partial}{\partial\theta} p(x_k ; \theta)dx_k = \int_{-\infty}^\infty \left[\dfrac{\partial}{\partial\theta}\ln p(x_k ; \theta)\right] p(x_k ; \theta)dx_k = E\left(\dfrac{\partial}{\partial\theta}\ln p(x_k ; \theta)\right) = 0$

Denoting

$Z = \dfrac{\partial}{\partial\theta}\ln \left(\prod_{k=1}^{n}p(x_k ; \theta)\right) = \sum_{k=1}^{n}\dfrac{\partial}{\partial\theta}\ln p(x_k ; \theta)$

Then

$E(Z) =\sum_{k=1}^{n} E\left(\dfrac{\partial}{\partial\theta}\ln p(x_k ; \theta)\right) = 0$

$E(Z^2) = var(Z) = \sum_{k=1}^{n} var\left(\dfrac{\partial}{\partial\theta}\ln p(x_k ; \theta)\right) = \sum_{k=1}^{n} E\left(\left(\dfrac{\partial}{\partial\theta}\ln p(x_k ; \theta)\right)^2\right)=nI(\theta)$

On the other hand,

$g(\theta) = \int_{-\infty}^\infty\cdots\int_{-\infty}^\infty T(x_1,x_2,...,x_n)\prod_{k=1}^{n}p(x_k; \theta)dx_1dx_2\cdots dx_n$

$g\text{'}(\theta) = \int_{-\infty}^\infty\cdots\int_{-\infty}^\infty T(x_1,x_2,...,x_n)\dfrac{\partial}{\partial \theta}\prod_{k=1}^{n}p(x_k; \theta)dx_1dx_2\cdots dx_n$

$= \int_{-\infty}^\infty\cdots\int_{-\infty}^\infty T(x_1,x_2,...,x_n)\left[\dfrac{\partial}{\partial \theta}\ln \left(\prod_{k=1}^{n}p(x_k; \theta)\right)\right]\prod_{k=1}^{n}p(x_k; \theta)dx_1dx_2\cdots dx_n$

$= E\left(T(x_1,x_2,...,x_n) \dfrac{\partial}{\partial \theta}\ln \left(\prod_{k=1}^{n}p(x_k; \theta)\right) \right) = E(T\cdot Z)$

$g\text{'}(\theta) = E(T\cdot Z) = E\left((T-g(\theta))\cdot Z \right)=cov(T-g(\theta), Z)$

$(g\text{'}(\theta))^2 = \left(cov(T-g(\theta), Z)\right)^2 \leq var(T-g(\theta))var(Z) = var(T)var(Z)$

Therefore,

$var(T) \geq \dfrac{(g\text{'}(\theta))^2}{var(Z)} = \dfrac{(g\text{'}(\theta))^2}{nI(\theta)}$

Example 4

Assume $X1, X_2, ..., X_n$ are samples from exponential distribution $\Exp(\lambda)$ , verify that $g(\lambda) = \dfrac{1}{\lambda}$ is a minimum variance unbiased estimation.

Solution.

$E(\bar(X)) = g(\lambda) = \dfrac{1}{\lambda}, var(\bar{X}) = \dfrac{1}{n\lambda^2}$

$I(\lambda) = E\left(\left(\dfrac{\partial}{\partial \lambda}\ln p(X; \lambda)\right)^2\right) = E\left(\left(\dfrac{\partial}{\partial \lambda}\ln (\lambda e^{-\lambda X})\right)^2\right) = E\left(\left( \dfrac{1}{\lambda} - X\right)^2\right) = \dfrac{1}{\lambda^2}$

Therefore,

$\dfrac{(g\text{'}(\lambda))^2}{nI(\lambda)} = \dfrac{1}{n\lambda^2} = var(\bar{X})$

Example 5

Assume $X1, X_2, ..., X_n$ are samples from Poisson distribution $P(\lambda)$ , verify that $g(\lambda) = \lambda$ is a minimum variance unbiased estimation.

Solution.

$E(\bar(X)) = g(\lambda) = \lambda, var(\bar{X}) = \dfrac{\lambda}{n}$

$I(\lambda) = E\left(\left(\dfrac{\partial}{\partial \lambda}\ln p(X; \lambda)\right)^2\right) = E\left(\left(\dfrac{\partial}{\partial \lambda}\ln \left(e^{-\lambda}\dfrac{\lambda^X}{X!}\right)\right)^2\right) = E\left(\left( \dfrac{X}{\lambda} - 1\right)^2\right) = \dfrac{1}{\lambda}$

Therefore,

$\dfrac{(g\text{'}(\lambda))^2}{nI(\lambda)} = \dfrac{\lambda}{n} = var(\bar{X})$

Example 6

Assume $X1, X_2, ..., X_n$ are samples from normal distribution $N(\mu,\sigma^2)$ , $\sigma^2$ is known, verify that $g(\mu) = \mu$ is a minimum variance unbiased estimation.

Solution.

$E(\bar(X)) = g(\mu) = \mu, var(\bar{X}) = \dfrac{\sigma^2}{n}$

$I(\mu) = E\left(\left(\dfrac{\partial}{\partial \mu}\ln \left( \dfrac{1}{\sqrt{2\pi\sigma}}exp\left[-\dfrac{1}{2\sigma^2}(x-\mu)^2\right]\right)\right)^2\right) = E\left(\left( \dfrac{1}{\sigma^2}(x-\mu)\right)^2\right) = \dfrac{1}{\sigma^2}$

Therefore,

$\dfrac{(g\text{'}(\mu))^2}{nI(\mu)} = \dfrac{\sigma^2}{n} = var(\bar{X})$

Improvement of Unbiased Estimation

Rao-Blackwell Inequality

Assume propability density function of the population is $p(x; \theta)$ , $x_1,x_2,...,x_n$ are samples. $T = T(x_1,x_2,...,x_n)$ is sufficient statistics of $\theta$ , then for any unbiased estimation of $\theta$ $\hat{\theta} = \hat{\theta}(x_1,x_2,...,x_n)$ , $\tilde{\theta}=E(\tilde{\theta}|T)$ is also an unbiased estimation of $\theta$ , and there is $var(\tilde{\theta})\leq var(\hat{\theta})$ .

Sufficient Statistics

Assume $x_1,x_2,...,x_n$ are samples from a population and the distribution function is $F(x; \theta)$ , if given $T$ , the distribution of $x_1,x_2,...,x_n$ is independent from $\theta$ , then $T=T(x_1,x_2,...,x_n)$ is called sufficient statistics of $\theta$ .

Theorem. Assume the distribution function of the population is $F(x; \theta)$ , $x_1,x_2,...,x_n$ are samples from the population, then $T=T(x_1,x_2,...,x_n)$ is sufficient statistics of $\theta$ , if and only if: there exist two functions $g(t,\theta)$ and $h(x_1,x_2,...,x_n)$ , for any $\theta$ and $x_1,x_2,...,x_n$ , there is $f(x_1,x_2,...,x_n; \theta) = g(T(x_1,x_2,...,x_n),\theta)\cdot h(x_1,x_2,...,x_n)$ .

Example 7

Assume $X_1,X_2,...,X_n$ are samples from a normal distribution $N(\mu,\sigma^2)$ . Then, $(\bar{X},S^2)$ is sufficient statistic of $\theta = (\mu,\sigma^2)$ .

Solution.

$p(x_1,x_2,...,x_n; \mu,\sigma^2) = \dfrac{1}{(2\pi)^{n\text{/}2}\sigma^n}\exp\left\{-\dfrac{1}{2\sigma^2}\sum_{i=1}^{n}(x_i-\mu)^2\right\}$

$p(x_1,x_2,...,x_n; \mu,\sigma^2) = \dfrac{1}{(2\pi)^{n\text{/}2}\sigma^n}\exp\left(-\dfrac{1}{2\sigma^2}(\bar{x}-\mu)^2\right)\exp\left(-\dfrac{1}{2\sigma^2}\sum_{i=1}^{n}(x_i-\bar{x})^2\right)$

$p(x_1,x_2,...,x_n; \mu,\sigma^2) = \dfrac{1}{(2\pi)^{n\text{/}2}\sigma^n}\exp\left(-\dfrac{1}{2\sigma^2}(\bar{x}-\mu)^2\right)\exp\left(-\dfrac{(n-1)s^2}{2\sigma^2}\right)$

Example 8

Assume the arrival of the customers every nimute follows Poisson distribution $P(\lambda)$ , $X_1,X_2,...,X_n$ are samples of consecutive $n$ minutes, estimate the probability that no customer comes within one minute $e^{-\lambda}$ .

Solution.

$\theta_0 = \{1, \text{ if } x_1 = 0; 0, \text{otherwise}\}$ is an unbiased estimation of $e^{-\lambda}$

$S_n = \sum_{k=1}^{n} X_k$ is sufficient statistics of $\lambda$

$\theta_1=E(\theta_0|S_n) = \left(1-\dfrac{1}{n}\right)^{S_n}$ is an improvement of unbiased estimation of $e^{-\lambda}$

Twitter Facebook LinkedIn

Chao Huang

Statistics [10]: Evaluation of Point Estimation

Properties of Point Estimation

Consistency

Bias

Efficiency

Example 1

Example 2

Mean Square Error (MSE)

Example 3

Minimum Variance Unbiased (MVU)

Cramer-Rao Inequality

Example 4

Example 5

Example 6

Improvement of Unbiased Estimation

Rao-Blackwell Inequality

Sufficient Statistics

Example 7

Example 8

Table of Contents

Comments