Statistics [11]: Parameter Interval Estimation

3 minute read

Published: January 11, 2021

Point estimation refers to constructing certain statistical quantities using a point, and interval estimation is to estimate the unknown parameters using an interval.

Definition

Assume $x_1,x_2,...,x_n$ are samples from $X\sim F(x; \theta)$ , $\theta$ is an unknown parameter. $I(x_1,x_2,...,x_n)$ is a random interval, which is determined by the samples. Interval estimation of $\theta$ with a confidence level $1-\alpha (0<\alpha<1)$ refres to

$P_{\theta}(\theta\in I(x_1,x_2,...,x_n)) \geq 1 - \alpha,\ \ \forall \theta\in\Theta$

where

$I = \lfloor\hat{\theta}_1(x_1,x_2,...,x_n),\hat{\theta}_2(x_1,x_2,...,x_n)\rceil$ is a two-sided confidence interval; $I = \lfloor\hat{\theta}_L(x_1,x_2,...,x_n),\infty\rceil$ and $I = \lfloor-\infty,\hat{\theta}_U(x_1,x_2,...,x_n)\rceil$ are one-sided confidence intervals.

Pivot Method

Construct a pivot, $G(x_1,x_2,...,x_n,\theta)$ , which is determined by samples and the unknown parameters completely and the distribution of $G$ doesn’t rely on the unknown parameters.
Choose two constants $c,d$ , for the given $\alpha (0 < \alpha < 1)$ , there is $P(c\leq G\leq d)\geq 1-\alpha$
Solve $c\leq G(x_1,x_2,...,x_n,\theta)\leq d$ , yielding $\hat{\theta}_1(x_1,x_2,...,x_n)\leq\theta\leq\hat{\theta}_2(x_1,x_2,...,x_n)$ .

Example 1

Assume $x_1,x_2,...,x_n$ are samples from normal distribution $N(\mu,\sigma^2)$ , $\sigma^2$ is known and $\mu$ is unknown, find the confidence interval of $\mu$ with confidence level $1-\alpha$ .

Solution. Firstly, the point estimation of $\mu$ is $\bar{x}\sim N\left(\mu,\dfrac{\sigma^2}{n}\right)$ , the pivot can be chosen as

$Z = \dfrac{\bar{x}-\mu}{\sqrt{\sigma^2/n}} = \dfrac{\sqrt{n}(\bar{x}-\mu)}{\sigma} \sim N(0,1)$

Hence,

$\Phi(d)-\Phi(c) = P\left( c\leq \dfrac{\sqrt{n}(\bar{x}-\mu)}{\sigma}\leq d\right) = 1 - \alpha$

Next, determine $c$ and $d$ that minimize the length of the confidence interval.

Assume the distribution function of the pivot is $F(x)$ and the probability density function is $p(x)$ , the problem becomes an optimization problem with constraint $F(d) - F(c) = 1-\alpha$ .

To solve this optimization problem, we can use the method of Lagrange multipliers, let the Lagrange multiplier be $\lambda$ , then

$L(c,d; \lambda) = d - c + \lambda(F(d)-F(c)-1+\alpha)$

Differentiate $L(c,d; \lambda)$ with respect to $c, d$ and $\lambda$ respectively, yielding

$\dfrac{\partial L(c,d; \lambda)}{\partial c} = -1 - \lambda p(c) = 0$

$\dfrac{\partial L(c,d; \lambda)}{\partial d} = 1 + \lambda p(d) = 0$

$\dfrac{\partial L(c,d; \lambda)}{\partial \lambda} = F(d)-F(c)-1+\alpha = 0$

Solving these equations, we have $p(c)=p(d)$ and $\Phi(d)-\Phi(c)=1-\alpha$ . Hence,

$\Phi(d) = 1 - \dfrac{\alpha}{2}$ and $\Phi(c) = \dfrac{\alpha}{2} \Rightarrow d = \Phi^{-1}\left(1 - \dfrac{\alpha}{2}\right), c = \Phi^{-1}\left(\dfrac{\alpha}{2}\right)$

Therefore, $c$ and $d$ are the $\dfrac{\alpha}{2}$ qunatile and $1-\dfrac{\alpha}{2}$ quantile of the standard normal distribution respectively.

Lower and Upper Quantiles

Assume $X$ is a continuous random variable, if $P(X\leq a) = \alpha$ , then $\alpha$ is called the lower $\alpha$ quantile.

$\alpha$ quantile of standard normal distribution: $u_\alpha$
$\alpha$ quantile of $\chi^2$ with $n$ degree of freedom: $\chi^2_{\alpha}(n)$
$\alpha$ quantile of $t$ distribution with $n$ degree of freedom: $t_\alpha(n)$
$\alpha$ quantile of $F$ distribution with $(m,n)$ degree of freedom: $F_\alpha(m,n)$

Assume $X$ is a continuous random variable, if $P(X\geq a) = \alpha$ , then $\alpha$ is called the upper $\alpha$ quantile.

Example 2

Assume $X\sim N(\mu,\sigma^2)$ , $\sigma^2$ is unknown, find the $1-\alpha$ confidence interval of $\mu$ .

Solution. Firstly, we have $X_1 = \dfrac{\bar{x}-\mu}{\sigma/\sqrt{n}}\sim N(0,1)$ and $X_2 = \dfrac{(n-1)s^2}{\sigma^2}\sim \chi^2(n-1)$ , hence

$\dfrac{X_1}{\sqrt{X_2/{n-1}}} = \dfrac{\bar{x}-\mu}{s/\sqrt{n}} = \dfrac{\sqrt{n}(\bar{x} - \mu)}{s} \sim t(n-1)$

Therefore,

$P\left(t_{\alpha\text{/}2}(n-1)\leq \dfrac{\sqrt{n}(\bar{x}-\mu)}{s}\leq t_{1-\alpha\text{/}2}(n-1)\right) = 1 - \alpha$

Using the lookup table, we have

$\bar{x} - \dfrac{s}{\sqrt{n}}t_{1-\alpha\text{/}2}(n-1)\leq \mu \leq \bar{x} + \dfrac{s}{\sqrt{n}}t_{1-\alpha\text{/}2}(n-1)$

Example 3

Assume $X\sim N(\mu,\sigma^2)$ , $\sigma^2$ and $\mu$ are unknown, find the $1-\alpha$ confidence interval of $\sigma^2$ .

SOlution. Firstly, we have

$\dfrac{(n-1)s^2}{\sigma^2}\sim \chi^2(n-1)$

which can be chosen as the pivot. Then,

$P\left(\chi^2_{\alpha\text{/}2}\leq \dfrac{(n-1)s^2}{\sigma^2}\leq \chi^2_{1-\alpha\text{/}2}\right) = 1 - \alpha$

Therefore,

$\sigma^2 \in \left[\dfrac{(n-1)s^2}{\chi^2_{1-\alpha\text{/}2}}, \dfrac{(n-1)s^2}{\chi^2_{\alpha\text{/}2}}\right]$

Example 4

Assume $x_1,x_2,...,x_n$ are samples from $\Exp(\lambda)$ , find the confidence interval of $\lambda$ .

Solution. Firstly, we can proof that

$X = 2\lambda(x_1 + x_2 + \cdots + x_n)\sim \chi^2(2n) \Rightarrow 2n\lambda\bar{x}\sim \chi^2(2n)$

Let the random variable $X$ has the $\chi^2$ distribution with $n$ degrees of freedom with probability density function

$f_X(x) = \dfrac{1}{2^{n\text{/}2}\Gamma(n\text{/}2)}x^{n\text{/}2-1}e^{-x\text{/}2}, x>0$

and random variable $Y$ have the exponential distribution with mean $\lambda$ with probability density function

$f_{Y}(y) = \lambda e^{-\lambda x}, y > 0$

For $n=2$ ,

$f_X(x) = \dfrac{1}{2^{2\text{/}2}\Gamma(2\text{/}2)}x^{2\text{/}2-1}e^{-x\text{/}2} = \dfrac{1}{2}e^{-x\text{/}2}, x>0$

For $\lambda = \dfrac{1}{2}$ ,

$f_{Y}(y) = \dfrac{1}{2} e^{-x\text{/}2}, y > 0$

Thus, $f_X(x) = f_{Y}(y)$ for $n=2, \lambda = \dfrac{1}{2}$ .

With this relation, we have

$P\left(\chi^2_{\alpha\text{/}2}(2n)\leq 2n\lambda\bar{x} \leq \chi^2_{1-\alpha\text{/}2}(2n)\right) = 1 - \alpha$

Therefore,

$\lambda \in \left[\dfrac{\chi^2_{\alpha\text{/}2}(2n)}{2n\bar{x}}, \dfrac{\chi^2_{1-\alpha\text{/}2}(2n)}{2n\bar{x}}\right]$

Large Sample Interval Estimation

When the samples are large enough, the confidence interval can be constructed using asymptotic distribution.

Example 5

Assume $x_1,x_2,...,x_n$ are samples from $b(1,p)$ , find the confidence interval of $p$ .

Solution. The expectation and variance of the samples are respectively

$E(\bar{X}) = p, var(\bar{X}) = \dfrac{p(1-p)}{n}$

According to the central-limit theorem, when $n$ is large enough, there exists asymptotic distribution

$\bar{X} \sim N\left(p,\dfrac{p(1-p)}{n}\right)$

which can be standardized to get the pivot

$\dfrac{\bar{X}-p}{\sqrt{\dfrac{p(1-p)}{n}}}\sim N(0,1)$

Hence,

$P\left(\left|\dfrac{\bar{X}-p}{\sqrt{p(1-p)\text{/}n}}\right|\leq u_{1-\alpha\text{/}2}\right)\approx 1 - \alpha \Rightarrow (\bar{X}-p)^2\leq u^2_{1-\alpha\text{/}2}\dfrac{p(1-p)}{n}$

Solve this inequality, we have

$\dfrac{1}{1+ c}\left(\bar{X} + \dfrac{c}{2} - \sqrt{\dfrac{\bar{X}(1-\bar{X})}{n}u^2_{1-\alpha\text{/}2}+ \dfrac{c^2}{4}}\right)\leq p \leq \dfrac{1}{1+ c}\left(\bar{X} + \dfrac{c}{2} + \sqrt{\dfrac{\bar{X}(1-\bar{X})}{n}u^2_{1-\alpha\text{/}2}+ \dfrac{c^2}{4}}\right)$

where $c = u^2_{1-\alpha\text{/}2}\text{/}n$ , when $n$ is large, $c$ is very small, therefore, the $1-\alpha$ confidence level of $p$ can be estimated as

$\left[\bar{X} - \sqrt{\dfrac{\bar{X}(1-\bar{X})}{n}u^2_{1-\alpha\text{/}2}}, \bar{X} + \sqrt{\dfrac{\bar{X}(1-\bar{X})}{n}u^2_{1-\alpha\text{/}2}}\right]$

Twitter Facebook LinkedIn

Chao Huang

Statistics [11]: Parameter Interval Estimation

Definition

Pivot Method

Example 1

Lower and Upper Quantiles

Example 2

Example 3

Example 4

Large Sample Interval Estimation

Example 5

Table of Contents

Comments