Statistics [09]: Parameter Point Estimation

1 minute read

Published: January 09, 2021

Assume $X_1,X_2,...,X_n$ are samples from a population, point estimation refers to constructing certain statistical quantities $\hat{\theta} = \theta(X_1,X_2,...,X_n)$ that can be used to estimate the distribution of the population. The method is not unique, the most commonly used methods are the method of moments and the method of maximum likelihood.

Method of Moments

The basic idea is to equate sample moments with theoretical moments, where the moments can be raw moments or central moments or variance.

Example 1

Assume $X_1,X_2,...,X_n$ are samples from uniform distribution $X\sim U(-a,a)$ , estimate $a$ using the method of moments.

Solution 1.

$E(X)=0,\ \ var(X) = \dfrac{a^2}{3} = s^2 \Rightarrow \hat{a} = \sqrt{3}s$

Solution 2.

$E(X)=0,\ \ E(X^2) = \dfrac{a^2}{3} = \dfrac{X_1^2 + X_2^2 + ... + X_n^2 }{n}$

$\hat{a} = \sqrt{3}\sqrt{\dfrac{X_1^2 + X_2^2 + ... + X_n^2 }{n}}$

Example 2

Assume $X_1,X_2,...,X_m$ are samples from binomial distribution $X\sim b(n,p)$ , estimate $n,p$ using the method of moments.

Solution 1.

$E(X)=np = \bar{x},\ \ var(X) = np(1-p) = s^2$

$\hat{p} = 1 - \dfrac{s^2}{\bar{x}},\ \ \hat{n} = \dfrac{\bar{x}^2}{\bar{x}-s^2}$

Solution 2.

$E(X)=np = \bar{x},\ \ E(X^2) = np(1-p) + n^2p^2 = \dfrac{X_1^2 + X_2^2 + ... + X_m^2 }{m} = \bar{x}(1-p) + \bar{x}^2$

$\hat{p} = 1 + \bar{x} - \dfrac{X_1^2 + X_2^2 + ... + X_m^2 }{m\bar{x}},\ \ \hat{n} = \dfrac{m\bar{x}^2}{m\bar{x} + m\bar{x}^2 - X_1^2 + X_2^2 + ... + X_m^2 }$

Method of Maximum Likelihood

The abasic idea of maximum likelihood is to estimate the unknown parameters $\theta$ that would maximize the probability of getting the data we have observed.

Assume we have a random sample $X_1,X_2,...,X_n$ , each with a probability function $p(x_i; \theta)$ . Then, the likelihood function can be expressed as

$L(\theta; x_1,x_2,...,x_n) = {\displaystyle \prod_{k=1}^n p(x_k; \theta)}$

If $\hat{\theta} = \hat{\theta}(x_1,x_2,...,x_n)$ satisfies $L(\hat{\theta}) = {\displaystyle \max_{\theta\in\Theta}L(\theta)}$ , then $\hat{\theta}$ is called maximum likelihood estimation (MLE).

In practice, we often use the log form of the likelihood function, then it becomes

$\ln(L(\theta; x_1,x_2,...,x_n)) = {\displaystyle \sum_{k=1}^n \ln(p(x_k; \theta))}$

Example 3

Assume $x_1,x_2,...,x_n$ are samples from exponential distribution $\Exp(\lambda)$ , find $\lambda$ .

Solution.

$L(\lambda; x_1,x_2,...,x_n) = {\displaystyle \prod_{k=1}^n \lambda\exp(-\lambda x_i) = \lambda^n\exp(-\lambda(x_1+x_2+ ... x_n))}$ .

$\ln(L(\lambda; x_1,x_2,...,x_n)) = {\displaystyle n\ln\lambda - \lambda(x_1+x_2+ ... x_n)}$

$\max\ln L(\lambda)\Rightarrow {\displaystyle \dfrac{d\ln L(\lambda)}{d\lambda}= \dfrac{n}{\lambda} - (x_1+x_2+ ... x_n) = 0}$

$\hat{\lambda} = \dfrac{1}{(x_1+x_2+ ... x_n)/n} \Rightarrow \hat{\lambda} = \dfrac{1}{\bar{x}}$

Example 4

Assume $x_1,x_2,...,x_n$ are samples from Poisson distribution $P(\lambda)$ , find $\lambda$ .

Solution.

$L(\lambda; x_1,x_2,...,x_n) = {\displaystyle \prod_{k=1}^n \dfrac{\lambda^{x_k}}{x_k!}e^{-\lambda} = \dfrac{1}{x_1!x_2!...x_n!}\lambda^{x_1+x_2+ ... x_n}e^{-n\lambda}}$ .

$\ln L(\lambda) = {\displaystyle C+ (x_1+x_2+ ... x_n)\ln\lambda - n\lambda}$

$\max\ln L(\lambda)\Rightarrow {\displaystyle \dfrac{d\ln L(\lambda)}{d\lambda}= \dfrac{1}{\lambda}(x_1+x_2+ ... x_n) - n = 0}$

$\hat{\lambda} = \dfrac{x_1+x_2+ ... x_n}{n}$

Example 5

Assume $x_1,x_2,...,x_n$ are samples from normal distribution $N(\mu,\sigma^2)$ , find $\mu, \sigma^2$ .

Solution.

$L(\mu,\sigma^2; x_1,x_2,...,x_n) = {\displaystyle \prod_{k=1}^n \dfrac{1}{\sqrt{2\pi}\sigma}\exp{\left(-\dfrac{(x_k-\mu)^2}{2\sigma^2}\right)} = \left(\dfrac{1}{\sqrt{2\pi}\sigma}\right)^n\exp\left(-\dfrac{1}{2\sigma^2}\sum_{i=1}^n(x_i-\mu)^2\right)}$ .

$\ln L(\mu,\sigma^2) = {\displaystyle n\ln\left(\dfrac{1}{\sqrt{2\pi}}\right) - \dfrac{n}{2}\ln\sigma^2 - \dfrac{1}{2\sigma^2}\sum_{i=1}^n(x_i-\mu)^2}$

For $\mu$ ,

$\dfrac{\partial\ln L(\mu,\sigma^2)}{\partial\mu}= \dfrac{1}{\sigma^2}\sum_{i=1}^n(x_i-\mu) = 0 \Rightarrow \hat{\mu} = \dfrac{1}{n}\sum_{i=1}^n x_i = \bar{x}$

For $\sigma^2$ ,

$\dfrac{\partial\ln L(\mu,\sigma^2)}{\partial\sigma^2}= -\dfrac{n}{2}\dfrac{1}{\sigma^2} + \dfrac{1}{2(\sigma^2)^2}\sum_{i=1}^n(x_i-\mu)^2 = 0 \Rightarrow \hat{\sigma}^2 = \dfrac{1}{n}\sum_{i=1}^n (x_i - \mu)^2 = \dfrac{1}{n}\sum_{i=1}^n (x_i - \bar{x})^2$

Twitter Facebook LinkedIn

Chao Huang

Statistics [09]: Parameter Point Estimation

Method of Moments

Example 1

Example 2

Method of Maximum Likelihood

Example 3

Example 4

Example 5

Table of Contents

Comments