Statistics [15]: Analysis of Variance - F test

4 minute read

Published: January 15, 2021

In statistics, analysis of variance (ANOVA) is a collection of statistical models, and their associated procedures, in which the observed variance is partitioned into components due to different explanatory variables. In its simplest form ANOVA gives a statistical test of whether the means of several groups are all equal, and therefore generalizes Student’s two-sample $t$ test to more than two groups.

One-Way ANOVA

The one-way ANOVA compares the means between the groups we are interested in and determines whether any of those means are statistically significantly different from each other. Specifically, it tests the null hypothesis:

$H_0: \mu_1 = \mu_2 = \cdots = \mu_k$

where $\mu = \text{group mean}, k = \text{number of groups}$ .

ANOVA Model

The independent variable (grouping variable) that we are interested is called factor, and each factor may have two or more levels. In the table below, $A_1, A_2, ..., A_r$ are $r$ levels of factor $A$ ; $n_i, i=1,2,...,r$ is number of independent tests under each level; $X_{ij}, i=1,2,...,r, j=1,2,...,n_i$ is the test results.

level \ test number	$1$	$2$	$\cdots$	$n_i$
$A_1$	$X_{11}$	$X_{12}$	$\cdots$	$X_{1n_1}$
$A_2$	$X_{21}$	$X_{22}$	$\cdots$	$X_{2n_2}$
$\vdots$	$\vdots$	$\vdots$	$\ddots$	$\vdots$
$A_r$	$X_{r1}$	$X_{r2}$	$\cdots$	$X_{rn_r}$

Assume

$X_{ij}\sim N(\mu_i, \sigma^2), i=1,2,...,r, j=1,2,...,n_i$

$\varepsilon_{ij} = X_{ij} - \mu_i \sim N(0,\sigma^2), i=1,2,...,r, j=1,2,...,n_i$

$X_{ij} = \mu_i + \varepsilon_{ij}$

Further, let

$n={\displaystyle \sum_{i=1}^r n_i, \mu = \dfrac{1}{n}\sum_{i=1}^rn_i\mu_i} \ \ \text{and} \ \ \alpha_i = \mu_i-\mu, i=1,2,...,r$

We have

$X_{ij}=\mu + \alpha_i + \varepsilon_{ij}, i=1,2,...,r, j=1,2,...,n_i$

$\sum_{i=1}^rn_i\alpha_i = 0$

The hypothesis could be

$H_0: \alpha_1 = \alpha_2 = \cdots = \alpha_r = 0$

ANOVA Problem

The hypothesis test problem:

$H_0: \mu_1 = \mu_2 = \cdots = \mu_r,\ \ H_1: \text{at two of}\ \ \mu_1,\mu_2,...,\mu_r \ \ \text{are not equal}$

The test statistics:

$\bar{X} = {\displaystyle \dfrac{1}{n}\sum_{i=1}^r\sum_{j=1}^{n_i}X_{ij} = \dfrac{1}{n}\sum_{i=1}^rn_i\bar{X}_i}$

$s_T = {\displaystyle \sum_{i=1}^r\sum_{j=1}^{n_i}(X_{ij} - \bar{X})^2}$

Variance Analysis

$s_T = {\displaystyle \sum_{i=1}^r\sum_{j=1}^{n_i}(X_{ij} - \bar{X})^2 = \sum_{i=1}^r\sum_{j=1}^{n_i}\left[(X_{ij} - \bar{X}_i) + (\bar{X}_i - \bar{X})\right]^2}$

${=\displaystyle\sum_{i=1}^r\sum_{j=1}^{n_i}(X_{ij} - \bar{X}_i)^2 + \sum_{i=1}^r\sum_{j=1}^{n_i}(\bar{X}_i - \bar{X})^2 + 2\sum_{i=1}^r\sum_{j=1}^{n_i}(X_{ij} - \bar{X}_i)(\bar{X}_i - \bar{X})}$

${=\displaystyle\sum_{i=1}^r\sum_{j=1}^{n_i}(X_{ij} - \bar{X}_i)^2 + \sum_{i=1}^r\sum_{j=1}^{n_i}(\bar{X}_i - \bar{X})^2}$

${=\displaystyle s_e + s_A}$

where

Within group sum of square: $s_e = \sum_{i=1}^r\sum_{j=1}^{n_i}(X_{ij} - \bar{X}_i)^2$

Between group sum of square: $s_A = {\displaystyle \sum_{i=1}^r\sum_{j=1}^{n_i}(\bar{X}_i - \bar{X})^2 = \sum_{i=1}^rn_i(\bar{X}_i - \bar{X})^2}$

For $s_e$ ,

$\bar{X}_i = \dfrac{1}{n_i}\sum_{j=1}^{n_i}(\mu + \alpha_i + \varepsilon_{ij}) = \mu + \alpha_i + \bar{\varepsilon}_i, \bar{\varepsilon}_i = \dfrac{1}{n}\sum_{j=1}^{n_i}\varepsilon_{ij}$

$s_e = \sum_{i=1}^r\sum_{j=1}^{n_i}(X_{ij} - \bar{X}_i)^2 = \sum_{i=1}^r\sum_{j=1}^{n_i}(\varepsilon_{ij} - \bar{\varepsilon}_i)^2$

$E(s_e) =\sum_{i=1}^rE\left[\sum_{j=1}^{n_i}(\varepsilon_{ij} - \bar{\varepsilon}_i)^2\right] = \sum_{i=1}^r(n_i-1)\sigma^2 = (n-r)\sigma^2$

For $s_A$ ,

$\bar{X} = \dfrac{1}{n}\sum_{i=1}^r\sum_{j=1}^{n_i}(\mu + \alpha_i + \varepsilon_{ij}) = \mu + \bar{\varepsilon}, \bar{\varepsilon} = \dfrac{1}{n}\sum_{i=1}^r\sum_{j=1}^{n_i}\varepsilon_{ij}$

$s_A = \sum_{i=1}^rn_i(\bar{X}_i - \bar{X})^2 = \sum_{i=1}^rn_i(\alpha_i + \bar{\varepsilon}_i - \bar{\varepsilon})$

$E(s_A) =E\left[\sum_{i=1}^rn_i\left(\alpha_i^2 + 2\alpha_i(\bar{\varepsilon}_i - \bar{\varepsilon}) + (\bar{\varepsilon}_i - \bar{\varepsilon})^2\right)\right]$

$= E\left[\sum_{i=1}^rn_i \alpha_i^2\right] + E\left[\sum_{i=1}^rn_i (\bar{\varepsilon}_i - \bar{\varepsilon})^2\right] = \sum_{i=1}^rn_i \alpha_i^2 + (r-1)\sigma^2$

When hypothesis $H_0$ is true,

$E\left(\dfrac{s_A}{r-1}\right) = \sigma^2 \Rightarrow \dfrac{s_A}{\sigma^2}\sim \chi^2(r-1)$

$E\left(\dfrac{s_e}{n-r}\right) = \sigma^2 \Rightarrow \dfrac{s_e}{\sigma^2}\sim \chi^2(n-r)$

Hence,

$\dfrac{s_A/(r-1)}{s_e/(n-r)}\sim F(r-1,n-r)$

The hypothesis is thus rejected for large values of $F$ .

Example 1

Comparison of pesticides.

	$A_1$	$A_2$	$A_3$	$A_4$	$A_5$	$A_6$
$1$	87	90	56	55	92	75
$2$	85	88	62	48	99	72
$3$	80	87			95	81
$4$		94			91

ANOVA

	Sum of Squares	df	Mean Square	F	Sig.
Between Groups	3794.500	5	758.900	51.162	.000
Within Groups	178.000	12	14.833
Total	3972.500	17

The significance is obvious.

Example 2

Lifetime of lightbulbs.

	1	2	3	4	5	6	7	8
$A_1$	1600	1610	1650	1680	1700	1700	1780
$A_2$	1500	1640	1400	1700	1750
$A_3$	1640	1550	1600	1620	1640	1600	1740	1800
$A_4$	1510	1520	1530	1570	1640	1680

ANOVA

	Sum of Squares	df	Mean Square	F	Sig.
Between Groups	39776.456	3	13258.819	1.638	.209
Within Groups	178088.929	22	8089.951
Total	217865.385	25

The significance is not obvious.

Two-Way ANOVA

A two-way ANOVA is used to estimate how the mean of a quantitative variable changes according to the levels of two categorical variables. Use a two-way ANOVA when we want to know how two independent variables, in combination, affect a dependent variable.

ANOVA Model

Assume there are two factors $A$ and $B$ with levels $A_1, A_2, ..., A_r$ and $B_1,B_2,...,B_s$ respectively. Then, there will be $sr$ different combinations $(A_i,B_j)$ . Perform $t$ independent tests under each combination, $srt$ tests in total would be necessary.

$A$ \ $B$	$B_1$	$B_2$	—	$B_s$
$A_1$	$X_{111},X_{112},\cdots,X_{11t}$	$X_{121},X_{122},\cdots,X_{12t}$	$\cdots$	$X_{1s1},X_{1s2},\cdots,X_{1st}$
$A_2$	$X_{211},X_{212},\cdots,X_{21t}$	$X_{221},X_{222},\cdots,X_{22t}$	$\cdots$	$X_{2s1},X_{2s2},\cdots,X_{2st}$
$\vdots$	$\vdots$	$\vdots$	$\ddots$	$\vdots$
$A_r$	$X_{r11},X_{r12},\cdots,X_{r1t}$	$X_{r21},X_{r22},\cdots,X_{r2t}$	$\cdots$	$X_{rs1},X_{rs2},\cdots,X_{rst}$

Assume

$X_{ijk}\sim N(\mu_{ij}, \sigma^2), i=1,2,...,r, j=1,2,...,s, k=1,2,...,t$

$\varepsilon_{ijk} = X_{ijk} - \mu_{ij} \sim N(0,\sigma^2), i=1,2,...,r, j=1,2,...,s, k=1,2,...,t$

$X_{ijk} = \mu_{ij} + \varepsilon_{ijk}$

Let

$\mu = \dfrac{1}{rs}\sum_{i=1}^r\sum_{j=1}^{s}\mu_{ij}$

$\mu_{i\text{*}} = \dfrac{1}{s}\sum_{j=1}^{s}\mu_{ij}, \ \ \alpha_i = \mu_{i\text{*}} - \mu, \ \ i=1,2,...,r$

$\mu_{\text{*}j} = \dfrac{1}{r}\sum_{j=1}^{r}\mu_{ij}, \ \ \beta_j = \mu_{\text{*}j} - \mu, \ \ i=1,2,...,s$

We have

$\sum_{i=1}^r\alpha_i = 0,\ \ \sum_{j=1}^s\beta_{j} = 0, \ \ \gamma_{ij} = (\mu_{ij} - \mu) - \alpha_i - \beta_j$

$\mu_{ij} = \mu + \alpha_i + \beta_j + \gamma_{ij}$

where $\mu_{ij} - \mu$ reflects the total effects of $(A_i,B_j)$ on the experiment results, $\gamma_{ij}$ is called interactive effect and $\alpha_i$ and $\beta_j$ are called main effects.

Further,

$\sum_{i=1}^r\gamma_{ij} = \sum_{j=1}^s\gamma_{ij} = 0$

In summary,

$X_{ijk} = \mu + \alpha_i + \beta_j + \gamma_{ij} + \varepsilon_{ijk}, i=1,2,...,r, j = 1,2,...,s, k=1,2,...,t$

$\sum_{i=1}^r\alpha_i = 0,\ \ \sum_{j=1}^s\beta_{j} = 0, \ \ \sum_{i=1}^r\gamma_{ij} = \sum_{j=1}^s\gamma_{ij} = 0$

$\varepsilon_{ijk}\sim N(0,\sigma^2)$ and all $\varepsilon_{ijk}$ are independent.

Hypothesis

$H_{01}: \alpha_1 = \alpha_2 = \cdots = \alpha_r = 0$

$H_{02}: \beta_1 = \beta_2 = \cdots = \beta_s = 0$

$H_{03}: \gamma_{ij} = 0, i=1,2,...,r, j = 1,2,...,s$

Variance Analysis

$\bar{X} = {\displaystyle \dfrac{1}{rst}\sum_{i=1}^r\sum_{j=1}^s\sum_{k=1}^tX_{ijk}}$

$\bar{X}_{ij\text{*}} = \dfrac{1}{t}\sum_{k=1}^tX_{ijk}, \bar{X}_{i\text{**}} = \dfrac{1}{st}\sum_{j=1}^s\sum_{k=1}^tX_{ijk}, \bar{X}_{\text{*}j\text{*}} = \dfrac{1}{rt}\sum_{i=1}^r\sum_{k=1}^tX_{ijk}$

$s_T = \sum_{i=1}^r\sum_{j=1}^s\sum_{k=1}^t(X_{ijk}-\bar{X})^2 = s_e + s_A + s_B + s_{A\times B}$

where

$s_e = \sum_{i=1}^r\sum_{j=1}^s\sum_{k=1}^t(X_{ijk}-\bar{X}_{ij\text{*}})^2$

$s_A = st\sum_{i=1}^r(\bar{X}_{i\text{*}\text{*}}-\bar{X})^2$

$s_B = rt\sum_{i=1}^r(\bar{X}_{\text{*}j\text{*}}-\bar{X})^2$

$s_{A\times B} = t\sum_{i=1}^r\sum_{j=1}^s(\bar{X}_{ij\text{*}}-\bar{X}_{i\text{*}\text{*}} - \bar{X}_{\text{*}j\text{*}} + \bar{X})^2$

Let

$\bar{\varepsilon} = {\displaystyle \dfrac{1}{rst}\sum_{i=1}^r\sum_{j=1}^s\sum_{k=1}^t\varepsilon_{ijk}}$

$\bar{\varepsilon}_{ij\text{*}} = \dfrac{1}{t}\sum_{k=1}^t\varepsilon_{ijk}, \bar{\varepsilon}_{i\text{**}} = \dfrac{1}{st}\sum_{j=1}^s\sum_{k=1}^t\varepsilon_{ijk}, \bar{\varepsilon}_{\text{*}j\text{*}} = \dfrac{1}{rt}\sum_{i=1}^r\sum_{k=1}^t\varepsilon_{ijk}$

Then,

$\bar{X}_{ij\text{*}} = \mu + \alpha_i + \beta_j + \gamma_{ij} + \bar{\varepsilon}_{ij\text{*}}, \bar{X} = \mu + \bar{\varepsilon}$

$\bar{X}_{i\text{**}} = \mu + \alpha_i + \bar{\varepsilon}_{i\text{**}}, \bar{X}_{\text{*}j\text{*}}=\mu + \beta_j + \bar{\varepsilon}_{\text{*}j\text{*}}$

$s_e = \sum_{i=1}^r\sum_{j=1}^s\sum_{k=1}^t(\varepsilon_{ijk}-\bar{\varepsilon}_{ij\text{*}})^2$

$s_A = st\sum_{i=1}^r(\alpha_i + \bar{\varepsilon}_{i\text{*}\text{*}}-\bar{\varepsilon})^2$

$s_B = rt\sum_{i=1}^r(\beta_j + \bar{\varepsilon}_{\text{*}j\text{*}}-\bar{\varepsilon})^2$

$s_{A\times B} = t\sum_{i=1}^r\sum_{j=1}^s(\gamma_{ij} + \bar{\varepsilon}_{ij\text{*}}-\bar{\varepsilon}_{i\text{*}\text{*}} - \bar{\varepsilon}_{\text{*}j\text{*}} + \bar{\varepsilon})^2$

The expectation would be

$E(s_e) = rs(t-1)\sigma^2$

$E(s_A) = (r-1)\sigma^2 + st\sum_{i=1}^r\alpha_i^2$

$E(s_B) = (s-1)\sigma^2 + rt\sum_{j=1}^s\beta_j^2$

$E(s_{A\times B}) = (r-1)(s-1)\sigma^2 + t\sum_{i=1}^r\sum_{j=1}^s\gamma_{ij}^2$

Summary,

Source	Sum of Square	DOF	Mean Square	$F$ value
$A$	$s_A$	$r-1$	$\bar{s}_A = s_A/(r-1)$	$F_A = \bar{s}_A/\bar{s}_e$
$B$	$s_B$	$s-1$	$\bar{s}_B = s_B/(s-1)$	$F_B = \bar{s}_B/\bar{s}_e$
$A\times B$	$s_{A\times B}$	$(r-1)(s-1)$	$\bar{s}_{A\times B} = s_{A\times B}/((r-1)(s-1))$	$F_{A\times B} = \bar{s}_{A\times B}/\bar{s}_e$
$e$	$s_e$	$rs(t-1)$	$\bar{s}_e = s_e/(rs(t-1))$
$T$	$s_T$	$rst-1$