Statistics [24]: Variance Reducing Techniques

Several techniques to reduce variance of the estimation, including antithetic variates, control variates, stratified sampling and importance sampling.

Antithetic Variates

Consider we have an even number of samples, , from . One approach is to generate correlated samples to reduce the variance by cancellations in thier sum. The estimate:

where

The variance ,

Conclusion:

1. When , the variance remains the same;
2. When , the variance decreases;
3. WHen , the variance increases.

Example 1

Estimate .

Assume and are two samples, let the two samaples be and , then

Hence,

Control Variates

Suppose we have a function for which is known. Let

The estimate can be given by

The variance of is given by

Differentiate with respect to

Then,

where

Example 2

Estimate .

Let , there is

Then

Stratified Sampling

Let’s divide the whole space into subspaces, the final results would be the sum of all partial results.

The MC estimate becomes

where is the number of points on , and is the volume of the subspace.

The variance becomes

where

Importance Sampling

The pdf under the integral, , may not be the best pdf for MC integration. In this case, we can use a different and simpler pdf from which we can draw the samples. is called the importance density. Hence,

By generating samples , the estimate becomes

where

is called importance weight.

Notice that , we have

Hence,

where

are the normalized importance weights. We can see that the average sum becomes weighted sum, reflecting the relative importance of the sample (point). This is the basis for particle filtering.

The variance of is given by

where

The equation holds when

Example 3

Estimate .

Let and , then

Table of Contents

• Probability vs Statistics
• Shakespear’s New Poem
• Some Common Discrete Distributions
• Some Common Continuous Distributions
• Statistical Quantities
• Order Statistics
• Multivariate Normal Distributions
• Conditional Distributions and Expectation
• Problem Set [01] - Probabilities
• Parameter Point Estimation
• Evaluation of Point Estimation
• Parameter Interval Estimation
• Problem Set [02] - Parameter Estimation
• Parameter Hypothesis Test
• t Test
• Chi-Squared Test
• Analysis of Variance
• Summary of Statistical Tests
• Python [01] - Data Representation
• Python [02] - t Test & F Test
• Python [03] - Chi-Squared Test
• Experimental Design
• Monte Carlo
• Variance Reducing Techniques
• From Uniform to General Distributions
• Problem Set [03] - Monte Carlo
• Unitary Regression Model
• Multiple Regression Model
• Factor and Principle Component Analysis
• Clustering Analysis
• Summary