James–Stein estimator

The James–Stein estimator is a biased estimator of the mean of Gaussian random vectors. It can be shown that the James–Stein estimator dominates the 'ordinary' least squares approach, i.e., it has lower mean squared error. It is the best-known example of Stein's phenomenon. The James–Stein estimator is a biased estimator of the mean of Gaussian random vectors. It can be shown that the James–Stein estimator dominates the 'ordinary' least squares approach, i.e., it has lower mean squared error. It is the best-known example of Stein's phenomenon. An earlier version of the estimator was developed by Charles Stein in 1956, and is sometimes referred to as Stein's estimator. The result was improved by Willard James and Charles Stein in 1961. Suppose the vector θ {displaystyle {oldsymbol { heta }}} is the unknown mean of a m {displaystyle m} -variate normally distributed (with known covariance matrix σ 2 I {displaystyle sigma ^{2}I} ) random variable Y {displaystyle {mathbf {Y} }} : We are interested in obtaining an estimate θ ^ {displaystyle {widehat {oldsymbol { heta }}}} of θ {displaystyle {oldsymbol { heta }}} , based on a single observation, y {displaystyle {mathbf {y} }} , of Y {displaystyle {mathbf {Y} }} . This is an everyday situation in which a set of parameters is measured, and the measurements are corrupted by independent Gaussian noise. Since the noise has zero mean, it is very reasonable to use the measurements themselves as an estimate of the parameters. This is the approach of the least squares estimator, which is θ ^ L S = y {displaystyle {widehat {oldsymbol { heta }}}_{LS}={mathbf {y} }} . As a result, there was considerable shock and disbelief when Stein demonstrated that, in terms of mean squared error E ⁡ [ ‖ θ − θ ^ ‖ 2 ] {displaystyle operatorname {E} left} , this approach is suboptimal. The result became known as Stein's phenomenon. If σ 2 {displaystyle sigma ^{2}} is known, the James–Stein estimator is given by James and Stein showed that the above estimator dominates θ ^ L S {displaystyle {widehat {oldsymbol { heta }}}_{LS}} for any m ≥ 3 {displaystyle mgeq 3} , meaning that the James–Stein estimator always achieves lower mean squared error (MSE) than the maximum likelihood estimator. By definition, this makes the least squares estimator inadmissible when m ≥ 3 {displaystyle mgeq 3} . Notice that if ( m − 2 ) σ 2 < ‖ y ‖ 2 {displaystyle (m-2)sigma ^{2}<|{mathbf {y} }|^{2}} then this estimator simply takes the natural estimator y {displaystyle mathbf {y} } and shrinks it towards the origin 0. In fact this is not the only direction of shrinkage that works. Let ν be an arbitrary fixed vector of length m {displaystyle m} . Then there exists an estimator of the James-Stein type that shrinks toward ν, namely