Minimax estimator

In statistical decision theory, where we are faced with the problem of estimating a deterministic parameter (vector) θ ∈ Θ {displaystyle heta in Theta } from observations x ∈ X , {displaystyle xin {mathcal {X}},} an estimator (estimation rule) δ M {displaystyle delta ^{M},!} is called minimax if its maximal risk is minimal among all estimators of θ {displaystyle heta ,!} . In a sense this means that δ M {displaystyle delta ^{M},!} is an estimator which performs best in the worst possible case allowed in the problem. In statistical decision theory, where we are faced with the problem of estimating a deterministic parameter (vector) θ ∈ Θ {displaystyle heta in Theta } from observations x ∈ X , {displaystyle xin {mathcal {X}},} an estimator (estimation rule) δ M {displaystyle delta ^{M},!} is called minimax if its maximal risk is minimal among all estimators of θ {displaystyle heta ,!} . In a sense this means that δ M {displaystyle delta ^{M},!} is an estimator which performs best in the worst possible case allowed in the problem. Consider the problem of estimating a deterministic (not Bayesian) parameter θ ∈ Θ {displaystyle heta in Theta } from noisy or corrupt data x ∈ X {displaystyle xin {mathcal {X}}} related through the conditional probability distribution P ( x | θ ) {displaystyle P(x| heta ),!} . Our goal is to find a 'good' estimator δ ( x ) {displaystyle delta (x),!} for estimating the parameter θ {displaystyle heta ,!} , which minimizes some given risk function R ( θ , δ ) {displaystyle R( heta ,delta ),!} . Here the risk function is the expectation of some loss function L ( θ , δ ) {displaystyle L( heta ,delta ),!} with respect to P ( x | θ ) {displaystyle P(x| heta ),!} . A popular example for a loss function is the squared error loss L ( θ , δ ) = ‖ θ − δ ‖ 2 {displaystyle L( heta ,delta )=| heta -delta |^{2},!} , and the risk function for this loss is the mean squared error (MSE). Unfortunately in general the risk cannot be minimized, since it depends on the unknown parameter θ {displaystyle heta ,!} itself (If we knew what was the actual value of θ {displaystyle heta ,!} , we wouldn't need to estimate it). Therefore additional criteria for finding an optimal estimator in some sense are required. One such criterion is the minimax criterion. Definition : An estimator δ M : X → Θ {displaystyle delta ^{M}:{mathcal {X}} ightarrow Theta ,!} is called minimax with respect to a risk function R ( θ , δ ) {displaystyle R( heta ,delta ),!} if it achieves the smallest maximum risk among all estimators, meaning it satisfies Logically, an estimator is minimax when it is the best in the worst case. Continuing this logic, a minimax estimator should be a Bayes estimator with respect to a prior least favorable distribution of θ {displaystyle heta ,!} . To demonstrate this notion denote the average risk of the Bayes estimator δ π {displaystyle delta _{pi },!} with respect to a prior distribution π {displaystyle pi ,!} as Definition: A prior distribution π {displaystyle pi ,!} is called least favorable if for any other distribution π ′ {displaystyle pi ',!} the average risk satisfies r π ≥ r π ′ {displaystyle r_{pi }geq r_{pi '},} . Theorem 1: If r π = sup θ R ( θ , δ π ) , {displaystyle r_{pi }=sup _{ heta }R( heta ,delta _{pi }),,} then: Corollary: If a Bayes estimator has constant risk, it is minimax. Note that this is not a necessary condition. Example 1, Unfair coin: Consider the problem of estimating the 'success' rate of a Binomial variable, x ∼ B ( n , θ ) {displaystyle xsim B(n, heta ),!} . This may be viewed as estimating the rate at which an unfair coin falls on 'heads' or 'tails'. In this case the Bayes estimator with respect to a Beta-distributed prior, θ ∼ Beta ( n / 2 , n / 2 ) {displaystyle heta sim { ext{Beta}}({sqrt {n}}/2,{sqrt {n}}/2),} is