Improved Stein-type shrinkage estimators for the high-dimensional multivariate normal covariance matrix
97
Citation
29
Reference
10
Related Paper
Citation Trend
Keywords:
Scatter matrix
Shrinkage estimator
Matrix (chemical analysis)
Abstract Data analysis in cosmology requires reliable covariance matrices. Covariance matrices derived from numerical simulations often require a very large number of realizations to be accurate. When a theoretical model for the covariance matrix exists, the parameters of the model can often be fit with many fewer simulations. We write a likelihood-based method for performing such a fit. We demonstrate how a model covariance matrix can be tested by examining the appropriate χ 2 distributions from simulations. We show that if model covariance has amplitude freedom, the expectation value of second moment of χ 2 distribution with a wrong covariance matrix will always be larger than one using the true covariance matrix. By combining these steps together, we provide a way of producing reliable covariances without ever requiring running a large number of simulations. We demonstrate our method on two examples. First, we measure the two-point correlation function of halos from a large set of 10000 mock halo catalogs. We build a model covariance with 2 free parameters, which we fit using our procedure. The resulting best-fit model covariance obtained from just 100 simulation realizations proves to be as reliable as the numerical covariance matrix built from the full 10000 set. We also test our method on a setup where the covariance matrix is large by measuring the halo bispectrum for thousands of triangles for the same set of mocks. We build a block diagonal model covariance with 2 free parameters as an improvement over the diagonal Gaussian covariance. Our model covariance passes the χ 2 test only partially in this case, signaling that the model is insufficient even using free parameters, but significantly improves over the Gaussian one.
Matérn covariance function
CMA-ES
Scatter matrix
Covariance mapping
Cite
Citations (7)
Matérn covariance function
Covariance mapping
Scatter matrix
Cite
Citations (2)
Some patterned covariance matrices used to model multivariate normal data that do not have explicit maximum likelihood estimates can be viewed as submatrices of larger patterned covariance matrices that do have explicit maximum likelihood estimates. In such cases, the smaller covariance matrix can be viewed as the covariance matrix for observed variables and the larger covariance matrix can be viewed as the covariance matrix for both observed and missing variables. The advantage of this perspective is that the em algorithm can be used to calculate the desired maximum likelihood estimates for the original problem. Two examples are presented.
CMA-ES
Matérn covariance function
Scatter matrix
Block matrix
Restricted maximum likelihood
Matrix (chemical analysis)
Covariance mapping
Cite
Citations (57)
Classical multivariate methods are often based on the sample covariance matrix, which is very sensitive to outlying observations. One alternative to the covariance matrix is the a ne equivariant rank covariance matrix (RCM) that has been studied for example in Visuri et al. (2003). In this article we assume that the covariance matrix is partially known and study how to estimate the corresponding RCM. We use the properties that the RCM is a ne equivariant and that the RCM is proportional to the inverse of the regular covariance matrix, and reduce the problem of estimating the RCM to estimating marginal rank covariance matrices. This is a great advantage when the dimension of the original data vectors is large.
Matérn covariance function
Scatter matrix
CMA-ES
Rank (graph theory)
Matrix (chemical analysis)
Equivariant map
Cite
Citations (0)
Data analysis in cosmology requires reliable covariance matrices. Covariance matrices derived from numerical simulations often require a very large number of realizations to be accurate. When a theoretical model for the covariance matrix exists, the parameters of the model can often be fit with many fewer simulations. We write a likelihood-based method for performing such a fit. We demonstrate how a model covariance matrix can be tested by examining the appropriate $χ^2$ distributions from simulations. We show that if model covariance has amplitude freedom, the expectation value of second moment of $χ^2$ distribution with a wrong covariance matrix will always be larger than one using the true covariance matrix. By combining these steps together, we provide a way of producing reliable covariances without ever requiring running a large number of simulations. We demonstrate our method on two examples. First, we measure the two-point correlation function of halos from a large set of $10000$ mock halo catalogs. We build a model covariance with $2$ free parameters, which we fit using our procedure. The resulting best-fit model covariance obtained from just $100$ simulation realizations proves to be as reliable as the numerical covariance matrix built from the full $10000$ set. We also test our method on a setup where the covariance matrix is large by measuring the halo bispectrum for thousands of triangles for the same set of mocks. We build a block diagonal model covariance with $2$ free parameters as an improvement over the diagonal Gaussian covariance. Our model covariance passes the $χ^2$ test only partially in this case, signaling that the model is insufficient even using free parameters, but significantly improves over the Gaussian one.
Matrix (chemical analysis)
Cite
Citations (1)
The estimation of parameters of a multivariate p-dimensional random vector is considered for a banded covariance structure under the constrain that the covariances σij = 0 for |i − j| > 1. Explicit analytical estimators for the mean and the covariance matrix are presented. The estimators are unbiased and consistent for the mean and consistent for the covariance matrix. Likelihood based tests which are asymptotically equivalent to likelihood ratio tests are presented and hypotheses for covariance matrices are tested.
Scatter matrix
Covariance mapping
Matérn covariance function
Cite
Citations (2)
Abstract Estimation of covariance components in the multivariate random-effect model with nested covariance structure is discussed. There are two covariance matrices to be estimated, namely, the between-group and the within-group covariance matrices. These two covariance matrices are most often estimated by forming a multivariate analysis of variance and equating mean square matrices to their expectations. Such a procedure involves taking the difference between the between-group mean square and the within-group mean square matrices, and often produces an estimated between-group covariance matrix that is not nonnegative definite. We present estimators of the two covariance matrices that are always proper covariance matrices. The estimators are the restricted maximum likelihood estimators if the random effects are normally distributed. The estimation procedure is extended to more complicated models, including the twofold nested and the mixed-effect models. A numerical example is presented to illustrate the use of the estimation procedure.
Matérn covariance function
Scatter matrix
Cite
Citations (86)
Scatter matrix
Shrinkage estimator
Matrix (chemical analysis)
Cite
Citations (97)
When inferring parameters from a Gaussian-distributed data set by computing a likelihood, a covariance matrix is needed that describes the data errors and their correlations. If the covariance matrix is not known a priori, it may be estimated and thereby becomes a random object with some intrinsic uncertainty itself. We show how to infer parameters in the presence of such an estimated covariance matrix, by marginalising over the true covariance matrix, conditioned on its estimated value. This leads to a likelihood function that is no longer Gaussian, but rather an adapted version of a multivariate t-distribution, which has the same numerical complexity as the multivariate Gaussian. As expected, marginalisation over the true covariance matrix improves inference when compared with Hartlap et al.'s method, which uses an unbiased estimate of the inverse covariance matrix but still assumes that the likelihood is Gaussian.
CMA-ES
Matérn covariance function
Scatter matrix
Matrix (chemical analysis)
Cite
Citations (209)
Matrix norm
CMA-ES
Scatter matrix
Matérn covariance function
Matrix (chemical analysis)
Cite
Citations (15)