Bounded Approximations for Marginal Likelihoods

We discuss novel approaches to evaluation of both upper and lower bounds on log marginal likelihoods for model comparison in Bayesian analysis. From posterior Monte Carlo samples, we show how existing variational approximation methods defining lower bounds on marginal likelihoods can be extended to also define upper bounds, and develop optimization methods to minimize such upper bounds. Further, using this new approach to upper bound evaluation, we suggest and exemplify a new quasi-optimized lower bound that can often be obtained with trivial computations compared to current methods. We further discuss the use of partial analytic marginalization of some model parameters as a way of significantly reducing the differences between upper and lower bounds to improve marginal likelihood approximation. To implement this, however, traditional variational methods are intractable, and we provide solution in terms of a novel Monte Carlo Stochastic Approximation (MCSA). We provide theoretical results on convergence of the resulting approximations to true bounds, and several simulation examples in regression and mixture models to demonstrate the accuracy and efficacy ofthe new