On the Expressiveness of Approximate Inference in Bayesian Neural Networks.

While Bayesian neural networks (BNNs) hold the promise of being flexible, well-calibrated statistical models, inference often requires poorly understood approximations. We study the impact of approximate inference in BNNs on the quality of uncertainty quantification, focusing on methods that use parametric approximating distributions. For single-hidden layer ReLU BNNs, we prove a fundamental limitation in function-space of two of the most ubiquitous distributions defined in weight-space: mean-field Gaussian and Monte Carlo dropout. In particular, neither method can have substantially increased uncertainty in between well-separated regions of low uncertainty. In contrast, for deep networks, we prove a universality result showing that there exist distributions in the above classes which provide flexible uncertainty estimates. However, we find that in practice pathologies of the same form as in the single-hidden layer case often persist when performing variational inference in deeper networks. Our results motivate careful consideration of the implications of approximate inference methods in BNNs.