Model Selection and Assessment Using Cross-indexing

Cross-indexing is a method for selecting the optimal model complexity and estimating the corresponding performance. It aims to reduce the optimistic selection bias that may emerge when too many models are compared to each other using cross-validation as the performance estimator. In this paper, a generalization is introduced that covers the previously presented variations (cross-indexing A and B) as special cases. Originally, cross-indexing was suggested for decreasing the selection bias in a feature selection setting. In order to apply it to a generic model selection problem that presents a large number of candidate model structures and an infinite number of potential hyperparameter values, the method needs to be modified. This paper also describes one way of doing such modifications, and reports the promising results of using the consequent method in three open competitions.