Simplifying the Algorithm Selection Using Reduction of Rankings of Classification Algorithms

The average ranking method (AR) is one of the simplest and effective algorithms selection methods. This method uses metadata in the form of test results of a given set of algorithms on a given set of datasets and calculates an average rank for each algorithm. The ranks are used to construct the average ranking. In this paper we investigate the problem of how the rankings can be reduced by removing non-competitive and redundant algorithms, thereby reducing the number of tests a user needs to conduct on a new dataset to identify the most suitable algorithm. The method proposed involves two phases. In the first one, the aim is to identify the most competitive algorithms for each dataset used in the past. This is done with the recourse to a statistical test. The second phase involves a covering method whose aim is to reduce the algorithms by eliminating redundant variants. The proposed method differs from one earlier proposal in various aspects. One important one is that it takes both accuracy and time into consideration. The proposed method was compared to the baseline strategy which consists of executing all algorithms from the ranking. It is shown that the proposed method leads to much better performance than the baseline.