Data set quality in Machine Learning: Consistency measure based on Group Decision Making

Abstract Performance of Machine Learning models heavily depends on the quality of the training dataset. Among others, the quality of training data relies on the consistency of the labels assigned to similar items. Indeed, the labels should be coherently assigned (or collected) by avoiding inconsistencies for increasing the performance of the machine learning model. This study focuses on evaluating training data consistency for machine learning algorithms dealing with ranking problems, i.e., the Learning to Rank methods (LTR). This work defines a training data consistency measure based on the consensus value introduced in Group Decision Making. It investigates the statistical relationship between the proposed consistency measure and the performance of a deep neural network implementing an LTR method. This measure could drive data filtering at the training stage and guide model update decisions. Experimentation reveals a strong correlation between the proposed consistency measure and the performance of the model.