Mutual equidistant-scattering criterion: A new index for crisp clustering

Abstract Clustering algorithms usually assume that the number K of clusters is known, although there is often no prior knowledge about the underlying set. Consequently, the significance of the defined groups needs to be validated. Cluster validity indexes are commonly used to perform the validation of clustering results. However, most of them are considered to be dependent on the number of data objects and often tend to ignore small and low-density groups. Furthermore, suboptimal clustering solutions are frequently selected when the clusters are in a certain degree of overlapping or low separation. Thus, we propose a new non-parametric internal validity index based on within-cluster mutual equidistant-scattering for crisp clustering. Eight different validity indexes were analysed to detect the number of clusters in a data set. Experiments on both synthetic and real-world data show the effectiveness and reliability of our approach to evaluate the hyperparameter K .