Experimental Design and Analysis of Sound Event Detection Systems: Case Studies

Sound Event Detection (SED) systems have attracted a widespread attention from the Machine Learning community due to their potential applications. As in many sound analysis scenarios, it is of paramount importance to assess and compare the performances of such systems trained from sound data. The main goal of system evaluation and comparison is to derive conclusions unaffected by chance and are therefore significant. Both the design of experiments and the analysis of results through statistical tests should be conducted properly in order to insure significance. In this paper, we first discuss the principles of machine learning experiments in the context of SED. Then, we show the proper methodologies through case studies. To this end, we have examined four classification models (Support Vector Machine, Convolutional Neural Network, Adaboost and Random Forest) trained using Mel Frequency Cepstral Coefficients (MFCCs). Most importantly, we have supported our analysis and discussions with numerous statistical tests. Without the use of a proper data augmentation approach, the experimental results indicate the superiority of the ensemble-based classifiers (Random Forest and Adaboost), with an overall detection accuracy of 83%. Furthermore, adding the first and second derivative of MFCCs significantly improves the performance of SVM-based systems.