Enhancing keyword correlation for event detection in social networks using SVD and k-means: Twitter case study

Extracting textual features from tweets is a challenging task due to the noisy nature of the content and the weak signal of most of the words used. In this paper, we propose using singular value decomposition (SVD) with clustering to group related words as enhanced signals for textual features in tweets in order to improve the correlation with events. The proposed method applies SVD to the time series vector for each feature to factorize the matrix of feature/day counts, to ensure the independence of the feature vectors. Then, k-means clustering is applied to build a look-up table that maps members of each cluster to the cluster centroid. The look-up table is used to map each feature in the original data to the centroid of its cluster. Then, we calculate the sum of the term-frequency vectors of all features in each cluster to the term-frequency vector of the cluster centroid. To evaluate the method, we calculated the correlations of the cluster centroids with the golden standard record vector before and after summing the vectors of the cluster members to the centroid vector. The proposed method is applied to multiple correlation techniques including the Pearson, Spearman, distance correlation, and Kendal Tao. The experiments also considered the different word forms and lengths of the features including keywords, n grams, skip grams, and bags-of-words. The correlation results are enhanced significantly as the highest correlation scores have increased from 0.22 to 0.70, and the average correlation scores have increased from 0.22 to 0.60.