Inferring emotions from heterogeneous social media data: A Cross-media Auto-Encoder solution

Social media is rocking the world in recent year, which makes modeling social media contents important. However, the heterogeneity of social media data is the main constraint. This paper focuses on inferring emotions from large-scale social media data. Tweets on social media platform, always containing heterogeneous information from different combinations of modalities, are utilized to construct a cross-media dataset. How to integrate cross-media information and solve the problem of modality deficiency are main challenges. To address those challenges, this paper proposes a Cross-media Auto-Encoder(CAE) to infer emotions on cross-media data, and CAE is designed to reconstruct missing modalities and integrate heterogeneous representations. In our experiments, We employ a dataset of 226,113 tweets to infer emotions of tweets, and our method outperforms several machine learning methods (+11.11% in terms of F1-measure). Feature contribution analysis also verifies the importance of adopting cross-media features.