HWE: Word Embedding with Heterogeneous Features

Distributed word representation is widely used in Natural Language Processing. However, traditional approaches, which learn word representations from the co-occurrence information in large corpora, might not capture fine-grained syntactic and semantic information. In this paper, we propose a general and flexible framework to incorporate heterogeneous features (e.g., word-sense, part-of-speech, topic) for learning feature-specific word embeddings in an explicit fashion, namely Heterogeneous Word Embedding (HWE). Experimental results on both intrinsic and extrinsic tasks show that HWE outperforms the baseline and various state-of-the-art models. Moreover, through the concatenation over HWE and the corresponding feature embeddings, each word would have different contextual representation under different contexts, which achieves even more significant improvement. Finally, we illustrate the insight of our model via visualization of the learned word embeddings.