A CNN-LSTM network with attention approach for learning universal sentence representation in embedded system

Abstract The model for obtaining universal sentence representation is getting larger and larger, making it unsuitable for small embedded systems. The paper presents an extended encoder-decoder model with introduced an attention mechanism for learning distributed sentence representation. We can extract the CNN encoder and apply to other NLP downstream tasks on small embedded systems. Inspired by the linguistic features of the word embeddings, the different dimensions of the sentence representation can be aligned to especially linguistic features. The decoder which decodes one word will focus on the partial dimension of sentence representation into a fixed-length vector where CNN is more effective than LSTM, especially on devices with limited computing power. Moreover, the expanded LSTM with attention mechanism as the decoder to learn multitask that reconstruct the original sentence and predict the next sentence. The model was trained on an extensive collection of a novel to learning sentence representation encoder. Finally, the small-scale CNN encoder obtained encouraging results on several benchmark datasets and multiple task.