Two-Stage Encoder for Pointer-Generator Network with Pretrained Embeddings

Abstractive summarization has become the mainstream of automatic text summarization because of its unique flexibility. Pointer-Generator Network(PGN) has already been the de facto standard for the text summarization model. However, this model still has two main shortcomings: first, since the word embeddings obtained from the word embedding matrix lack of contextual semantic information, the model tends to ignore the global information of the input text when generating the summary. Second, the embedding matrix is only learned on a single corpus from scratch, which decreases the convergence speed of the PGN model and affects the generalization ability of the word embeddings. In this paper, we propose a new text summarization framework, Two-Stage Encoder for pointer-generator network with Pretrained Embeddings, which aims to solve the problems mentioned above. We introduce an encoder based on the multi-head self-attention for the first-stage encoding. This encoder module can improve the word representation by solving the polysemy problem in the PGN model and alleviate the problem of ignoring global information. Besides, we apply the pretrained word embeddings GloVe to initialize the word embedding matrix and then fine-tune it. A range of experiments are conducted on the CNN/Daily Mail dataset. For the CNN/Daily mail dataset, comparing with the baseline model, the proposed model enhances the ROUGE metrics by 0.75, 0.74 and 0.77 respectively.