Multi-Headed Self-Attention-based Hierarchical Model for Extractive Summarization

In this paper, we present a novel model for extractive document summarization using a single layer of multi-headed attention on the top of representations from an RNN (GRU). The model is hierarchical and learns the sentence and document representations hierarchically. It achieves performance better or comparable to the earlier such models. The paper establishes the utility of multi-headed attention in refining the representations and capturing the dependencies of words in a sentence and sentences in a document.