A Theoretical Analysis of the Repetition Problem in Text Generation
4
Citation
0
Reference
10
Related Paper
Citation Trend
Abstract:
Text generation tasks, including translation, summarization, language models, and etc. see rapid growth during recent years. Despite the remarkable achievements, the repetition problem has been observed in nearly all text generation models undermining the generation performance extensively. To solve the repetition problem, many methods have been proposed, but there is no existing theoretical analysis to show why this problem happens and how it is resolved. In this paper, we propose a new framework for theoretical analysis for the repetition problem. We first define the Average Repetition Probability (ARP) to characterize the repetition problem quantitatively. Then, we conduct an extensive analysis of the Markov generation model and derive several upper bounds of the average repetition probability with intuitive understanding. We show that most of the existing methods are essentially minimizing the upper bounds explicitly or implicitly. Grounded on our theory, we show that the repetition problem is, unfortunately, caused by the traits of our language itself. One major reason is attributed to the fact that there exist too many words predicting the same word as the subsequent word with high probability. Consequently, it is easy to go back to that word and form repetitions and we dub it as the high inflow problem. Furthermore, we derive a concentration bound of the average repetition probability for a general generation model. Finally, based on the theoretical upper bounds, we propose a novel rebalanced encoding approach to alleviate the high inflow problem. The experimental results show that our theoretical framework is applicable in general generation models and our proposed rebalanced encoding approach alleviates the repetition problem significantly. The source code of this paper can be obtained from https://github.com/fuzihaofzh/repetition-problem-nlg.Keywords:
Repetition (rhetorical device)
Text summarization is a process of distilling the most important content from text documents. While human beings have proven to be extremely capable summarizers, computer based automatic abstracting and summarizing has proven to be extremely challenging tasks. In this paper we report our experience with applying extractive summarization techniques to process news articles, economic reports and nursing narratives. We present analysis of the effect of different summarization methods and parameters on the summarization results. We also compare the performance of the summarizers across the three different document genres. The learned lessons are discussed and the possibilities for applying the theory of Computing with Words in text summarization are elaborated.
Multi-document summarization
Cite
Citations (8)
This paper presents novel prompting techniques to improve the performance of automatic summarization systems for scientific articles. Scientific article summarization is highly challenging due to the length and complexity of these documents. We conceive, implement, and evaluate prompting techniques that provide additional contextual information to guide summarization systems. Specifically, we feed summarizers with lists of key terms extracted from articles, such as author keywords or automatically generated keywords. Our techniques are tested with various summarization models and input texts. Results show performance gains, especially for smaller models summarizing sections separately. This evidences that prompting is a promising approach to overcoming the limitations of less powerful systems. Our findings introduce a new research direction of using prompts to aid smaller models.
Content (measure theory)
Multi-document summarization
Cite
Citations (0)
The experience summarization,which undertakes the summarization of the monographic study,serves as an important way for teaching and scientific research.It has such features as cause analysis,applications and practicality.This paper shows that the experience summarization is to determine the subject discussed,write an outline,collect and analyze materials,express in words and correct mistakes.The paper also points out that the experience summarization should be applied to the principle of applications,creativeness and science.
Multi-document summarization
Cite
Citations (0)
In order to produce summaries from dynamic content, we address the definition of the dynamic summarization. In this paper, the issue of modeling of dynamic summarization is discussed, and then two solutions of model improvement with set theory and algorithm improvement with reranking are proposed for dynamic summarization from classic summarization. Finally, the performances of these two solutions are evaluated on the dataset of DUC 2007. Our results demonstrate that the model improvement solution is more effective, but as another stride towards summarization, dynamic summarization research still need further study.
STRIDE
Multi-document summarization
Cite
Citations (0)
In order to produce summaries from dynamic content, we address the definition of the dynamic summarization. In this paper, the issue of modeling of dynamic summarization is discussed, and then two solutions of model improvement with set theory and algorithm improvement with reranking are proposed for dynamic summarization from classic summarization. Finally, the performances of these two solutions are evaluated on the dataset of DUC 2007. Our results demonstrate that the model improvement solution is more effective, but as another stride towards summarization, dynamic summarization research still need further study.
STRIDE
Multi-document summarization
Cite
Citations (0)
Most of the text summarization research carried out to date has been concerned with the summarization of short documents (e.g., news stories, technical reports), and very little work if any has been done on the summarization of very long documents. In this paper, we try to address this gap and explore the problem of book summarization. We introduce a new data set specifically designed for the evaluation of systems for book summarization, and describe summarization techniques that explicitly account for the length of the documents.
Multi-document summarization
Cite
Citations (74)
We study correlation of rankings of text summarization systems using evaluation methods with and without human models. We apply our comparison framework to various well-established content-based evaluation measures in text summarization such as coverage, Responsiveness, Pyramids and Rouge studying their associations in various text summarization tasks including generic and focus-based multi-document summarization in English and generic single-document summarization in French and Spanish. The research is carried out using a new content-based evaluation framework called Fresa to compute a variety of divergences among probability distributions.
Multi-document summarization
Cite
Citations (78)
In order to produce summaries from dynamic content, we address the definition of the dynamic summarization. In this paper, the issue of modeling of dynamic summarization is discussed, and then two solutions of model improvement with set theory and algorithm improvement with reranking are proposed for dynamic summarization from classic summarization. Finally, the performances of these two solutions are evaluated on the dataset of DUC 2007. Our results demonstrate that the model improvement solution is more effective, but as another stride towards summarization, dynamic summarization research still need further study.
STRIDE
Multi-document summarization
Cite
Citations (1)
Extractive summarization and generative summarization are the two main ways to generate summarization.However,previous work treats both of them as two independent subtasks.In this paper,we obtain new summarization by combining extractive summarization and generative summarization.This method extracts the key information of the article firstly,and then generates the summarization of the extracted information.The experimental result shows that this method can significantly improve the quality of the generative text compared with extractive summarization,and can significantly improve the generative speed compared with generative summarization.
Multi-document summarization
Generative model
Cite
Citations (0)
To mitigate the lack of diverse dialogue summarization datasets in academia, we present methods to utilize non-dialogue summarization data for enhancing dialogue summarization systems. We apply transformations to document summarization data pairs to create training data that better befit dialogue summarization. The suggested transformations also retain desirable properties of non-dialogue datasets, such as improved faithfulness to the source text. We conduct extensive experiments across both English and Korean to verify our approach. Although absolute gains in ROUGE naturally plateau as more dialogue summarization samples are introduced, utilizing non-dialogue data for training significantly improves summarization performance in zero- and few-shot settings and enhances faithfulness across all training regimes.
Multi-document summarization
Cite
Citations (0)