Curious Cases of Automatically Generated Text and Detecting Probabilistic Context Free Grammar Sentences with Grammatical Structure Similarity

Automatically generated papers have been used to manipulate bibliography indexes on numerous occasions. This paper is interested in different means to generate texts such as by a recurrent neural network, a Markov model, or a probabilistic context free grammar and if it is possible to detect them using a current approach. Then, probabilistic context free grammar (PCFG) is focused on as the one most used. However, despite that there have been multiple approaches to detecting such types of paper. Yet, they are all working at the document level and are unable to detect a small amount of generated text inside a larger body of genuinely written text. Thus, we present the Grammatical Structure Similarity (GSS) measurement to detect sentences or short fragments of automatically generated text from known PCFG generators. The proposed approach is tested against a pattern checker and various common machine learning methods. Additionally, the ability to detect a modified generator is also tested.