Populating the knowledge base of a conversational agent: human vs. machine

In this paper we study the task of creating questions to populate the knowledge base of a domain-oriented conversational agent, constituted of question-answer pairs. Considering the growing interest in Question Generation, a question arises: how good are current systems in the task of automatically populating an agent's knowledge base? In this study, three systems are evaluated in several dimensions specifically tailored for the task in hand. We evaluate not only the capacity of these systems to create questions close to the ones suggested by humans, but also the human effort in editing the set of generated questions to be ready to be added to the agent's knowledge base. This experiment leads us to a second question: does the set of automatically generated question complement/extend the questions generated by humans? Results show that these state-of-the-art Question Generation systems, even as a whole, are far from being able to generate the majority of the questions proposed by humans (up to 20% of coverage), and that, on average, several edits are needed (around 3) to correct the generated questions. However, on the other hand, these systems also contribute with questions that humans have not thought about, thus contributing to extend the pool of questions generated by humans.