The concept of pretrained language models in the context of knowledge engineering

The article studies the concept and technologies of pre-trained language models in the context of knowledge engineering. The author substantiates the relevance of the issue of the existence of internalized and implicit knowledge, extracted from text corpora used for pre-training or transfer learning in pre-trained language models. The article provides a detailed overview of the existing approaches to the interpretation of this concept. The author reviews a number of recent studies related to pre-training and transfer learning methods in regards to language models. This article discusses the latest research on the augmentation of language models with knowledge. Moreover, it studies the current research on the use of pre-trained language models to search and retrieve knowledge, to aid in the process of building knowledge bases, as well as their use as independent knowledge bases. The content of the concept "pretrained language models" is explained. The author provides examples of the implementation of pre-trained language models in practice, including the discussion of the use of language models as knowledge bases. The essence of the concept of unsupervised pre-training of language models using large and unstructured text corpora before further training for a specific task (fine tuning), "transfer learning", is also touched on. The author examines the concept of "knowledge graph", which is now widely used both in general and in the context relevant to this article, as well as a number of recent research in the realm of pre-training and transfer learning in regards to language models.