Graph-Free Knowledge Distillation for Graph Neural Networks
35
Citation
24
Reference
10
Related Paper
Citation Trend
Abstract:
Knowledge distillation (KD) transfers knowledge from a teacher network to a student by enforcing the student to mimic the outputs of the pretrained teacher on training data. However, data samples are not always accessible in many cases due to large data sizes, privacy, or confidentiality. Many efforts have been made on addressing this problem for convolutional neural networks (CNNs) whose inputs lie in a grid domain within a continuous space such as images and videos, but largely overlook graph neural networks (GNNs) that handle non-grid data with different topology structures within a discrete space. The inherent differences between their inputs make these CNN-based approaches not applicable to GNNs. In this paper, we propose to our best knowledge the first dedicated approach to distilling knowledge from a GNN without graph data. The proposed graph-free KD (GFKD) learns graph topology structures for knowledge transfer by modeling them with multinomial distribution. We then introduce a gradient estimator to optimize this framework. Essentially, the gradients w.r.t. graph structures are obtained by only using GNN forward-propagation without back-propagation, which means that GFKD is compatible with modern GNN libraries such as DGL and Geometric. Moreover, we provide the strategies for handling different types of prior knowledge in the graph data or the GNNs. Extensive experiments demonstrate that GFKD achieves the state-of-the-art performance for distilling knowledge from GNNs without training data.Knowledge Sharing
Cite
Citations (0)
Knowledge Acquisition
Subject-matter expert
Cite
Citations (5)
The authors believe that current knowledge management practice significantly under-utilizes knowledge engineering technology, despite recent efforts to promote its use. They focus on two knowledge engineering processes: using knowledge acquisition processes to capture structured knowledge systematically; and using knowledge representation technology to store the knowledge, preserving important relationships that are far richer than those possible in conventional databases. To demonstrate the usefulness of these processes, we present a case study in which the drilling optimization group of a large oil and gas service company uses knowledge engineering practices to support the three facets of the knowledge management task: knowledge capture; knowledge storage; and knowledge deployment.
Knowledge Acquisition
Cite
Citations (99)
Processing knowledge is a prominent field of research and -- after the knowledge management hype a decade ago -- also in business domain. We observe two main trends, although not explicitly distinguishable. First the knowledge engineering approaches focusing on machine interpretable knowledge and second the knowledge management approaches that center on human interpretable knowledge. It is proven that both approaches can be supported by models, for knowledge engineering more formalized knowledge expressions and for knowledge management also informal knowledge expressions. The Open Knowledge Model (OKM) project on Open Models aims to bring these two communities together by applying an open model-based approach for modeling knowledge. A first prototype has been developed in the project plugIT. This paper will introduce the approach, the findings and provide an outlook on OKM.
Knowledge modeling
Cite
Citations (5)
In the era of big data, knowledge engineering faces fundamental challenges induced by fragmented knowledge from heterogeneous, autonomous sources with complex and evolving relationships. The knowledge representation, acquisition, and inference techniques developed in the 1970s and 1980s, driven by research and development of expert systems, must be updated to cope with both fragmented knowledge from multiple sources in the big data revolution and in-depth knowledge from domain experts. This article presents BigKE, a knowledge engineering framework that handles fragmented knowledge modeling and online learning from multiple information sources, nonlinear fusion on fragmented knowledge, and automated demand-driven knowledge navigation.
Knowledge Acquisition
Subject-matter expert
Cite
Citations (75)
Knowledge management systems are aimed to provide knowledge-intensive organisations with tools, and methods to better manage their knowledge capital. A great success is gained in the course of managing explicit knowledge in the form documented knowledge fragments. But, greater part of organisations knowledge is realised in tacit form which is volatile and hardly captured in a formal way. Managing this type of knowledge still represents one of the major challenges in knowledge management research. This paper proposes a knowledge model that caters for capturing both tacit and explicit organisational knowledge in the software-engineering domain.
Tacit Knowledge
Explicit knowledge
Cite
Citations (8)
The paper presents a scheme for categorizing knowledge engineering tools. The classification of knowledge acquisition systems has revealed some interesting facts about these systems. It seems that systems which are able to work on multiple tasks produce very shallow (i.e., not of expert-level) knowledge bases. On the other hand, systems which produce expert-level knowledge bases function on a single task. These insights have led to the design of ASKE, a knowledge acquisition system which can be used to build expert-level knowledge bases in several domains and for different task-types. The knowledge acquisition process is based on the notion of templates, the knowledge-bearing units of ASKE. Templates provide a convenient way of representing domain knowledge.
Knowledge Acquisition
Template
Legal expert system
Subject-matter expert
Cite
Citations (1)
A well-known problem in requirements engineering is the communication between stakeholders with different background. This communication problem is mostly attributed to the different "languages" spoken by these stakeholders based on their different background and domain knowledge. We experienced a related problem involved with transferring and sharing such knowledge, when stakeholders are reluctant to do this. So, we take a knowledge management perspective of requirements engineering and carry over ideas for the sharing of knowledge about requirements and the domain. We cast requirements engineering as a knowledge management process and adopt the concept of the spiral of knowledge involving transformations from tacit to explicit knowledge, and vice versa. In the context of a real-world problem, we found the concept of "knowledge holders" and their relations to categories of requirements and domain knowledge both useful and important. This project was close to become a failure until knowledge transfer has been intensified. The knowledge management perspective provided insights for explaining improved knowledge exchange.
Tacit Knowledge
Knowledge Sharing
Knowledge Transfer
Cite
Citations (28)