On Analysing Similarity Knowledge Transfer by Ensembles.

Knowledge transfer is the task of transferring the knowledge learned by a model A to a new model B. This task is essential in Deep Learning, since there are complex models with excellent results, but computationally costly to be executed. The Similarity Knowledge Transfer (SKT) method proposes an approach to transfer the knowledge layer-by-layer between a donor model and a receiver model. This transfer is carried out through the representations learned by the layers from the teacher model. Despite presenting good results, the SKT method proposes just a way to transfer knowledge between two models. Therefore, this work presents the Similarity Knowledge Transfer Ensemble (SKTE) method, a generic form of SKT that allows the transfer from several teachers to a single student model. We carried out experiments with the CIFAR10 benchmark, where the results obtained showed promising results in this activity.