Author(s: LA Tran Deep Learning Guide to help students learn from teachers. Knowledge distillation allows you to take the information from multiple models, and combine it in one model to allow for deployments in real-life. Geoffrey Hinton (the Godfather of AI), and Oriol Vinyals (Google’s co-workers in 2015. introduced knowledge distillation. Image from Prakhar’s post [source]. Knowledge distillation is the process of moving the learning behaviour of an inefficient model (teacher), to one that’s more manageable (student). The output of the teacher can then be used to train the student. The authors found surprising results using this approach to the MNIST dataset. They also showed how a group of knowledge can be combined into one model by distilling it. Knowledge Distillation in Image Classification Hinton, his co-authors and the paper Distilling Knowledge in a Neuronal Network first presented their knowledge distillation method for image classification in the paper. The simplest type of knowledge distillation, as mentioned in the paper is when the model that is distilled is used to train on a set with soft targets distribution. We have seen that two target types are available for the training of a student model. The correct labels are the hard target (hard target), and the soft targets (soft target), come from teacher networks. The objective function is therefore a weighted combination of two objective functions. The cross-entropy between student predictions and soft targets is the first objective function. The second objective function measures the loss of cross-entropy between student outputs and correct labels. According to the authors, the most effective results are generally achieved by using lower values for the second objective function. The paper contains some surprising results. For more information, see the original paper. Image from paper [source]. Image from paper[source]. Knowledge Distillation for Object Detection in NeurIPS Guobin Ch and co-authors published research on knowledge distillation coupled with hint learning to detect objects in their paper Learning Efficient Object Detection models with Knowledge Distillation. They also use a hint, which is the feature map from the intermediate layer that the teacher has created to guide the student in learning the behaviour of the teacher. An adaptation layer is also necessary to extract knowledge at its best. We will discuss this layer later. The object detection network that was used in the paper’s experiment is called Faster-RCNN. The figure shows their learning strategy. Image taken from the paper. [source] The learning objective function is written as below: where RCN and RPN stand for regression-and-classification network and region proposal network, respectively; N and M are the batch-size of RCN and RPN, respectively; L_RCN, L_RPN, and L_Hint are the losses of RCN, RPN, and hint, respectively; l (typically is 1) and g (is usually set to 0.5) are hyper-parameters for controlling the final loss. In the paper FitNets, Hints to Thin Deep Nets Adriana Romero from Hint Learning has shown that students can have better performance when they are given a hint about the process. This is the difference between the lost hint feature Z (the featuremap obtained from an intermediate layer by the teacher) or guided feature V (the student’s feature map from an intermediate layer) which can be calculated using either L1 distance (or L2 distance). The following figure shows the feature maps taken from the pre-trained YOLOv4 models that were trained with the WAYMO dataset. It is one of my research projects on object detection and knowledge distillation. The input image has been resized in these cases to 800x 800.. Photo by the Author. Photo by author. Knowledge distillation and hint learning require that both the guide feature (height x width x channel) have the same form. The guide feature and the hint feature won’t always be located in the same feature space. An adaptation layer is typically a 1×1 convolutional layers. This helps to facilitate the transfer of knowledge between student to teacher. This image is an adaptation of Prakhar’s post [source]. The figure below shows the object detection learning program that I used in my project. It was a network with only three levels for detection. I wanted to extract knowledge from the pre-trained YoLOv4. Photo by the Author. Guobin Chen’s excellent combination of knowledge distillation and hint learning to aid in object detection yields outstanding results. The original paper Learning Efficient Object Detection models with Knowledge Distillation contains more details. Photo taken from the paper. [source] Conclusions I briefly discussed knowledge distillation in this blog post. Knowledge distillation can be used to transfer the knowledge from a group of complex models into one that is smaller and more efficient. Combining hint learning with knowledge distillation can greatly improve neural network performance. My Facebook Fan Page is open to readers. It allows you to share information about Machine Learning: Diving into Machine Learning. Thank you for taking the time to read this! The Gentle Introduction to Hint Learning & Knowledge Distillation originally appeared in on Medium. People are responding by highlighting the story and commenting on it. Published via
