Gradient-based Methods for Deep Model Interpretability

In this dissertation, we propose gradient-based methods for characterizing model behaviour for the purposes of knowledge transfer and post-hoc model interpretation. Broadly, gradients capture the variation of some output feature of the model upon unit variation of an input feature, and thus encodes the local model behaviour while being agnostic to the underlying model architectural choices.Our first contribution is to propose a sample-efficient method to mimic the behaviour of a pre-trained teacher model with an untrained student model using gradient information. We interpret our approach as an efficient alternative to data augmentation used with canonical knowledge transfer approaches, where noise is added to the inputs. We apply this to distillation and a transfer learning task, where we show improved performance for small datasets.Our second contribution is to propose a novel saliency method to visualize the input features that are most relevant for predictions made by a given model. We first propose the full-gradient representation, which satisfies a property called completeness which provably cannot be satisfied by gradient-based saliency methods. Based on this, we propose an approximate saliency map representation called FullGrad which naturally captures the information within a model across feature hierarchies. Our experimental results show that FullGrad captures model behaviour better than other saliency methods.Our final contribution is to take a step back and ask why input-gradients are informative for standard neural network models in the first place, especially when their structure may as well be arbitrary. Our analysis here reveals that for a subset of gradient-based saliency maps, the map relies not on the underlying discriminative model p(y | x) but on a hidden density model p(x | y) implicit within softmax-based disciminative models. Thus we find that the reason input-gradients are informative is due to the alignment of the implicit density model with that of the ground truth density, which we verify experimentally.