Neural Tangent Kernel: Generalization in Deep Learning

This lecture explores the concept of the neural tangent kernel (NTK) in deep learning, showing how it converges to a constant value as the neural network width approaches infinity. It discusses the impact of network size on generalization, highlighting the role of random noise and fluctuations during training and initialization. The lecture also delves into the asymptotic behavior of generalization as the number of parameters increases, demonstrating the variance of the output and its relationship with network size. Various measures of generalization are analyzed, revealing patterns such as peaks and decays. The presentation concludes with insights on the behavior of wide neural networks under gradient descent and their evolution into linear models.