Implicit Regularization in Deep Learning: A View from Function Space - Baratin, Aristide; George, Thomas; Laurent, César; Hjelm, Devon; Lajoie, Guillaume; Vincent, Pascal; Lacoste-Julien, Simon (ArXiv preprint, 2020)
We approach the problem of implicit regularization in deep learning from a geometrical viewpoint. We highlight a possible regularization effect induced by a dynamical alignment of the neural tangent features introduced by Jacot et al, along a small number of task-relevant directions. By extrapolating a new analysis of Rademacher complexity bounds in linear models, we propose and study a new heuristic complexity measure for neural networks which captures this phenomenon, in terms of sequences of tangent kernel classes along in the learning trajectories.