Proceedings of the AAAI Conference on Artificial Intelligence · 2022 · 10 citations · 31 references
ComputingComputer ScienceArtificial IntelligenceMachine LearningOptimizationDeep LearningKernel MethodsLoss FunctionNeural NetworksOptimization For Machine LearningOverfittingApproximation ErrorBlack Box ModelingInfluence FunctionInfluence Function AnalysisNeural Network PredictionProbability Density AnalysisRegularizationRegularized Loss
Understanding the black-box prediction for neural networks is challenging. To achieve this, early studies have designed influence function (IF) to measure the effect of removing a single training point on neural networks. However, the classic implicit Hessian-vector product (IHVP) method for calculating IF is fragile, and theoretical analysis of IF in the context of neural networks is still lacking. To this end, we utilize the neural tangent kernel (NTK) theory to calculate IF for the neural network trained with regularized mean-square loss, and prove that the approximation error can be arbitrarily small when the width is sufficiently large for two-layer ReLU networks. We analyze the error bound for the classic IHVP method in the over-parameterized regime to understand when and why it fails or not. In detail, our theoretical analysis reveals that (1) the accuracy of IHVP depends on the regularization term, and is pretty low under weak regularization; (2) the accuracy of IHVP has a significant correlation with the probability density of corresponding training points. We further borrow the theory from NTK to understand the IFs better, including quantifying the complexity for influential samples and depicting the variation of IFs during the training dynamics. Numerical experiments on real-world data confirm our theoretical results and demonstrate our findings.
31
Gradient-based learning applied to document recognition
Yann LeCun, Léon Bottou, Yoshua Bengio et al. · Proceedings of the IEEE · 1998 · 58.2K citations · Full text
Understanding Black-box Predictions via Influence Functions
Pang Wei Koh, Percy Liang · arXiv (Cornell University) · 2017 · 1.2K citations · Full text
Foundations of Machine Learning
Ajit Singh · SSRN Electronic Journal · 2019 · 1.2K citations · Full text
The Kernel Recursive Least-Squares Algorithm
Yaakov Engel, Shie Mannor, Ron Meir · IEEE Transactions on Signal Processing · 2004 · 1.1K citations