arXiv (Cornell University) · 2016 · 37 citations · 0 references
ComputingComputer ScienceMachine LearningOptimizationStatistical LearningKernel MethodsLoss FunctionOptimization For Machine LearningRepresentation LearningLabel Noise MitigationLabel Noise RobustnessNoise RobustnessRobust LossRobust Loss FunctionsRobustness To NoiseWeak SupervisionWeakly Supervised Learning
We prove that the empirical risk of most well-known loss functions factors\ninto a linear term aggregating all labels with a term that is label free, and\ncan further be expressed by sums of the loss. This holds true even for\nnon-smooth, non-convex losses and in any RKHS. The first term is a (kernel)\nmean operator --the focal quantity of this work-- which we characterize as the\nsufficient statistic for the labels. The result tightens known generalization\nbounds and sheds new light on their interpretation.\n Factorization has a direct application on weakly supervised learning. In\nparticular, we demonstrate that algorithms like SGD and proximal methods can be\nadapted with minimal effort to handle weak supervision, once the mean operator\nhas been estimated. We apply this idea to learning with asymmetric noisy\nlabels, connecting and extending prior work. Furthermore, we show that most\nlosses enjoy a data-dependent (by the mean operator) form of noise robustness,\nin contrast with known negative results.\n