Exponentially many local minima for single neurons

Abstract

We show that for a single neuron with the logistic function as the transfer function the number of local minima of the error function based on the square loss can grow exponentially in the dimension. 1 INTRODUCTION Consider a single artificial neuron with d inputs. The neuron has d weights w 2 R d . The output of the neuron for an input pattern x 2 R d is y = OE(x \\Delta w), where OE : R ! R is a transfer function. For a given sequence of training examples h(x t ; y t )i 1tm ; each consisting of a pattern x t 2 R d and a desired output y t 2 R, the goal of the training phase for neural networks consists of minimizing the error function with respect to the weight vector w 2 R d . This function is the sum of the losses between outputs of the neuron and the desired outputs summed over all training examples. In notation, the error function is E(w) = m X t=1 L(y t ; OE(x t \\Delta w)) ; where L : R \\Theta R ! [0; 1) is the loss function. Acommon example of a transfer function...