Discriminative Densities from Maximum Contrast Estimation

Abstract

We propose a framework for classifier design based on\ndiscriminative densities for representation of the differences of the class-conditional distributions in a way that is optimal for classification. The densities are\nselected from a parametrized set by constrained maximization of some objective function which measures the average (bounded) difference, i.e. the contrast between discriminative densities. We show that maximiza-\ntion of the contrast is equivalent to minimization of an approximation of the Bayes risk. Therefore using suitable classes of probability density functions, the resulting\nmaximum contrast classifiers(MCCs) can approximate the Bayes rule for the general multiclass case. In particular\nfor a certain parametrization of the density functions we obtain MCCs which have the same functional form as the well-known Support Vector Machines (SVMs). We show that MCC-training in general requires some nonlinear optimization but under certain conditions the problem\nis concave and can be tackled by a single linear program. We indicate the close relation between SVM- and MCC-training and in particular we show that Linear Programming Machines can be viewed as an approxi-\nmate realization of MCCs. In the experiments on benchmark data sets, the MCC shows a competitive classification performance.