Constructive Algorithms for Hierarchical Mixtures of Experts

Abstract

We present two additions to the hierarchical mixture of experts (HME) architecture. We view the HME as a tree structured classifier. Firstly, by applying a likelihood splitting criteria to each expert in the HME we "grow" the tree adaptively during training. Secondly, by considering only the most probable path through the tree we may "prune" branches away, either temporarily, or permanently if they become redundant. We demonstrate results for the growing and pruning algorithms which show significant speed ups and more efficient use of parameters over the conventional algorithms in discriminating between two interlocking spirals and classifying 8-bit parity patterns. INTRODUCTION The HME (Jordan & Jacobs 1994) is a tree structured network whose terminal nodes are simple function approximators in the case of regression or classifiers in the case of classification. The outputs of the terminal nodes or experts are recursively combined upwards towards the root node, to form the overall out...