Generalizing Pooling Functions in Convolutional Neural Networks: Mixed,\n Gated, and Tree

Abstract

We seek to improve deep neural networks by generalizing the pooling\noperations that play a central role in current architectures. We pursue a\ncareful exploration of approaches to allow pooling to learn and to adapt to\ncomplex and variable patterns. The two primary directions lie in (1) learning a\npooling function via (two strategies of) combining of max and average pooling,\nand (2) learning a pooling function in the form of a tree-structured fusion of\npooling filters that are themselves learned. In our experiments every\ngeneralized pooling operation we explore improves performance when used in\nplace of average or max pooling. We experimentally demonstrate that the\nproposed pooling operations provide a boost in invariance properties relative\nto conventional pooling and set the state of the art on several widely adopted\nbenchmark datasets; they are also easy to implement, and can be applied within\nvarious deep neural network architectures. These benefits come with only a\nlight increase in computational overhead during training and a very modest\nincrease in the number of model parameters.\n