Deformable Convolutional Networks

TLDR

Convolutional neural networks are limited in modeling geometric transformations because their modules have fixed geometric structures. This work introduces deformable convolution and deformable RoI pooling modules to enhance CNNs’ ability to model transformations. The modules augment spatial sampling with learnable offsets, replace standard counterparts, and can be trained end‑to‑end via back‑propagation. Extensive experiments show that learning dense spatial transformations improves object detection and semantic segmentation, and the code is publicly available.

Abstract

Convolutional neural networks (CNNs) are inherently limited to model geometric transformations due to the fixed geometric structures in their building modules. In this work, we introduce two new modules to enhance the transformation modeling capability of CNNs, namely, deformable convolution and deformable RoI pooling. Both are based on the idea of augmenting the spatial sampling locations in the modules with additional offsets and learning the offsets from the target tasks, without additional supervision. The new modules can readily replace their plain counterparts in existing CNNs and can be easily trained end-to-end by standard back-propagation, giving rise to deformable convolutional networks. Extensive experiments validate the performance of our approach. For the first time, we show that learning dense spatial transformation in deep CNNs is effective for sophisticated vision tasks such as object detection and semantic segmentation. The code is released at https://github.com/msracver/Deformable-ConvNets.