Deep filter banks for texture recognition and segmentation

TLDR

Texture recognition research typically focuses on material recognition in uncluttered settings, which rarely matches real‑world applications. The study aims to evaluate material and texture attribute recognition in cluttered scenes using a new OpenSurface‑derived dataset and introduces the FV‑CNN descriptor. FV‑CNN is built by Fisher Vector pooling of a CNN filter bank, enabling multi‑scale, arbitrarily shaped region description. FV‑CNN outperforms prior methods, achieving 79.8 % on Flickr materials and 81 % on MIT indoor scenes—over 10 % absolute gains—while transferring across domains without adaptation and delivering state‑of‑the‑art segmentation on MSRC and promising cluttered material recognition on OpenSurfaces.

Abstract

Research in texture recognition often concentrates on the problem of material recognition in uncluttered conditions, an assumption rarely met by applications. In this work we conduct a first study of material and describable texture attributes recognition in clutter, using a new dataset derived from the OpenSurface texture repository. Motivated by the challenge posed by this problem, we propose a new texture descriptor, FV-CNN, obtained by Fisher Vector pooling of a Convolutional Neural Network (CNN) filter bank. FV-CNN substantially improves the state-of-the-art in texture, material and scene recognition. Our approach achieves 79.8% accuracy on Flickr material dataset and 81% accuracy on MIT indoor scenes, providing absolute gains of more than 10% over existing approaches. FV-CNN easily transfers across domains without requiring feature adaptation as for methods that build on the fully-connected layers of CNNs. Furthermore, FV-CNN can seamlessly incorporate multi-scale information and describe regions of arbitrary shapes and sizes. Our approach is particularly suited at localizing "stuff" categories and obtains state-of-the-art results on MSRC segmentation dataset, as well as promising results on recognizing materials and surface attributes in clutter on the OpenSurfaces dataset.