Describing Textures in the Wild

TLDR

Patterns and textures are key characteristics of many natural objects, such as striped shirts, veined butterfly wings, and scaly animal skin. The authors aim to support image understanding by describing textures with semantic attributes and creating the Describable Textures Dataset (DTD) to find optimal representations for recognizing these attributes. They identify a vocabulary of 47 texture terms and use it to annotate a large, in‑the‑wild dataset of patterns, forming the DTD. Using the DTD, the authors show that Improved Fisher Vector and Deep Convolutional‑Network Activation Features outperform specialized texture descriptors, achieving more than 10% improvement on FMD and KTH‑TIPS‑2b and producing intuitive material descriptions for images.

Abstract

Patterns and textures are key characteristics of many natural objects: a shirt can be striped, the wings of a butterfly can be veined, and the skin of an animal can be scaly. Aiming at supporting this dimension in image understanding, we address the problem of describing textures with semantic attributes. We identify a vocabulary of forty-seven texture terms and use them to describe a large dataset of patterns collected "in the wild". The resulting Describable Textures Dataset (DTD) is a basis to seek the best representation for recognizing describable texture attributes in images. We port from object recognition to texture recognition the Improved Fisher Vector (IFV) and Deep Convolutional-network Activation Features (DeCAF), and show that surprisingly, they both outperform specialized texture descriptors not only on our problem, but also in established material recognition datasets. We also show that our describable attributes are excellent texture descriptors, transferring between datasets and tasks, in particular, combined with IFV and DeCAF, they significantly outperform the state-of-the-art by more than 10% on both FMD and KTH-TIPS-2b benchmarks. We also demonstrate that they produce intuitive descriptions of materials and Internet images.