DeCAF: A Deep Convolutional Activation Feature for Generic Visual Recognition

TLDR

Generic visual tasks often differ from the original training set and lack sufficient labeled data, making conventional deep architecture adaptation difficult. The study evaluates whether deep convolutional activation features can be repurposed for generic visual tasks and releases DeCAF to enable such experimentation. The authors investigate and visualize semantic clustering of these features across scene recognition, domain adaptation, and fine‑grained tasks, and provide an open‑source implementation with network parameters. They demonstrate that features from various network levels form a fixed representation that significantly outperforms state‑of‑the‑art on several vision challenges.

Abstract

We evaluate whether features extracted from the activation of a deep convolutional network trained in a fully supervised fashion on a large, fixed set of object recognition tasks can be re-purposed to novel generic tasks. Our generic tasks may differ significantly from the originally trained tasks and there may be insufficient labeled or unlabeled data to conventionally train or adapt a deep architecture to the new tasks. We investigate and visualize the semantic clustering of deep convolutional features with respect to a variety of such tasks, including scene recognition, domain adaptation, and fine-grained recognition challenges. We compare the efficacy of relying on various network levels to define a fixed feature, and report novel results that significantly outperform the state-of-the-art on several important vision challenges. We are releasing DeCAF, an open-source implementation of these deep convolutional activation features, along with all associated network parameters to enable vision researchers to be able to conduct experimentation with deep representations across a range of visual concept learning paradigms.