Taskonomy: Disentangling Task Transfer Learning

TLDR

Visual tasks are interrelated, forming a structure that underlies transfer learning and enables efficient reuse of supervision across related tasks. The study proposes a fully computational approach to model the structure of visual tasks and examines its consequences. The authors build a latent‑space model of 26 visual tasks, identify first‑ and higher‑order transfer dependencies, and provide tools—including a solver—to compute and probe the resulting taxonomic structure. They produce a computational taxonomic map revealing nontrivial relationships among tasks, which can be exploited to reduce labeled‑data requirements.

Abstract

Do visual tasks have a relationship, or are they unrelated? For instance, could having surface normals simplify estimating the depth of an image? Intuition answers these questions positively, implying existence of a structure among visual tasks. Knowing this structure has notable values; it is the concept underlying transfer learning and provides a principled way for identifying redundancies across tasks, e.g., to seamlessly reuse supervision among related tasks or solve many tasks in one system without piling up the complexity. We proposes a fully computational approach for modeling the structure of space of visual tasks. This is done via finding (first and higher-order) transfer learning dependencies across a dictionary of twenty six 2D, 2.5D, 3D, and semantic tasks in a latent space. The product is a computational taxonomic map for task transfer learning. We study the consequences of this structure, e.g. nontrivial emerged relationships, and exploit them to reduce the demand for labeled data. We provide a set of tools for computing and probing this taxonomical structure including a solver users can employ to find supervision policies for their use cases.