Learning a meta-level prior for feature relevance from multiple related tasks

TLDR

Feature selection is crucial for generalization, yet most algorithms assume all features are equally likely to be relevant. The study aims to use transfer learning across related tasks to build an informative prior on feature relevance. The authors model feature relevance as a function of meta‑features using hyperparameters called meta‑priors, and propose a convex optimization algorithm that jointly learns these meta‑priors and feature weights across related tasks, enabling transfer of relevance information even when tasks have nonoverlapping features or varying relevance. The method improves predictive performance on two real datasets: a collaborative filtering rating prediction task and a verb argument classification task.

Abstract

In many prediction tasks, selecting relevant features is essential for achieving good generalization performance. Most feature selection algorithms consider all features to be a priori equally likely to be relevant. In this paper, we use transfer learning---learning on an ensemble of related tasks---to construct an informative prior on feature relevance. We assume that features themselves have meta-features that are predictive of their relevance to the prediction task, and model their relevance as a function of the meta-features using hyperparameters (called meta-priors). We present a convex optimization algorithm for simultaneously learning the meta-priors and feature weights from an ensemble of related prediction tasks which share a similar relevance structure. Our approach transfers the "meta-priors" among different tasks, which makes it possible to deal with settings where tasks have nonoverlapping features or the relevance of the features vary over the tasks. We show that learning feature relevance improves performance on two real data sets which illustrate such settings: (1) predicting ratings in a collaborative filtering task, and (2) distinguishing arguments of a verb in a sentence.