Active Learning with Feedback on Features and Instances

TLDR

Feature feedback can complement traditional active learning in applications such as news filtering, e‑mail classification, and personalization, where the human teacher can have significant knowledge on feature relevance. The study extends the active learning framework to incorporate feedback on features in addition to labeling instances, aiming to assess the impact of feature selection and human feedback on text categorization. The authors devise an algorithm that interleaves feature and document labeling, extending the traditional active learning framework to include feature feedback and accelerating learning in simulation experiments. Experiments demonstrate that feature re‑weighting guided by human feedback improves classifier performance beyond traditional active learning, with humans identifying over 50% of the most relevant features and labeling a feature taking far less time than labeling a document.

Abstract

We extend the traditional active learning framework to include feedback on features in addition to labeling instances, and we execute a careful study of the effects of feature selection and human feedback on features in the setting of text categorization. Our experiments on a variety of categorization tasks indicate that there is significant potential in improving classifier performance by feature re-weighting, beyond that achieved via membership queries alone (traditional active learning) if we have access to an oracle that can point to the important (most predictive) features. Our experiments on human subjects indicate that human feedback on feature relevance can identify a sufficient proportion of the most relevant features (over 50% in our experiments). We find that on average, labeling a feature takes much less time than labeling a document. We devise an algorithm that interleaves labeling features and documents which significantly accelerates standard active learning in our simulation experiments. Feature feedback can complement traditional active learning in applications such as news filtering, e-mail classification, and personalization, where the human teacher can have significant knowledge on the relevance of features.