FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation

TLDR

The authors introduce a 70,000‑sentence, 100‑relation dataset for few‑shot relation classification and identify multiple avenues for future research. They construct the dataset by distant supervision and crowdworker filtering, then apply recent few‑shot learning models to relation classification and evaluate them comprehensively. Experiments reveal that current few‑shot models perform poorly compared to humans, require diverse reasoning skills, and demonstrate that the task remains unsolved.

Abstract

We present a Few-Shot Relation Classification Dataset (dataset), consisting of 70, 000 sentences on 100 relations derived from Wikipedia and annotated by crowdworkers. The relation of each sentence is first recognized by distant supervision methods, and then filtered by crowdworkers. We adapt the most recent state-of-the-art few-shot learning methods for relation classification and conduct thorough evaluation of these methods. Empirical results show that even the most competitive few-shot learning models struggle on this task, especially as compared with humans. We also show that a range of different reasoning skills are needed to solve our task. These results indicate that few-shot relation classification remains an open problem and still requires further research. Our detailed analysis points multiple directions for future research.