A Retrospective Analysis of the Fake News Challenge Stance Detection\n Task

Abstract

The 2017 Fake News Challenge Stage 1 (FNC-1) shared task addressed a stance\nclassification task as a crucial first step towards detecting fake news. To\ndate, there is no in-depth analysis paper to critically discuss FNC-1's\nexperimental setup, reproduce the results, and draw conclusions for\nnext-generation stance classification methods. In this paper, we provide such\nan in-depth analysis for the three top-performing systems. We first find that\nFNC-1's proposed evaluation metric favors the majority class, which can be\neasily classified, and thus overestimates the true discriminative power of the\nmethods. Therefore, we propose a new F1-based metric yielding a changed system\nranking. Next, we compare the features and architectures used, which leads to a\nnovel feature-rich stacked LSTM model that performs on par with the best\nsystems, but is superior in predicting minority classes. To understand the\nmethods' ability to generalize, we derive a new dataset and perform both\nin-domain and cross-domain experiments. Our qualitative and quantitative study\nhelps interpreting the original FNC-1 scores and understand which features help\nimproving performance and why. Our new dataset and all source code used during\nthe reproduction study are publicly available for future research.\n