TUbilio (Technical University of Darmstadt) · 2018 · 68 citations · 36 references
ComputingComputer ScienceFeature EngineeringMachine LearningNatural Language ProcessingClassificationDeep LearningMisinformation DetectionNews MediaRepresentation LearningWord EmbeddingsAutomated Text ClassificationDataset CurationDataset DescriptionsEvaluation Metric DesignFactuality DetectionFake News Analysis
The 2017 Fake News Challenge Stage 1 (FNC-1) shared task addressed a stance classification task as a crucial first step towards detecting fake news. To date, there is no in-depth analysis paper to critically discuss FNC-1's experimental setup, reproduce the results, and draw conclusions for next-generation stance classification methods. In this paper, we provide such an in-depth analysis for the three top-performing systems. We first find that FNC-1's proposed evaluation metric favors the majority class, which can be easily classified, and thus overestimates the true discriminative power of the methods. Therefore, we propose a new F1-based metric yielding a changed system ranking. Next, we compare the features and architectures used, which leads to a novel feature-rich stacked LSTM model that performs on par with the best systems, but is superior in predicting minority classes. To understand the methods' ability to generalize, we derive a new dataset and perform both in-domain and cross-domain experiments. Our qualitative and quantitative study helps interpreting the original FNC-1 scores and understand which features help improving performance and why. Our new dataset and all source code used during the reproduction study are publicly available for future research.
36
Indexing by latent semantic analysis
Scott Deerwester, Susan Dumais, George W. Furnas et al. · Journal of the American Society for Information Science · 1990 · 12.8K citations
Fake News Detection on Social Media
Kai Shu, Amy Sliva, Suhang Wang et al. · ACM SIGKDD Explorations Newsletter · 2017 · 3.1K citations