Benchmarking Aggression Identification in Social Media.

TLDR

The shared task aimed to develop a classifier distinguishing overtly aggressive, covertly aggressive, and non‑aggressive social‑media posts. Participants trained on 15,000 annotated Facebook posts/comments in Hindi (Roman and Devanagari) and English, then tested on separate Facebook and other‑platform sets; 130 teams registered, 30 submitted runs, and 20 system papers were included. The best system achieved a weighted F‑score of 0.64 on the Facebook test sets and 0.60/0.50 on the surprise set for English/Hindi, illustrating the task’s difficulty and strong community engagement.

Abstract

In this paper, we present the report and findings of the Shared Task on Aggression Identification organised as part of the First Workshop on Trolling, Aggression and Cyberbullying (TRAC - 1) at COLING 2018. The task was to develop a classifier that could discriminate between Overtly Aggressive, Covertly Aggressive, and Non-aggressive texts. For this task, the participants were provided with a dataset of 15,000 aggression-annotated Facebook Posts and Comments each in Hindi (in both Roman and Devanagari script) and English for training and validation. For testing, two different sets - one from Facebook and another from a different social media - were provided. A total of 130 teams registered to participate in the task, 30 teams submitted their test runs, and finally 20 teams also sent their system description paper which are included in the TRAC workshop proceedings. The best system obtained a weighted F-score of 0.64 for both Hindi and English on the Facebook test sets, while the best scores on the surprise set were 0.60 and 0.50 for English and Hindi respectively. The results presented in this report depict how challenging the task is. The positive response from the community and the great levels of participation in the first edition of this shared task also highlights the interest in this topic.