SemEval-2023 Task 2: Fine-grained Multilingual Named Entity Recognition (MultiCoNER 2)

TLDR

SemEval‑2023 Task 2 focused on fine‑grained multilingual named‑entity recognition, a popular challenge within the SemEval community. The task aimed to evaluate methods for detecting complex entity types such as WRITTENWORK, VEHICLE, and MUSICALGRP across 12 languages in monolingual, multilingual, and noisy settings. Evaluation was conducted on the MultiCoNER V2 dataset, which contains 2.2 million instances in Bangla, Chinese, English, Farsi, French, German, Hindi, Italian, Portuguese, Spanish, Swedish, and Ukrainian. The competition received 842 submissions from 47 teams, with 34 system papers; results revealed that media titles and product names were the most difficult, external‑knowledge‑augmented transformers performed best, noisy data caused an average 10 % drop, and further research is needed to improve robustness on complex entities.

Abstract

We present the findings of SemEval-2023 Task 2 on Fine-grained Multilingual Named Entity Recognition (MultiCoNER 2). Divided into 13 tracks, the task focused on methods to identify complex fine-grained named entities (like WRITTENWORK, VEHICLE, MUSICALGRP) across 12 languages, in both monolingual and multilingual scenarios, as well as noisy settings. The task used the MultiCoNER V2 dataset, composed of 2.2 million instances in Bangla, Chinese, English, Farsi, French, German, Hindi, Italian., Portuguese, Spanish, Swedish, and Ukrainian. MultiCoNER 2 was one of the most popular tasks of SemEval-2023. It attracted 842 submissions from 47 teams, and 34 teams submitted system papers. Results showed that complex entity types such as media titles and product names were the most challenging. Methods fusing external knowledge into transformer models achieved the best performance, and the largest gains were on the Creative Work and Group classes, which are still challenging even with external knowledge. Some fine-grained classes proved to be more challenging than others, such as SCIENTIST, ARTWORK, and PRIVATECORP. We also observed that noisy data has a significant impact on model performance, with an average drop of 10% on the noisy subset. The task highlights the need for future research on improving NER robustness on noisy data containing complex entities.