Overview and Discussion of the Competition on Legal Information Extraction/Entailment (COLIEE) 2021

TLDR

The paper provides an overview of the 8th COLIEE competition, summarizing its tasks, participant approaches, evaluation methods, and analysis of results. The competition comprised five tasks covering case law and statute law, including information retrieval and entailment/question‑answering components, with open participation. Eight teams competed in case law tasks and eight in statute law tasks, submitting a total of 16 runs for Task 1, 17 for Task 2, 18 for Tasks 3 and 4, and 12 for Task 5.

Abstract

Abstract We summarize the 8th Competition on Legal Information Extraction and Entailment. In this edition, the competition included five tasks on case law and statute law. The case law component includes an information retrieval Task (Task 1), and the confirmation of an entailment relation between an existing case and an unseen case (Task 2). The statute law component includes an information retrieval Task (Task 3), an entailment/question answering task based on retrieved civil code statutes (Task 4) and an entailment/question answering task without retrieved civil code statutes (Task 5). Participation was open to any group based on any approach. Eight different teams participated in the case law competition tasks, most of them in more than one task. We received results from six teams for Task 1 (16 runs) and 6 teams for Task 2 (17 runs). On the statute law task, there were eight different teams participating, most in more than one task. Six teams submitted a total of 18 runs for Task 3, 6 teams submitted a total of 18 runs for Task 4, and 4 teams submitted a total of 12 runs for Task 5. Here we summarize the approaches, our official evaluation, and analysis on our data and submission results.