When does pretraining help?

TLDR

Self‑supervised learning has advanced NLP, yet it remains unclear when domain‑specific pretraining is beneficial, especially in legal language where few substantial gains have been documented. The authors hypothesize that the lack of gains stems from legal NLP tasks being too easy to trigger domain‑pretraining benefits. They introduce CaseHOLD, a 53,000‑question multiple‑choice dataset on case holdings, and evaluate performance gains from general and domain‑pretrained Transformers on this and other legal tasks. Domain‑pretrained Transformers achieve the largest gains—7.2 % F1 on CaseHOLD (12 % over BERT)—and performance improvements correlate with task similarity to the pretraining corpus, indicating when pretraining is worthwhile and that legal language embeddings differ from general language.

Abstract

While self-supervised learning has made rapid advances in natural language processing, it remains unclear when researchers should engage in resource-intensive domain-specific pretraining (domain pretraining). The law, puzzlingly, has yielded few documented instances of substantial gains to domain pretraining in spite of the fact that legal language is widely seen to be unique. We hypothesize that these existing results stem from the fact that existing legal NLP tasks are too easy and fail to meet conditions for when domain pretraining can help. To address this, we first present CaseHOLD (Case <u>H</u>oldings <u>O</u>n <u>L</u>egal <u>D</u>ecisions), a new dataset comprised of over 53,000+ multiple choice questions to identify the relevant holding of a cited case. This dataset presents a fundamental task to lawyers and is both legally meaningful and difficult from an NLP perspective (F1 of 0.4 with a BiLSTM baseline). Second, we assess performance gains on CaseHOLD and existing legal NLP datasets. While a Transformer architecture (BERT) pretrained on a general corpus (Google Books and Wikipedia) improves performance, domain pretraining (on a corpus of ≈3.5M decisions across all courts in the U.S. that is larger than BERT's) with a custom legal vocabulary exhibits the most substantial performance gains with CaseHOLD (gain of 7.2% on F1, representing a 12% improvement on BERT) and consistent performance gains across two other legal tasks. Third, we show that domain pretraining may be warranted when the task exhibits sufficient similarity to the pretraining corpus: the level of performance increase in three legal tasks was directly tied to the domain specificity of the task. Our findings inform when researchers should engage in resource-intensive pretraining and show that Transformer-based architectures, too, learn embeddings suggestive of distinct legal language.