2023 · 67 citations · 12 references
ComputingComputer ScienceKnowledge RepresentationLegal ReasoningNatural Language ProcessingDeductive ReasoningLegal InferenceQuestion AnsweringFew-shot LearningLegislative Text ComprehensionNatural Language InferencePrompting
Statutory reasoning is the task of reasoning with facts and statutes, which are rules written in natural language by a legislature. It is a basic legal skill. In this paper we explore the capabilities of the most capable GPT-3 model, text-davinci-003, on an established statutory-reasoning dataset called SARA. We consider a variety of approaches, including dynamic few-shot prompting, chain-of-thought prompting, and zero-shot prompting. While we achieve results with GPT-3 that are better than the previous best published results, we also identify several types of clear errors it makes. We investigate why these errors happen. We discover that GPT-3 has imperfect prior knowledge of the actual U.S. statutes on which SARA is based. More importantly, we create simple synthetic statutes, which GPT-3 is guaranteed not to have seen during training. We find GPT-3 performs poorly at answering straightforward questions about these simple synthetic statutes.
12
Jonathan H. Choi, Kristin E. Hickman, Amy Monahan et al. · SSRN Electronic Journal · 2023 · 491 citations · Full text
Education, Educational Assessment, Educational Technology +15
Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
Denny Zhou, Nathanael Schärli, Le Hou et al. · arXiv (Cornell University) · 2022 · 322 citations · Full text
STaR: Bootstrapping Reasoning With Reasoning
Eric Zelikman, Yuhuai Wu · arXiv (Cornell University) · 2022 · 116 citations · Full text