2023 · 12 citations · 21 references
ComputingComputer SciencePhysicsArtificial IntelligenceKnowledge RepresentationNatural Language ProcessingAutomated ReasoningCommonsense ReasoningLanguage ModelingLanguage ModelsNatural Language UnderstandingPractical ReasoningAssessment TemplatesAttribute ReasoningAttributesDomain AdaptationLarge Language ModelsObject PropertiesObjectsPhysical Reasoning
Large Language Models (LLMs), through their contextualized representations, have been empirically proven to encapsulate syntactic, semantic, word sense, and common-sense knowledge. However, there has been limited exploration of their physical reasoning abilities, specifically concerning the crucial attributes for comprehending everyday objects. To address this gap, we introduce NEWTON, a repository and benchmark for evaluating the physics reasoning skills of LLMs. Further, to enable domain-specific adaptation of this benchmark, we present a pipeline to enable researchers to generate a variant of this benchmark that has been customized to the objects and attributes relevant for their application. The NEWTON repository comprises a collection of 2800 object-attribute pairs, providing the foundation for generating infinite-scale assessment templates. The NEWTON benchmark consists of 160K QA questions, curated using the NEWTON repository to investigate the physical reasoning capabilities of several mainstream language models across foundational, explicit, and implicit reasoning tasks. Through extensive empirical analysis, our results highlight the capabilities of LLMs for physical reasoning. We find that LLMs like GPT-4 demonstrate strong reasoning capabilities in scenario-based tasks but exhibit less consistency in object-attribute reasoning compared to humans (50% vs. 84%). Furthermore, the NEWTON platform demonstrates its potential for evaluating and enhancing language models, paving the way for their integration into physically grounded settings, such as robotic manipulation. Project site: https://newtonreasoning.github.io
21
LoRA: Low-Rank Adaptation of Large Language Models
J. Edward Hu, Yelong Shen, Phillip Wallis et al. · arXiv (Cornell University) · 2021 · 2.5K citations · Full text
A Survey of Large Language Models
Wayne Xin Zhao, Kun Zhou, Junyi Li et al. · Frontiers of Computer Science · 2026 · 1.4K citations · Full text
Scaling Instruction-Finetuned Language Models
Hyung Won Chung, Le Hou, Shayne Longpre et al. · arXiv (Cornell University) · 2022 · 1.2K citations · Full text