2023 · 32 citations · 27 references
ComputingComputer ScienceForecastingMachine LearningNatural Language ProcessingKnowledge GraphsLanguage ModelingLanguage ModelsNatural Language UnderstandingTemporal Knowledge RepresentationTime Series PredictionHeuristicsIn-context LearningKnowledge Graph Representation LearningLanguage Model PretrainingLarge Language ModelsNeural Language ModelsPrior Related KnowledgeSemantic Information
Temporal knowledge graph (TKG) forecasting benchmarks challenge models to predict future facts using knowledge of past facts. In this paper, we develop an approach to use in-context learning (ICL) with large language models (LLMs) for TKG forecasting. Our extensive evaluation compares diverse baselines, including both simple heuristics and state-of-the-art (SOTA) supervised models, against pre-trained LLMs across several popular benchmarks and experimental settings. We observe that naive LLMs perform on par with SOTA models, which employ carefully designed architectures and supervised training for the forecasting task, falling within the (-3.6%, +1.5%) Hits@1 margin relative to the median performance. To better understand the strengths of LLMs for forecasting, we explore different approaches for selecting historical facts, constructing prompts, controlling information propagation, and parsing outputs into a probability distribution. A surprising finding from our experiments is that LLM performance endures (±0.4% Hit@1) even when semantic information is removed by mapping entities/relations to arbitrary numbers, suggesting that prior semantic knowledge is unnecessary; rather, LLMs can leverage the symbolic patterns in the context to achieve such a strong performance. Our analysis also reveals that ICL enables LLMs to learn irregular patterns from the historical context, going beyond frequency and recency biases
27
Transformers: State-of-the-Art Natural Language Processing
Thomas Wolf, Lysandre Debut, Victor Sanh et al. · 2020 · 8K citations · Full text
LLaMA: Open and Efficient Foundation Language Models
Hugo Touvron, Thibaut Lavril, Gautier Izacard et al. · arXiv (Cornell University) · 2023 · 3.9K citations · Full text
PaLM: Scaling Language Modeling with Pathways
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin et al. · arXiv (Cornell University) · 2022 · 2.1K citations · Full text
Automatic time series forecasting: the forecast package for R
Rob J. Hyndman, Yeasmin Khandakar · 2017 · 1K citations · Full text