Enhancing Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies

TLDR

In-context learning (ICL) has emerged as a new approach to various natural language processing tasks, utilizing large language models (LLMs) to make predictions based on context supplemented with a few examples or task‑specific instructions. The study aims to extend ICL to question answering tasks that use structured knowledge sources and to improve Text‑to‑SQL systems by exploring diverse prompt design strategies for LLMs. The authors systematically investigate prompt design for Text‑to‑SQL, evaluating demonstration selection methods, instruction formats, and leveraging SQL syntactic structure to retrieve diverse and similar examples, thereby enhancing LLM performance. Their approach, which combines syntactic SQL‑based demonstration retrieval, diversity–similarity selection, and database knowledge augmentation, improves execution accuracy on the Spider dataset by 2.5 points over the state of the art and 5.1 points over the best fine‑tuned system, demonstrating its effectiveness for Text‑to‑SQL.

Abstract

In-context learning (ICL) has emerged as a new approach to various natural language processing tasks, utilizing large language models (LLMs) to make predictions based on context that has been supplemented with a few examples or task-specific instructions. In this paper, we aim to extend this method to question answering tasks that utilize structured knowledge sources, and improve Text-to-SQL systems by exploring various prompt design strategies for employing LLMs. We conduct a systematic investigation into different demonstration selection methods and optimal instruction formats for prompting LLMs in the Text-to-SQL task. Our approach involves leveraging the syntactic structure of an example's SQL query to retrieve demonstrations, and we demonstrate that pursuing both diversity and similarity in demonstration selection leads to enhanced performance. Furthermore, we show that LLMs benefit from database-related knowledge augmentations. Our most effective strategy outperforms the state-of-the-art system by 2.5 points (Execution Accuracy) and the best fine-tuned system by 5.1 points on the Spider dataset. These results highlight the effectiveness of our approach in adapting LLMs to the Text-to-SQL task, and we present an analysis of the factors contributing to the success of our strategy.