Instruction Tuning with GPT-4

TLDR

Prior work has shown that fine‑tuning large language models with machine‑generated instruction‑following data yields remarkable zero‑shot capabilities on new tasks, eliminating the need for human‑written instructions. This study presents the first attempt to use GPT‑4 to generate instruction‑following data for fine‑tuning large language models. The authors generate 52 K English and Chinese instruction‑following examples with GPT‑4 and collect feedback and comparison data from GPT‑4 to support evaluation and reward‑model training. The GPT‑4‑generated data improves zero‑shot performance of instruction‑tuned LLaMA models over prior state‑of‑the‑art data, and the authors release the data and code publicly.

Abstract

Prior work has shown that finetuning large language models (LLMs) using machine-generated instruction-following data enables such models to achieve remarkable zero-shot capabilities on new tasks, and no human-written instructions are needed. In this paper, we present the first attempt to use GPT-4 to generate instruction-following data for LLM finetuning. Our early experiments on instruction-tuned LLaMA models show that the 52K English and Chinese instruction-following data generated by GPT-4 leads to superior zero-shot performance on new tasks to the instruction-following data generated by previous state-of-the-art models. We also collect feedback and comparison data from GPT-4 to enable a comprehensive evaluation and reward model training. We make our data generated using GPT-4 as well as our codebase publicly available.