Evaluating Large Language Models Trained on Code

TLDR

The authors introduce Codex, a GPT model fine‑tuned on public GitHub code to study its Python code‑writing abilities, and discuss its broader safety, security, and economic implications. Codex was trained on GitHub code and evaluated on the HumanEval benchmark—a new docstring‑to‑code dataset—where it solved 28.8% of the problems. Codex powers GitHub Copilot, achieves 28.8% accuracy on HumanEval versus 0% for GPT‑3 and 11.4% for GPT‑J, and with repeated sampling reaches 70.2% success, though it struggles with long operation chains and variable binding.

Abstract

We introduce Codex, a GPT language model fine-tuned on publicly available code from GitHub, and study its Python code-writing capabilities. A distinct production version of Codex powers GitHub Copilot. On HumanEval, a new evaluation set we release to measure functional correctness for synthesizing programs from docstrings, our model solves 28.8% of the problems, while GPT-3 solves 0% and GPT-J solves 11.4%. Furthermore, we find that repeated sampling from the model is a surprisingly effective strategy for producing working solutions to difficult prompts. Using this method, we solve 70.2% of our problems with 100 samples per problem. Careful investigation of our model reveals its limitations, including difficulty with docstrings describing long chains of operations and with binding operations to variables. Finally, we discuss the potential broader impacts of deploying powerful code generation technologies, covering safety, security, and economics.