2024 · 35 citations · 20 references
ComputingComputer ScienceHigh-performance ComputingParallel ComputingProgramming LanguagesScientific ComputingLanguage ModelingModel-driven DevelopmentParallel Programming ModelsPerformance ModelingHigh PerformanceHigh-performance Computing SystemsLanguage Model Fine-tuningLarge Language ModelsParallel ProgrammingPerformance PredictionProgramming Models
Parallel programs in high performance computing (HPC) continue to grow in complexity and scale in the exascale era. The diversity in hardware and parallel programming models make developing, optimizing, and maintaining parallel software even more burdensome for developers. One way to alleviate some of these burdens is with automated development and analysis tools. Such tools can perform complex and/or remedial tasks for developers that increase their productivity and decrease the chance for error. Until recently, such tools for code development and performance analysis have been limited in the complexity of tasks they can perform, especially for parallel programs. However, with recent advancements in language modeling, and the availability of large amounts of open-source code related data, these tools have started to utilize predictive language models to automate more complex tasks. In this paper, we show how large language models (LLMs) can be applied to tasks specific to high performance and scientific codes. We introduce a new dataset of HPC and scientific codes and use it to fine-tune several pre-trained models. We compare several pre-trained LLMs on HPC-related tasks and introduce a new model, HPC-Coder, fine-tuned on parallel codes. In our experiments, we show that this model can auto-complete HPC functions where generic models cannot, decorate for loops with OpenMP pragmas, and model performance changes in scientific application repositories as well as programming competition solutions.
20
Transformers: State-of-the-Art Natural Language Processing
Thomas Wolf, Lysandre Debut, Victor Sanh et al. · 2020 · 8K citations · Full text
Evaluating Large Language Models Trained on Code
arXiv (Cornell University) · 2021 · 1.4K citations · Full text
A Survey of Large Language Models
Wayne Xin Zhao, Kun Zhou, Junyi Li et al. · Frontiers of Computer Science · 2026 · 1.4K citations · Full text
A systematic evaluation of large language models of code
Frank F. Xu, Uri Alon, Graham Neubig et al. · 2022 · 498 citations · Full text
Code Llama: Open Foundation Models for Code
Baptiste Rozière, Jonas Gehring, Fabian Gloeckle et al. · arXiv (Cornell University) · 2023 · 398 citations · Full text