Generating Adversarial Examples for Holding Robustness of Source Code Processing Models

TLDR

Automated source code processing is essential in software development, yet state‑of‑the‑art deep learning models are vulnerable to adversarial attacks, a problem exacerbated by the structured nature of programming languages. The authors propose a Metropolis‑Hastings sampling‑based identifier renaming technique, \fullmethod (\method), to generate adversarial examples for deep learning models that process source code. The technique applies Metropolis‑Hastings sampling to systematically rename identifiers in source code, creating adversarial examples that challenge DL models. Evaluation on a functionality classification benchmark shows that \method effectively generates adversarial source code, and adversarial training with these examples improves model robustness and performance.

Abstract

Automated processing, analysis, and generation of source code are among the key activities in software and system lifecycle. To this end, while deep learning (DL) exhibits a certain level of capability in handling these tasks, the current state-of-the-art DL models still suffer from non-robust issues and can be easily fooled by adversarial attacks.Different from adversarial attacks for image, audio, and natural languages, the structured nature of programming languages brings new challenges. In this paper, we propose a Metropolis-Hastings sampling-based identifier renaming technique, named \fullmethod (\method), which generates adversarial examples for DL models specialized for source code processing. Our in-depth evaluation on a functionality classification benchmark demonstrates the effectiveness of \method in generating adversarial examples of source code. The higher robustness and performance enhanced through our adversarial training with \method further confirms the usefulness of DL models-based method for future fully automated source code processing.