On finding duplication and near-duplication in large software systems

TLDR

The study introduces dup, a tool for detecting duplication or near‑duplication in software systems. Dup identifies identical code blocks, near‑duplicates with systematic variable and constant substitutions, and longer sections differing by minor edits. Experiments on millions of lines from two large systems demonstrate that dup is both effective and fast, enabling tasks such as refactoring, reengineering, documentation, and debugging.

Abstract

This paper describes how a program called dup can be used to locate instances of duplication or near-duplication in a software system. Dup reports both textually identical sections of code and sections that are the same textually except for systematic substitution of one set of variable names and constants for another. Further processing locates longer sections of code that are the same except for other small modifications. Experimental results from running dup on millions of lines from two large software systems show dup to be both effective at locating duplication and fast. Applications could include identifying sections of code that should be replaced by procedures, elimination of duplication during reengineering of the system, redocumentation to include references to copies, and debugging.