DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuning

TLDR

Diffusion models generate high‑quality images, yet adapting large pre‑trained versions to new domains remains a challenging problem. This work introduces DiffFit, a parameter‑efficient fine‑tuning method that enables rapid domain adaptation of large diffusion models. DiffFit fine‑tunes only the bias terms and newly added scaling factors in selected layers, with theoretical analysis supporting the effectiveness of these scaling factors. DiffFit delivers a 2× training speed‑up, stores only 0.12 % of the parameters, matches or surpasses full fine‑tuning on eight downstream datasets, and sets a new ImageNet 512×512 FID of 3.02 after just 25 epochs, achieving 30× greater efficiency than the nearest competitor.

Abstract

Diffusion models have proven to be highly effective in generating high-quality images. However, adapting large pre-trained diffusion models to new domains remains an open challenge, which is critical for real-world applications. This paper proposes DiffFit, a parameter-efficient strategy to fine-tune large pre-trained diffusion models that enable fast adaptation to new domains. DiffFit is embarrassingly simple that only fine-tunes the bias term and newly-added scaling factors in specific layers, yet resulting in significant training speed-up and reduced model storage costs. Compared with full fine-tuning, DiffFit achieves 2× training speed-up and only needs to store approximately 0.12% of the total model parameters. Intuitive theoretical analysis has been provided to justify the efficacy of scaling factors on fast adaptation. On 8 downstream datasets, DiffFit achieves superior or competitive performances compared to the full fine-tuning while being more efficient. Remarkably, we show that DiffFit can adapt a pre-trained low-resolution generative model to a high-resolution one by adding minimal cost. Among diffusion-based methods, DiffFit sets a new state-of-the-art FID of 3.02 on ImageNet 512×512 benchmark by fine-tuning only 25 epochs from a public pre-trained ImageNet 256×256 checkpoint while being 30× more training efficient than the closest competitor.