End-to-End Diffusion Latent Optimization Improves Classifier Guidance

TLDR

Classifier guidance uses image‑classifier gradients to steer diffusion models, offering powerful creative control, but existing approaches rely on training noise‑aware models or a one‑step denoising approximation that misaligns gradients and limits effectiveness. This work proposes Direct Optimization of Diffusion Latents (DOODL) to provide plug‑and‑play guidance by directly optimizing latents with respect to classifier gradients. DOODL achieves this by back‑propagating gradients of a pre‑trained classifier on the true generated pixels through an invertible diffusion process, enabling memory‑efficient optimization. DOODL surpasses one‑step classifier guidance on both computational and human evaluation metrics, improving complex prompt generation on DrawBench, expanding Stable Diffusion’s vocabulary with fine‑grained classifiers, enabling image‑conditioned generation via a CLIP visual encoder, and enhancing image aesthetics using an aesthetic scoring network.

Abstract

Classifier guidance—using the gradients of an image classifier to steer the generations of a diffusion model—has the potential to dramatically expand the creative control over image generation and editing. However, currently classifier guidance requires either training new noise-aware models to obtain accurate gradients or using a one-step denoising approximation of the final generation, which leads to misaligned gradients and sub-optimal control. We highlight this approximation's shortcomings and propose a novel guidance method: Direct Optimization of Diffusion Latents (DOODL), which enables plug-and-play guidance by optimizing diffusion latents w.r.t. the gradients of a pre-trained classifier on the true generated pixels, using an invertible diffusion process to achieve memory-efficient backpropagation. Showcasing the potential of more precise guidance, DOODL outperforms one-step classifier guidance on computational and human evaluation metrics across different forms of guidance: using CLIP guidance to improve generations of complex prompts from DrawBench, using fine-grained visual classifiers to expand the vocabulary of Stable Diffusion, enabling image-conditioned generation with a CLIP visual encoder, and improving image aesthetics using an aesthetic scoring network.