Controllable Diverse Sampling Through Pullback Directions

Summary

Conditional diffusion models can produce images that closely match a prompt, but different random seeds often yield visually similar results. Existing diversity-enhancement methods may add optimization or sampling overhead, while stronger perturbations can hurt image quality and prompt alignment. The paper introduces a training-free approach based on the local pullback geometry of the mapping from conditioning-embedding space to image space. It identifies directions where small changes to the conditioning signal produce the greatest output sensitivity, then steers separate sampling trajectories along those directions. The goal is to increase variation while keeping generations consistent with the original conditioning. Experiments on MS-COCO and BrushNet report improved diversity for prompt-conditioned image generation and inpainting.