ViDiT — Learn Once, Edit Anywhere

TL;DR

Text prompts are a descriptive bottleneck — many fine-grained edits are obvious in a pair of images but hard to put into words. ViDiT captures the visual delta directly from a few before-and-after pairs as a single continuous direction, then applies it zero-shot to any image with granular control over edit intensity.

01Abstract

The rapid advancement of diffusion models has enabled the generation of high-fidelity images from textual prompts, yet achieving precise, disentangled control over specific attributes remains a significant challenge. A fundamental limitation arises because visual differences between images are often far more descriptive and nuanced than what can be captured through human-crafted text descriptions. To address this, we introduce ViDiT (Visual Direction Transfer for Diffusion), a framework that expands the editing vocabulary by capturing latent semantics directly from image-edit pairs. ViDiT learns the underlying transformation by optimizing a single, global, and continuous editing direction from a small set of "before-and-after" examples, transferring visual changes into the diffusion model's conditioning space. Operating on a "Learn Once" principle, it eliminates the need for model fine-tuning or expensive per-image optimization at inference. Once learned, these continuous directions enable "Edit Anywhere" capabilities — highly disentangled manipulations of facial features, animal attributes, or artistic styles applied to any image in a zero-shot manner with granular control over edit intensity. Quantitative and qualitative evaluations demonstrate that ViDiT outperforms existing text-based editing methods in maintaining input faithfulness while achieving precise, scalable attribute control.

02Method

Given image-edit pairs from any source, ViDiT optimizes a single direction d in the diffusion model's conditioning space — with the base model and encoder kept entirely frozen.

ViDiT framework: semantic alignment, edit-pair sampling, and latent alignment

Framework. A semantic alignment loss uses a frozen vision-language (CLIP) encoder to place d in the right conceptual neighborhood, while a latent alignment loss leverages the denoiser's dense, spatially-aligned predictions to preserve fine-grained identity. Only d is updated.

Collect image-edit pairs

A small set of before-and-after pairs (x, x′) from any editing source — GANs, domain-adaptation models, or diffusion editors — encodes the precise visual delta, with no linguistic ambiguity.

Learn one direction

A dual objective optimizes a single continuous direction d: semantic alignment locks in the right concept; latent alignment sharpens spatial precision. The model and encoder stay frozen.

Edit anywhere, zero-shot

At inference, d is injected additively into classifier-free guidance with a scale λ_e for continuous, bidirectional control — and multiple directions compose without retraining.

03Algorithm

Learning a direction is a one-time optimization per concept. Only $\mathbf{d}$ is updated — the diffusion model $\epsilon_\theta$ and the CLIP encoder $E_I$ stay completely frozen.

Learning direction $\mathbf{d}$ with ViDiT

Require: frozen diffusion model $\epsilon_\theta(x_t, c)$, frozen CLIP image encoder $E_I(x)$, $N$ image-edit pairs $\{(x_i, x_i')\}$ from any editing source, randomly initialized direction $\mathbf{d}$, learning rate $\eta$.

1 while training do for each pair $(x_i, x_i')$ do

2 sample $\epsilon \sim \mathcal{N}(0, 1)$, $t \sim \mathcal{U}(1, T)$ ↳ a fresh noise sample and timestep each step

3 $x_{i,t} = x_i + \alpha^t \epsilon$, $x_{i,t}' = x_i' + \alpha^t \epsilon$ ↳ add the same noise to both images, so only the edit distinguishes them

4 $\mathcal{L}_\text{latent} = -\lVert \epsilon_\theta(x_{i,t}', \mathbf{d}) - \epsilon_\theta(x_{i,t}, \mathbf{d}) \rVert_2^2$ ↳ latent alignment: push edited vs. original apart in the denoiser's dense prediction space → preserves fine detail

5 $\mathcal{L}_\text{sem} = 1 - \text{cossim}(E_I(x_i'), \mathbf{d}) + \text{cossim}(E_I(x_i), \mathbf{d})$ ↳ semantic alignment: pull $\mathbf{d}$ toward the edited image, away from the original, in CLIP space → locks in the concept

6 $\mathbf{d} \leftarrow \mathbf{d} - \eta\, \nabla_{\mathbf{d}} (\mathcal{L}_\text{sem} + \mathcal{L}_\text{latent})$ ↳ update only $\mathbf{d}$; the base model and encoder remain frozen

04Continuous control

Because the edit is a continuous direction $\mathbf{d}$, the scale $\lambda_e$ turns a single learned concept into a smooth, bidirectional dial — from removing the attribute ($-\lambda_e$) to amplifying it ($+\lambda_e$) — all from the same direction, on the same image.

Edit interpolation: Beard and Age attributes scaled from minus to plus on generated portraits

Edit interpolation on generated images. Sweeping $\lambda_e$ for the Beard (top) and Age (bottom) directions on Stable-Diffusion portraits. The identity and artistic style are preserved throughout — only the targeted attribute moves along the direction.

05Results

Fine-grained directions drawn from diverse supervision sources, transferred zero-shot to in-the-wild images.

ViDiT directions across faces, animals, artistic styles, and real portraits

(a) Facial attributes (Bald, Gender, White Hair) from StyleGAN2/FFHQ · (b) Animal attributes (Lynx, Lion, Skin Color) from StyleGAN2/AFHQ · (c) Artistic styles (Ukiyoe, Pixar) from StyleGAN-NADA · (d) Semantic edits (Gender, Beard) from Prompt2Prompt. Across all cases, edits stay disentangled and preserve identity, background, and style.

06Gallery

The same learned directions transfer across wildly different image distributions — synthetic generations, classical paintings, and real in-the-wild photographs — while preserving identity, background, and style.

Editing results on Stable Diffusion generated images

Generated Images. ViDiT applies the desired semantic without inversion, while preserving the stylization of the generated image.

Artistic Paintings. Edits are applied without altering the overall artistic style of the image (e.g. an oil-painting look is retained).

Editing results on real in-the-wild photographs

Real Images. ViDiT generalizes beyond face-centered crops to diverse scenes and subjects, preserving identity and background.