Semantic Discrepancy

Overview概览

When a text-to-image diffusion model is used for image-to-image translation — turning a photo into a painting, say — everything hinges on an initial noise that keeps the original image's information, and the common choice, SDEdit, simply adds random noise to the image. We find that this causes “semantic discrepancy”: the model misreads relations in the image and invents content that was never there, because the noise breaks the image's semantic integrity and makes U-Net upsampling link unrelated regions. We propose a training-free, plug-and-play fix for Stable Diffusion that exploits the determinism of DDIM inversion: a single inversion step of the reference image supplies accurate features and correlations, which replace the faulty ones in generation. Across translation tasks on COCO, ImageNet and ImageNet-R, it reduces semantic discrepancy and improves fidelity, outperforming PnP and similar methods with fewer priors while running 11.2× faster.

用文生图扩散模型做图像转换（比如把照片变成油画）时，关键是找到既保留原图信息、又便于转换的初始噪声，常用的 SDEdit 做法是直接给原图加随机噪声。我们发现这会引起“语义偏差”：模型误解原图中物体之间的关系，凭空生成原图里没有的内容；剖析 Stable Diffusion 的网络后，我们定位到原因是加噪破坏了图像的语义完整性，使 U-Net 上采样时在不相关的区域之间建立了错误关联。我们提出一种免训练、即插即用的方法，利用 DDIM 反演的确定性，只需对参考图做一步反演，就能得到正确的特征和关联，用来替换生成过程中的错误部分。在 COCO、ImageNet 和 ImageNet-R 的多种图像转换任务上，该方法有效缓解了语义偏差、提升了保真度，用更少的先验超过 PnP 等方法，速度快 11.2 倍。

Highlights要点

1. The cause: added noise breaks the image's semantics, so U-Net upsampling links unrelated regions.定位病因：加噪破坏了图像的语义完整性，让 U-Net 上采样在不相关的区域之间建立错误关联。
2. Training-free and plug-and-play: one DDIM inversion step of the reference image supplies the right features and correlations.免训练、即插即用：对参考图做一步 DDIM 反演，就能得到正确的特征和关联。
3. Higher fidelity on COCO, ImageNet and ImageNet-R, and 11.2× faster than PnP and similar methods.在 COCO、ImageNet 和 ImageNet-R 上保真度更高，比 PnP 等方法快 11.2 倍。

Citation引用

@article{yuan2025semantic,
  title   = {Dissecting and Mitigating Semantic Discrepancy in Stable Diffusion for Image-to-Image Translation},
  author  = {Yifan Yuan and Guanqun Yang and James Z. Wang and Hui Zhang and Hongming Shan and Feiyue Wang and Junping Zhang},
  journal = {IEEE/CAA Journal of Automatica Sinica},
  volume  = {12},
  number  = {4},
  pages   = {705--718},
  year    = {2025},
  doi     = {10.1109/JAS.2024.124800}
}