A diffusion model is a class of generative models that learns to reverse a gradual noising process to synthesize data. It trains a denoiser (often a U‑Net) to estimate noise or score functions so that, starting from Gaussian noise, iterative denoising steps recover a clean sample in pixel or latent space. Conditioning (e.g., text, class labels) guides the reverse process to produce desired outputs.
What is Diffusion Model?
Diffusion models define a forward process that progressively corrupts data x0 into xt with Gaussian noise over T steps and a learned reverse process pθ(xt−1|xt, c) that denoises step by step, optionally conditioned on c (text, class, image). Training minimizes objectives equivalent to score matching or noise prediction (ε‑ or v‑prediction). Variants include continuous‑time score‑based SDEs, DDIM non‑Markovian samplers, and DPM‑Solver ODE integrators that reduce steps. Latent diffusion compresses images with an autoencoder and runs diffusion in latent space for efficiency (e.g., Stable Diffusion). Classifier‑free guidance mixes unconditional and conditional predictions to trade fidelity for adherence. Control signals (edges, poses) and adapters (LoRA, ControlNet) steer structure without retraining the base.
Where it’s used and why it matters
Diffusion dominates image generation for its mode coverage, high fidelity, and controllability. It powers text‑to‑image, inpainting/outpainting, image‑to‑image, video, audio, and 3D asset creation. Enterprises use it for creative tooling, product imagery, synthetic data, restoration, and design exploration—balanced against compute cost, content governance, and provenance/watermarking needs.
Types
- DDPM/score‑based SDEs: discrete vs continuous training objectives for denoising/score learning.
- DDIM/DPM‑Solver: fast samplers that cut steps with ODE‑style integration.
- Latent diffusion: diffusion in VAE latents for memory/speed gains.
- Guidance and control: classifier‑free guidance, ControlNet, LoRA adapters for conditioning and structure.
FAQs
- How do diffusion models compare to GANs? Diffusion offers better mode coverage and stability; GANs are faster at inference but can suffer mode collapse.
- How does text conditioning work? A text encoder (e.g., CLIP/Transformer) provides embeddings consumed via cross‑attention in the U‑Net during denoising.
- Why so many steps? Iterative denoising approximates the reverse process; accelerated samplers/distillation reduce steps to tens or even a few.
- Do they handle video/audio? Yes—temporal extensions add 3D (x,y,t) convolution/attention or spectrogram conditioning.
- What are key risks? Copyright/data provenance, unsafe content, bias, and misuse; mitigations include filters, usage policies, and watermarking.