Small Vision Transformers: Diffusion Models as Image Classifiers
A diffusion model that was never trained to classify gets 86.7% on CIFAR-10 with 1,000 calls. A joint image and label model gets 87.2% with one call.
Lead AI Engineer · Writing since 2016
I write about reinforcement learning, transformers, multi-agent systems, and the tools I build along the way.
A diffusion model that was never trained to classify gets 86.7% on CIFAR-10 with 1,000 calls. A joint image and label model gets 87.2% with one call.
RL fine-tuning of a DiT with a ViT reward and a held-out judge. Data regularization wins on reward and FID; a 10x learning rate shows reward hacking.
A shortcut model learns big steps from pairs of small ones, with no JVP: FID-10k 22.5 at 4 calls and near 16.8 from 16 to 128 calls, against MeanFlow.
MeanFlow learns average velocities, so one model call makes an image: FID-10k 39.9 at NFE 1 and 17.5 at NFE 2. The first run diverged; here is why.
Same DiT, data and 200 epochs: flow matching gets FID-10k 20.2 with 32 model calls. DDPM needs 256 calls to reach 19.9. Euler and Heun samplers compared.
A 9.8 M parameter Diffusion Transformer learns to draw CIFAR-10 images in 86 minutes on one GPU. The loss is flat long before the samples look right.
ViT-B/16 on Flowers-102 with 10 images per class: 39.0% from scratch, 98.4% with a linear probe and 99.2% with LoRA, plus a check for leaked test images.
Read in order
Multi-part guides that build on each other, start to finish.
Subscribe to the RSS feed, or follow along on GitHub, LinkedIn and the RL Factory YouTube channel.