Flows Without Steps
Every flow so far was a stack of discrete layers. Now take the limit: infinitely many, infinitesimally small. The stack becomes a velocity field, the ledger becomes an integral, and the awkward training problem gets a fix so simple it powers today's best image and video generators.
From a stack of layers to a field of arrows
A residual-flavored layer nudges each point a little: x ← x + ε·v(x). Stack K of them with ε = 1/K and let K → ∞, and the discrete stack becomes a differential equation. The model is no longer a list of matrices; it is one velocity field v(x, t) — an arrow at every location and every moment — and a point's journey is what you get by following the arrows:
This is a continuous normalizing flow (CNF), introduced with Neural ODEs in 2018. Two of Part 1's house rules now come free rather than by construction. No tearing: solutions of a well-behaved ODE depend continuously on where they started. No folding: two trajectories can never meet, because at a meeting point the field would have to point two ways at once — so invertibility is automatic, and you undo the flow by simply integrating backwards in time. No triangular Jacobians, no masks, no architectural contortions at all: v can be any neural network.
Discrete flows had to be built from invertible pieces. A continuous flow is invertible because trajectories cannot cross — the constraint moves from the architecture into the mathematics, and the network is finally unconstrained.
Divergence: the receipt rate
If layers dissolve into a field, what happens to the log-det ledger? It becomes an integral of a rate. The instantaneous change-of-variables formula says that as a point rides the field, its log-density falls at exactly the rate at which the field spreads space apart there — the divergence, tr(∂v/∂x):
The physical reading is worth pausing on: this is the continuity equation of fluid mechanics, ∂p/∂t + ∇·(pv) = 0 — the statement that probability is a fluid that is transported, never created or destroyed. Part 1's clay has become water, and the whole subject is revealed as transport.
Below is a real continuous flow with an exactly known field: it carries a plain 2-D bell curve into a two-blob mixture, and every arrow, trajectory and density you see is computed from the closed-form velocity — which satisfies the continuity equation to one part in a million. Scrub time, or watch it run.
Turn on divergence and watch where the receipts are paid: red where the field pulls space apart (density thinning), blue where it converges (density concentrating). Notice the arrows never cross and the trails never touch — that is invertibility, visible.
Why nobody could train these — until they could
CNFs were beautiful and nearly unusable. To get one log-likelihood you had to numerically integrate the ODE — dozens to hundreds of network evaluations per sample — and then backpropagate through the solver (or through an adjoint ODE) for every gradient step. Worse, as training sharpened the field, the ODE grew stiff and the solver silently spent more and more steps. Training was measured in GPU-weeks for results discrete flows got in hours.
The fix, arriving around 2022 as flow matching (and rectified flow, and stochastic interpolants — three groups, one idea), is almost embarrassing in hindsight. Stop asking the model to discover a path. Choose the path yourself, then teach the network to name its velocity. Pick a noise sample z and a data sample x, and simply interpolate along the straight line between them:
The magic is in what the regression converges to. Any individual pair's straight line is “wrong” — it ignores every other pair. But the least-squares solution at each point is the conditional average of all the velocities passing through it, and that average field is provably the one whose transport carries noise exactly onto the data distribution. Each training example is a noisy, unbiased hint about the true field; averaging is the whole algorithm.
Why anyone cares: fewer steps at generation time
Training got cheap; generating still means solving an ODE. Here the choice of interpolation path pays a second dividend. If the learned trajectories are nearly straight, a crude solver with very few steps tracks them well — and straight-line interpolation is exactly what encourages that. Push further with rectified flow: generate pairs from the trained model, retrain on those (now better-aligned) pairs, and the paths straighten further, until a handful of steps — sometimes one — suffices.
Below, the same generative task is solved by a curved field and a straightened one, both integrated with the same crude Euler solver. Drag the step budget down and watch which one survives.
This is the entire reason flow matching took over. Stable Diffusion 3, Flux, Meta's Movie Gen and a growing list of video models are trained this way — not because likelihoods matter to them, but because straight paths mean few-step sampling, and few-step sampling is what makes a model shippable.
One picture, two vocabularies
If you have met diffusion models, everything above should feel familiar — and it should, because they are the same object in different clothes. A diffusion model destroys data with noise and learns to reverse that process; its reverse-time dynamics can be written either as a stochastic differential equation or as a deterministic probability-flow ODE. That ODE is a continuous normalizing flow. Diffusion's “noise prediction” and flow matching's “velocity prediction” are related by a change of variables; the practical differences are the choice of path (diffusion's curved variance-preserving schedule vs flow matching's straight line) and the training objective's parameterization.
| Family | Exact likelihood | Training cost | Sampling | Where it shines |
|---|---|---|---|---|
| Discrete flow (RealNVP, Glow) | exact, cheap | low | 1 pass | Density estimation, integration, science. |
| CNF via max-likelihood | exact | very high | ODE solve | Historically important; largely superseded. |
| Flow matching / rectified | available, post-hoc | low | few steps | Today's images, video, audio, molecules. |
| Diffusion | indirect (via its ODE) | low | many steps | The incumbent; converging with the row above. |
So the arc of this series ends where the field is now. Part 1's demand — invertible maps with exact receipts — bought exactness at the price of architectural contortion. The continuous limit removed the contortion. Flow matching removed the training cost. What remains, in the largest generative models running today, is the picture you have had since the first animation: probability, flowing smoothly from a bell curve onto data.
Four questions
Five parts, one idea
You began with clay and a slogan, and end with the transport equation and the training recipe behind modern generative models. The thread was never broken: reshape a simple distribution with smooth, reversible motion, and keep exact account of the stretching. Everything else — Jacobians, coupling layers, KL divergences, divergence integrals, straight-line regression — is engineering in service of that one sentence.
- Papamakarios et al., Normalizing Flows for Probabilistic Modeling and Inference — the standard survey.
- Chen et al., Neural Ordinary Differential Equations (2018) — where the continuous limit begins.
- Lipman et al., Flow Matching for Generative Modeling (2022); Liu et al., Rectified Flow.
- Song et al., Score-Based Generative Modeling through SDEs — the bridge to diffusion and the probability-flow ODE.