Machines That Bend
Parts 1 and 2 bent space with maps designed by hand. Real flows learn their bends. This part builds the workhorse layer of the whole field — the coupling layer — tours its relatives, and then trains an actual normalizing flow live, in this page, on data you choose.
Wanted: a layer that bends but never folds
Part 2 left us with a checklist. A flow layer must be invertible, and its inverse must be cheap — sampling and density-evaluation each need one direction, and training hammers one of them millions of times. Its log-det must be cheap — O(d), not O(d³) — which in practice means a triangular or otherwise structured Jacobian. It must be expressive, because the whole point is to bend a bell curve into molecules and faces. And every part must be differentiable, so gradients can reach the knobs.
Notice what this rules out: an ordinary neural network layer. A generic ReLU layer folds space (many inputs, one output — the crime from Part 1), has no usable inverse, and its Jacobian is unstructured. The genius of the field's standard answer is where it chooses to hide the neural network: not as the map itself, but as the map's steering.
The coupling layer
Split the coordinates into two halves, A and B. Half A passes through untouched. Half B gets a plain stretch-and-shift — but the amounts of stretch and shift are computed from half A, by an arbitrary neural network (the “conditioner”):
Run the checklist. Invertible, cheaply: the output still contains yA = xA in the clear, so the inverter can recompute the very same s and t and undo the affine map — xB = (yB − t) ⊙ e−s. The conditioner network is never inverted — it only ever runs forward, in both directions of the flow. Cheap log-det: the Jacobian is block-triangular (yA ignores xB), and its diagonal is just the stretches es, so:
— add up the s outputs, done. No determinant is ever “computed” at all. Expressive: all the deep-learning firepower lives inside the conditioner, which is unconstrained — convolutions, transformers, whatever — because it never needs inverting and its own Jacobian never enters the receipts. The layer is a bureaucratic shell with a genius employee. And you have already met one: Part 2's “warp” was exactly this layer with a tiny hand-written conditioner — which is why its top-right Jacobian entry was pinned at zero.
Masks, mixes, and Glow
One coupling layer never touches half A — stack ten with the same split and half your coordinates emerge un-bent. So layers alternate the mask: A transforms B, then B transforms A, back and forth (our live model below alternates every layer). For images, RealNVP used checkerboard and channel-wise masks; Glow went further and inserted a learned invertible 1×1 convolution — a trainable linear remix of the channels — between couplings, so the “split” itself is learned. A linear mix's log-det is just log |det W| of one small matrix, cheap at channel count. Add a normalization layer (actnorm) and that triple — actnorm, 1×1 conv, coupling — repeated dozens of times, plus a multi-scale scheme that factors out coordinates as resolution drops, is the whole recipe that generated photorealistic faces in 2018.
A small family tree
Coupling layers are one point on a dial. Turn the dial toward maximum structure and you get autoregressive flows: coordinate i is transformed conditioned on all coordinates before it — the fully triangular extreme, the most expressive per-layer shape a cheap determinant allows. The price is a direction asymmetry. In a masked autoregressive flow (MAF), computing the density of a given point is one parallel pass, but generating a sample must produce coordinates one at a time — d sequential steps. Its mirror image (IAF) samples in one pass but pays d passes to score a given point. Coupling layers are the compromise that is fast both ways, at some cost in per-layer power.
A separate upgrade swaps the affine stretch-and-shift for something curvier: spline flows (neural spline flows, NSF) let the conditioner output the control points of a monotone rational-quadratic spline — an elementwise map that can wiggle, saturate, and sharpen. Same triangular bookkeeping (the log-det is the sum of log spline slopes), far more bend per layer. Beyond these live more exotic species — invertible residual networks, Sylvester flows — trading cheap exact receipts for other virtues, but the table below is the working core of the family.
| Layer family | Sample | Score (density) | Per-layer character |
|---|---|---|---|
| Coupling (NICE, RealNVP, Glow) | 1 pass | 1 pass | Fast both directions; needs more layers for the same bend; conditioner unconstrained. |
| Masked autoregressive (MAF) | d steps | 1 pass | Most expressive triangular shape; the density-first choice. |
| Inverse autoregressive (IAF) | 1 pass | d steps | Same power, mirrored; the sampling-first choice. |
| Spline coupling (NSF) | 1 pass | 1 pass | Coupling's speed with much curvier elementwise maps; today's default for tabular data. |
A real flow, training in this page
Below is a genuine RealNVP — six affine coupling layers with alternating masks, each steered by a small tanh network, 588 parameters in all — trained by Adam on the exact objective Part 2 derived: maximize log pbell(backward image) plus the ledger of log-dets. Nothing is precomputed; your browser is doing the gradient descent. Pick a dataset and press train.
The four windows are the four faces of one object. Top-left: samples — draw from the bell curve, run forward. Top-right: the exact density elog q painted over the plane — no histogram, the actual formula. Bottom-left: the normalizing direction — the training data run backward; as the loss falls, watch it melt into a round bell-curve cloud inside the dashed circles. That gaussianization is the literal meaning of the model fitting. Bottom-right: the loss itself — the negative mean log-likelihood in nats; every downtick means the data got more probable under the model.
Things worth trying: reset the weights and watch the density start as a featureless blob (near-identity initialization — the model is a bell curve at birth). Switch to the two-armed spiral — the very shape we built by hand in Part 1 — and watch six coupling layers rediscover it from data alone, no swirl formula provided. And notice the ring takes longest to close up: wrapping a simply-connected bell curve smoothly around a hole is genuinely awkward for a smooth bijection, and you can watch the model negotiate that constraint (the faint “seam” it keeps is not a bug — it's topology).