Bending the Bell Curve
Normalizing flows are one of the friendliest big ideas in machine learning: take a plain bell curve and bend it, carefully, until it matches your data. This page explains how — no background needed.
Where do things tend to land?
Ask a thousand adults their height and you won't get a thousand random numbers. The answers pile up around 170 cm, thin out toward 150 and 190, and almost never reach 250. There is a shape to the answers. Statisticians call that shape a probability distribution — a map of where values tend to land, and where they basically never do.
Everything a machine-learning model generates — a face, a voice, a molecule — is a draw from some distribution. So two abilities matter enormously. First, sampling: produce a brand-new example that looks like it belongs with the real ones. Second, scoring: given an example, say how plausible it is. Scoring is what lets a model whisper “this jet-engine reading looks weird” or “this molecule is unlikely to occur.”
For a bell curve, both are easy — it's one tidy formula with two knobs (where's the middle, how wide is the spread). But the distribution of, say, real handwritten digits lives in hundreds of dimensions and has a wildly complicated shape. Nobody can write its formula down by hand.
Simple distributions are workable but wrong. Real distributions are right but unworkable. Normalizing flows resolve this with a trick: keep the simple one — and reshape it.
Probability as clay
Picture the bell curve as a mound of soft clay. Its thickness at any spot is the probability of values landing there. Now reshape the mound — stretch it here, squeeze it there — until it matches the complicated shape you actually want. You never created or destroyed clay. You only moved it.
A normalizing flow is exactly this, done with math. It's a chain of small reshaping steps applied to a simple distribution. But the steps must obey two house rules:
Rule 1 — no tearing. The reshaping must be smooth. Neighboring points stay neighbors; nothing teleports. (This is why it's called a flow — the points drift like particles in slow water.)
Rule 2 — no folding. Two different points may never land on the same spot. Every destination has exactly one origin, so the whole reshaping can be run backwards. In math terms, each step is invertible.
One slider, one transformation
Below is a real normalizing flow — a tiny one, with a single reshaping step. The dots are samples from a bell curve; the curve above them is the distribution's shape. The bend slider applies a smooth, invertible stretch that pushes points away from the center. Drag it and watch a valley form: one bump becomes two.
Notice where the curve gets low: exactly where the dots got stretched apart. That's not a coincidence — it's the entire secret, and it's next.
Probability is never created or destroyed
Look at the shaded slice in the demo above. Before the bend, it holds some fixed share of all probability — say 6%. After the bend it holds… still exactly 6%. The slice got wider, so the curve over it must get lower, because the amount of clay inside is locked. Stretch a region by 2× and its density drops to ½. Squeeze it to half and the density doubles.
This is why the two house rules matter so much. No folding means each y has exactly one x to look up. No tearing (smoothness) means “how much it stretched here” is a well-defined number at every point. The transformation effectively keeps a receipt for every stretch and squeeze it performs — and the receipts are what let us compute exact probabilities in the final, complicated shape.
In two or more dimensions the idea is identical, just with area instead of width: take a tiny square, see how much the transformation inflates or deflates it. That local area-stretch factor has a fancy name — the Jacobian determinant — but it is still just “how much did this patch stretch.”
So why is it called a “normalizing flow”?
Flow, because samples drift smoothly through each small step, like fluid. Normalizing, because of the backward direction: run real data backwards through the chain and it settles into a normal distribution — the bell curve's formal name.
And how does it learn?
The reshaping steps have adjustable knobs (in practice, neural networks). Training shows the model real examples and asks: run these backwards — did they land in the fat middle of the bell curve, or out in the tails? Then it nudges the knobs so the real data becomes as plausible as possible. The stretch receipts keep the model honest: it can't just squash everything into the middle, because extreme squeezing is penalized by exactly the amount it squeezed. That's the whole training loop — no adversary, no approximation, just “make the data likely.”
A peek at the actual equation (optional)
Write the whole chain of steps as one invertible function f taking bell-curve points to data. For a data point x, its exact log-probability is:
First term: “where did x land when traced back, and how likely is that spot on the bell curve?” Second term: “sum of the stretch receipts along the way.” Training maximizes this quantity, averaged over real data, by gradient descent. Everything is exact — which is the signature feature of flows.
Simple moves, stacked, get you anywhere
One step can only reshape so much. The real power move is stacking: each layer is almost embarrassingly simple, but their composition is expressive. Below, 750 points from a 2-D bell-curve cloud pass through just two layers. Layer 1 is as plain as it gets — stretch the horizontal axis, squeeze the vertical — turning the blob into a bar. Layer 2 is a swirl: each point rotates by an amount that grows with its distance from the center, winding the bar's ends into arms. A galaxy, from a blob, in two moves.
Drag the slider leftward and you're running the flow in its normalizing direction — the spiral melts back into a bell curve. Real flows stack dozens of learned layers like these (with names like coupling layers), each simple enough that its stretch receipt is cheap to compute, together flexible enough to model faces, audio, or molecules.
In the wild
Flows had a star turn in image generation — OpenAI's Glow produced sharp faces and let you smoothly morph attributes, because every image maps to a point in bell-curve space where arithmetic is easy. NVIDIA's WaveGlow synthesized speech. In the sciences, flows act as Boltzmann generators, proposing plausible molecule configurations in one shot instead of simulating them for hours. And because flows compute exact plausibility scores, they quietly power anomaly detection — flagging the sensor reading, transaction, or particle event that real data says shouldn't happen.
| Generative model | Exact plausibility score? | Its character, in one line |
|---|---|---|
| Normalizing flow | yes — exact | Reversible by construction; its honesty (exact scores) costs it some architectural freedom. |
| GAN | no | A forger and a detective train against each other; sharp samples, but it can't score anything. |
| VAE | approximate only | Compresses data through a bottleneck; scores exist but are estimates, not exact. |
| Diffusion | indirect / costly | Learns to undo noise, step by step; today's image champion, but slower and scores take work. |
Flows are no longer the loudest model in the room — diffusion took that seat — but the idea didn't fade; it dissolved into everything. Modern “flow matching” methods, used inside several state-of-the-art generators, are the same picture you've been playing with: probability, flowing smoothly from a bell curve to data.
Three questions
- Distribution
- A map of plausibility: which values tend to occur, and how often.
- Density
- The height of that map at one spot. Higher = values land here more often.
- Base distribution
- The simple starting shape, almost always a bell curve (a “standard normal”).
- Invertible
- Undoable. Every output has exactly one input, so the map runs backwards too.
- Jacobian determinant (juh-KOH-bee-un)
- The local stretch factor: how much a tiny patch of space grew or shrank at a given point.
- Change of variables
- The rule that carries density through a transformation: new height = old height ÷ stretch.
- Coupling layer
- A popular flow layer that shifts and scales half the coordinates based on the other half — expressive, yet trivially invertible.
- Likelihood
- The plausibility score a model assigns to data. Flows compute it exactly; training pushes it up.
- Lilian Weng, Flow-based Deep Generative Models — the classic gentle-but-technical walkthrough.
- Eric Jang, Normalizing Flows Tutorial — two parts, with code.
- The papers that built the field: NICE (2014), RealNVP (2016), Glow (2018).
- Papamakarios et al., Normalizing Flows for Probabilistic Modeling and Inference — the thorough survey, for when the pictures here stop being enough.