Bending the Bell Curve

Where do things tend to land?

Ask a thousand adults their height and you won't get a thousand random numbers. The answers pile up around 170 cm, thin out toward 150 and 190, and almost never reach 250. There is a shape to the answers. Statisticians call that shape a probability distribution — a map of where values tend to land, and where they basically never do.

Everything a machine-learning model generates — a face, a voice, a molecule — is a draw from some distribution. So two abilities matter enormously. First, sampling: produce a brand-new example that looks like it belongs with the real ones. Second, scoring: given an example, say how plausible it is. Scoring is what lets a model whisper “this jet-engine reading looks weird” or “this molecule is unlikely to occur.”

For a bell curve, both are easy — it's one tidy formula with two knobs (where's the middle, how wide is the spread). But the distribution of, say, real handwritten digits lives in hundreds of dimensions and has a wildly complicated shape. Nobody can write its formula down by hand.

Probability as clay

Picture the bell curve as a mound of soft clay. Its thickness at any spot is the probability of values landing there. Now reshape the mound — stretch it here, squeeze it there — until it matches the complicated shape you actually want. You never created or destroyed clay. You only moved it.

A normalizing flow is exactly this, done with math. It's a chain of small reshaping steps applied to a simple distribution. But the steps must obey two house rules:

Rule 1 — no tearing. The reshaping must be smooth. Neighboring points stay neighbors; nothing teleports. (This is why it's called a flow — the points drift like particles in slow water.)

Rule 2 — no folding. Two different points may never land on the same spot. Every destination has exactly one origin, so the whole reshaping can be run backwards. In math terms, each step is invertible.

One slider, one transformation

Below is a real normalizing flow — a tiny one, with a single reshaping step. The dots are samples from a bell curve; the curve above them is the distribution's shape. The bend slider applies a smooth, invertible stretch that pushes points away from the center. Drag it and watch a valley form: one bump becomes two.

Notice where the curve gets low: exactly where the dots got stretched apart. That's not a coincidence — it's the entire secret, and it's next.

Probability is never created or destroyed

Look at the shaded slice in the demo above. Before the bend, it holds some fixed share of all probability — say 6%. After the bend it holds… still exactly 6%. The slice got wider, so the curve over it must get lower, because the amount of clay inside is locked. Stretch a region by 2× and its density drops to ½. Squeeze it to half and the density doubles.

This is why the two house rules matter so much. No folding means each y has exactly one x to look up. No tearing (smoothness) means “how much it stretched here” is a well-defined number at every point. The transformation effectively keeps a receipt for every stretch and squeeze it performs — and the receipts are what let us compute exact probabilities in the final, complicated shape.

In two or more dimensions the idea is identical, just with area instead of width: take a tiny square, see how much the transformation inflates or deflates it. That local area-stretch factor has a fancy name — the Jacobian determinant — but it is still just “how much did this patch stretch.”

So why is it called a “normalizing flow”?

Flow, because samples drift smoothly through each small step, like fluid. Normalizing, because of the backward direction: run real data backwards through the chain and it settles into a normal distribution — the bell curve's formal name.

And how does it learn?

The reshaping steps have adjustable knobs (in practice, neural networks). Training shows the model real examples and asks: run these backwards — did they land in the fat middle of the bell curve, or out in the tails? Then it nudges the knobs so the real data becomes as plausible as possible. The stretch receipts keep the model honest: it can't just squash everything into the middle, because extreme squeezing is penalized by exactly the amount it squeezed. That's the whole training loop — no adversary, no approximation, just “make the data likely.”

Simple moves, stacked, get you anywhere

One step can only reshape so much. The real power move is stacking: each layer is almost embarrassingly simple, but their composition is expressive. Below, 750 points from a 2-D bell-curve cloud pass through just two layers. Layer 1 is as plain as it gets — stretch the horizontal axis, squeeze the vertical — turning the blob into a bar. Layer 2 is a swirl: each point rotates by an amount that grows with its distance from the center, winding the bar's ends into arms. A galaxy, from a blob, in two moves.

Drag the slider leftward and you're running the flow in its normalizing direction — the spiral melts back into a bell curve. Real flows stack dozens of learned layers like these (with names like coupling layers), each simple enough that its stretch receipt is cheap to compute, together flexible enough to model faces, audio, or molecules.

In the wild

Flows had a star turn in image generation — OpenAI's Glow produced sharp faces and let you smoothly morph attributes, because every image maps to a point in bell-curve space where arithmetic is easy. NVIDIA's WaveGlow synthesized speech. In the sciences, flows act as Boltzmann generators, proposing plausible molecule configurations in one shot instead of simulating them for hours. And because flows compute exact plausibility scores, they quietly power anomaly detection — flagging the sensor reading, transaction, or particle event that real data says shouldn't happen.

Flows are no longer the loudest model in the room — diffusion took that seat — but the idea didn't fade; it dissolved into everything. Modern “flow matching” methods, used inside several state-of-the-art generators, are the same picture you've been playing with: probability, flowing smoothly from a bell curve to data.

Point the samples where it counts

Away from generative modeling, flows have a quieter superpower: computing integrals — the total area under a complicated curve. In serious applications that curve lives in dozens or thousands of dimensions: the odds of particle-collision outcomes, the light arriving at a camera pixel, the energies of a molecule's configurations. No grid can cover that much space, so the one method that survives is sampling — drop random points, average what you find.

But random points are wasteful. When nearly all of the area hides in a couple of narrow bumps, samples from a plain bell curve mostly land where there is nothing, and the estimate wobbles badly. The classic fix, importance sampling, says: draw your samples from a distribution q shaped like the thing you're integrating, then undo the favoritism by dividing each sample's contribution by q's height there. The catch: you must be able to draw from q and know its exact, honestly-normalized height at any point. Very few flexible models can do both. A flow can — sampling is its forward direction, and exact heights are its stretch receipts.

But what is a vote, really? A single sample can only see one thing — the integrand's height where it landed. A height becomes an area only when multiplied by a width, so each sample speaks for the stretch of axis around it. And the width of that stretch is set by the crowding: where q is tall, samples pack densely and each owns a thin sliver; where q is low, a lone sample must represent a wide stretch — width about 1 / (n·q). So a sample's honest contribution is its patch of area: height × the width it speaks for. The vote f ÷ q is exactly that patch scaled up n-fold — the sample saying, “if everyone's patch looked like mine, the total would be this.” Averaging the votes stitches the patches back into the whole area. Dividing by q is no trick; it converts a height into that sample's fair share of the area.

Try it. The orange shape below is an integrand f; we want its area. The blue curve is the proposal q — the very same bendable bell curve from Part 3. Two hundred samples are drawn from q, and each casts a vote of f ÷ q, drawn as a stem: tall stem, loud vote. Tick “show each vote's patch” to see the patches themselves — the tinted rectangles' combined area literally is the estimate, and at the sweet spot they tile the orange shape almost seamlessly. Bend the proposal until it hugs the integrand, and watch the votes even out while the estimate locks onto the true area.

Why is bend = 0 so bad? Because the votes are wildly unequal. A sample landing at x = 0 — where the plain bell curve loves to put samples but the integrand is low — votes about 0.7. A sample landing out on a bump votes about 17. The average becomes a tug-of-war: dozens of samples muttering “0.7” against the rare one shouting “17!”. Over enough samples these exactly balance — the estimate is unbiased — but any single batch of 200 depends on how many shouters it happened to catch; hence the ±30% wobble. Near the sweet spot every sample casts nearly the same vote, so the same 200 points pin the area to under a percent. And since sampling error shrinks only with the square root of the sample count, matching the shapes buys what brute force would pay for with roughly a thousand times more samples.

This exact recipe — with learned bends, in many dimensions — is how flows do serious integrals. In particle physics the integrand is the rate of a collision outcome, summed over every way it can happen (flows are the neural descendants of the classic VEGAS method). In film rendering it is a pixel's brightness, summed over every path light can take to reach it. In chemistry it is a molecule's free energy, summed over its possible configurations. Each time, a flow is trained to be the shrunk copy of a monstrous integrand — and then a modest number of samples does an impossible-looking integral.

So what is the loss function?

You have already been staring at it. The demo's “typical wobble” readout is a loss: one number measuring how badly the proposal's shape mismatches the integrand, bottoming out exactly when every vote is equal. The bend slider is a model with a single knob, and dragging it toward the sweet spot by hand is precisely what training does by gradient descent — a real flow just has millions of knobs (the weights of the small neural networks inside each layer), far too many for any hand.

Which loss you write down depends on what you hold. If you hold samples — real faces, real molecule snapshots — you train exactly as Part 4 described: run each example backwards and maximize its log-probability under the flow, receipts included. Statisticians call this minimizing a “forward” Kullback–Leibler divergence; the tutorial calls it make the data likely. If you hold only a formula — an unnormalized f, like a physics energy landscape, with no samples to learn from — the flow trains on its own output: draw samples from q and penalize either the average mismatch log q − log f (the “reverse” KL divergence) or, most directly, the variance of the votes themselves. Here is the elegant part: f's unknown normalization constant only shifts these losses up or down by a constant, so it never gets in the way of training — and when training ends, that constant is exactly what the average vote hands you.

Notice the quiet dependency: every one of these losses contains log q or a vote f ÷ q, so every one needs the proposal's exact density — the receipts again. That is why this game belongs to flows: the loss is exactly computable, differentiable through every bend, and flat votes are its minimum. (And VEGAS's bin re-weighting, from the box above, is the variance loss once more — minimized axis-by-axis with a cheap iteration instead of gradients.)

Three questions

Distribution

A map of plausibility: which values tend to occur, and how often.

Density

The height of that map at one spot. Higher = values land here more often.

Base distribution

The simple starting shape, almost always a bell curve (a “standard normal”).

Invertible

Undoable. Every output has exactly one input, so the map runs backwards too.

Jacobian determinant (juh-KOH-bee-un)

The local stretch factor: how much a tiny patch of space grew or shrank at a given point.

Change of variables

The rule that carries density through a transformation: new height = old height ÷ stretch.

Coupling layer

A popular flow layer that shifts and scales half the coordinates based on the other half — expressive, yet trivially invertible.

Likelihood

The plausibility score a model assigns to data. Flows compute it exactly; training pushes it up.

• Lilian Weng, Flow-based Deep Generative Models — the classic gentle-but-technical walkthrough.
• Eric Jang, Normalizing Flows Tutorial — two parts, with code.
• The papers that built the field: NICE (2014), RealNVP (2016), Glow (2018).
• Papamakarios et al., Normalizing Flows for Probabilistic Modeling and Inference — the thorough survey, for when the pictures here stop being enough.