The Continuous Latent
Diffusion Language Model

The ODE's density obeys the continuity equation $\partial_t p_t(z) + \nabla\!\cdot\!\big(p_t(z)\,v_\psi(z,t)\big)=0$. Following one trajectory, this collapses to the instantaneous change-of-variables formula, then integrates:

Plain English: a normal generative prior you can only sample from. This one hands you the actual probability number: flow the latent forward to noise, read off the Gaussian density where you land $\log p_1(z_1)$, and add the total log-volume the flow stretched along the way $\int \nabla\!\cdot v$. The divergence integral is the "how much did space squeeze" ledger.

Training maximizes $\mathcal{L}_{\mathrm{ELBO}}$, never $\log p(x)$ itself. The gap is precisely how far the encoder $q_\phi$ is from the true posterior $p(z_0\mid x)$ — a fixed, non-negative tax. A poor encoder is a permanent likelihood penalty (this is the variational gap that returns in Appendix D & F).

This is the formal statement behind the budget widget's bar ③. The "target cloud" $\bar q_\phi(z_0)=\int q_\phi(z_0\mid x)\,p_{\text{data}}(x)\,dx$ is the marginal of all encoder outputs; the DiT prior is trained to match exactly it.

One equation, three jobs for the encoder: realize text from the latent, decide how many nats the latent carries $I_q(X;Z_0)$, and set how hard the prior must work. The reason "raise the rate" doesn't monotonically help: it lifts reconstruction but is subtracted and also inflates the KL the prior must close.

Directly maximizing $\log p_\psi(z_0)$ needs ODE solves and divergence estimates every step — expensive. FM instead regresses the velocity field onto the straight bridge paths; its pointwise optimum is the conditional-mean velocity. It learns the same prior $p_\psi$ far more cheaply, but you must not read $\mathcal{L}_{\mathrm{FM}}$ as if it were the negative prior log-likelihood. This distinction is the seed of the "likelihood ≠ generation" story (Appendix F).

Evaluating $\log p_\psi(z_0^{(k)})$ means solving the state and a log-density accumulator together, then estimating the divergence — which is a $d\times d$ Jacobian trace — with the Hutchinson estimator:

Plain English: integrate a passenger $\ell$ alongside the latent that tallies the log-volume change; at $t=1$, $\log p_\psi(z_0)=\log p_1(z_1)+\ell_1$. The exact trace is brutal in high-$d$, so one random projection $\epsilon^\top J\epsilon$ (a single vector-Jacobian product) estimates it — with the same $\epsilon$ frozen across the whole solve so the trajectory stays self-consistent.

Same importance weights $\log w^{(k)}=\log p_\theta(x\mid z_0^{(k)})+\log p_\psi(z_0^{(k)})-\log q_\phi(z_0^{(k)}\mid x)$, two ways to average. Average-then-log (IWAE) beats log-then-average (ELBO) by Jensen, and tightens toward the truth as $K\!\to\!\infty$. Both are still lower bounds — so an ELBO-based PPL is an upper bound on the true perplexity.

The exact identity $\log p(x^{\text{res}}\mid x^{\text{pre}})=\log p(x^{\text{pre}},x^{\text{res}})-\log p(x^{\text{pre}})$ is run with two estimators (Algorithm A.2). Caveat the main text glosses: subtracting two bounds does not inherit a bound property — the difference can land on either side of the truth, so this is a practical estimator, not a certified lower bound. For ranking a single fresh block, the block-level score (B.21–B.22) suffices.

Set states $S_i:=x_{1:i}$. Then $(S_i)$ is a Markov chain whose one-step kernel is the AR conditional $p_\eta(x_i\mid x_{<i})$. Its true inductive bias isn't Markovianity — it's that conditioning is locked to the growing prefix $\sigma(X_{1:1})\subset\dots\subset\sigma(X_{1:L})$. Exact token likelihood, but a frozen left-to-right order.

LLaDA's masking is a continuous-time Markov chain that absorbs each token into a mask state with probability $t$ (C.16) — a reverse recovery over discrete states. Plaid does the same in a continuous token-aligned space $h_0=\mathrm{Embed}(x)$; as noise $\to 0$ its state stays glued to the observation (C.17). In the $\sigma_0^2\!\to\!0$ limit (C.19), Cola would degenerate to Plaid — which pinpoints the genuinely new ingredient: the latent decomposition itself, not the continuity.

The flow $z_1\!\sim\!p_1,\ z_0=\Phi^\psi_{0\leftarrow1}(z_1)$ never sees $x$. The encoder $q_\phi$ appears only in the variational bound (C.22), so it belongs to inference; in Plaid/LLaDA the forward process is part of the model definition. That is the precise sense in which Cola is "first and foremost a hierarchical latent-variable LM with a CNF prior, where flow is just a way to make the prior family expressive."

For any two candidate priors $p_a,p_b$, the average-ELBO gain of swapping $a\!\to\!b$ is exactly the reduction in KL-to-aggregated-posterior. So whenever the flow/CNF prior sits closer to $\bar q_\phi$ than a plain Gaussian does, the average ELBO provably rises. "Why diffusion" is not about escaping max-likelihood — it is about buying a more expressive prior family that closes this KL.

Every family's population risk is $H(p_{\text{data}})+\text{model mismatch}+\text{objective gap}$. Cola pays an extra inference gap $\mathcal{G}^{\text{infer}}=\E\,\KL(q_\phi\|p_{\theta,\psi}(z_0\mid x))\ge 0$ that AR (exact NLL) never pays. So the clean verdict (D.9): Cola beats AR iff $\mathfrak{R}_{\text{Cola}}<\mathfrak{R}_{\text{AR}}$ — superiority is never automatic from "more machinery."

The mutual-information identity $H_q(X\mid Z_0)=H(p_{\text{data}})-I_q(X;Z_0)$ turns the reconstruction floor into a rate problem:

Plain English: $\mathcal{D}(R)$ is the best reconstruction you can buy if the latent is allowed at most $R$ nats. If it drops fast at small $R$, a cheap global summary exists → the bottleneck helps. If you only get reconstruction near $R\!\approx\!H(X)$, the data is near-incompressible → the bottleneck is pure overhead. This is the slider in the widget above.

If $p_{\text{data}}(x)=\int p^\star(x\mid g)p^\star(g)\,dg$ with $H(X\mid G)\ll H(X)$ and $\dim G\ll \dim E(X)$, Cola's inductive bias is the data's structure: it splits one hard problem into "learn $p^\star(g)$ + learn $p^\star(x\mid g)$" (D.17). Where that fails, the three explicit costs (D.18) — inference gap, the elevated reconstruction floor $H(X\mid Z_0)$ from the bottleneck, and joint-training complexity — dominate. And the variational gap is always present (D.19): $\log p(x)-\mathcal{L}_{\mathrm{ELBO}}=\KL(q_\phi\|p(z_0\mid x))$. Success is a competition among three curves — $\mathcal{D}(R)$, prior-approximation, and the inference gap — and only when all three favor Cola is the decomposition a real advantage.

Suppose the objective decomposes additively over independent, identically-behaving latent dimensions, with a shift-response of identical functional form:

That is the formal meaning of "no shared structure": dimension $d$ only rescales/offsets the same per-dimension curve $j(\delta)$.

Plain English: a positive rescale $a_d$ and a constant offset $b_d$ never move the location of a maximum. So if the latent were truly separable, the best timeshift would be pinned across $d$. The contrapositive (Cor E.3): an observed stable, monotonic, reproducible drift — not explainable by parameter count, under-training, or noise — rejects the null. The drift in the widget above is that rejection.

Write the forward process $z_t=\alpha_t z+\sigma_t\epsilon$ and decompose the latent into a semantic signal plus residual, $z=\phi(s)+u$. Then what reaches the DiT is $z_t=\alpha_t\phi(s)+(\alpha_t u+\sigma_t\epsilon)$ — so what matters is not the raw timestep but how much information about $s$ survives. Under the separable null, $I(s;z_t)=\sum_i I(s_i;z_{t,i})$ and varying $d$ only rescales it — again no shift in the optimal regime.

Let many dimensions observe one low-dimensional shared factor, $z_i=A_i g+\xi_i$. Standard linear-Gaussian inference then gives a recovery SNR that grows with $d$, and a recoverable-information that grows logarithmically:

More dimensions watching the same factor ⇒ stronger effective SNR ⇒ the shift must compensate logarithmically to keep training in the same semantic-recovery regime. This is the dashed "Appendix-E prediction" line the widget plots — and it is structurally homologous to the resolution-dependent timestep shift in Stable Diffusion (Remark E.4).

Even at fixed $d$, lowering the VAE logSNR raises posterior variance $\Sigma_u$, so the total noise seen by the semantic variable is $\Sigma_{\text{noise}}(t)=\alpha_t^2\Sigma_u+\sigma_t^2 I$. A smoother latent ⇒ the same raw timestep corresponds to a lower effective semantic SNR ⇒ the shift must be recalibrated. Latent dimension and VAE logSNR look like two different knobs but act on one object: the effective mutual-information curve $I(s;z_t)$ along diffusion time. (This is the bridge to the noise-schedule deep-dive on the RQ3 page.)

Specifying $\lambda(t)$ fixes $(\alpha_t,\sigma_t)$ and vice-versa. A timeshift $\lambda_\delta(t)=\lambda(t)+\delta$ therefore doesn't reweight a loss after the fact — it re-maps the same raw timestep to a different logSNR interval.

Change variables $t\to\lambda$ in the FM objective. The uniform-$t$ measure pushes forward to a non-uniform measure on the logSNR axis, and the supervised target velocity rescales:

Plain English: shifting the schedule changes (i) which noise regimes get sampled most, and (ii) how hard the regression target is in each regime. So uniform-timestep training is not equivalent to uniform-logSNR training unless $\lambda(t)$ is affine (Prop G.1). The schedule is part of the objective, not a knob bolted on top.

The schedule controls the curve $t\mapsto I(s;z_t)$. Choosing the timeshift is therefore an effective-semantic-information calibration problem — it depends jointly on latent dimension $d$, posterior uncertainty $\Sigma_u$, latent geometry $\mathcal{G}$, and block size $B$. (Remark G.3: block size has no closed-form law but couples to the schedule through the same curve — which is why block 16 and loc 1.0 are co-selected.)

The VAE logSNR (H.7) is the signal-to-noise of the encoder posterior — larger ⇒ cleaner, more deterministic latent. The timestep shift is implemented as a LogitNormal sampler: $s=t/T\sim\mathrm{LogitNormal}(\mu,\sigma^2)$, density $p(s)=\frac{1}{\sigma\sqrt{2\pi}}\frac{1}{s(1-s)}\exp\!\big(-\frac{(\log\frac{s}{1-s}-\mu)^2}{2\sigma^2}\big)$ (H.9). Larger $\mu$ pushes sampling mass toward later timesteps; $\sigma$ controls how concentrated it is. That is precisely the "loc" the widget sliders and Fig 17 visualize — a reshaping of which logSNR regime is emphasized, not a numeric reweighting.

The squared FM loss has a unique optimum: the conditional-mean velocity. When the conditional response distribution is multimodal or broad-peaked, FM learns an average transport into a reasonable region — it never promises local density calibration around any one gold sample. That is the root cause.

If the context admits several valid continuations, the prior's global mean sits between the modes — far from the gold latent the posterior selected. Generation is still fine as long as the modes' mass lands in a decoder-good region.

Plain English: generation only needs the prior to drop an $\alpha$-fraction of mass somewhere in the big good region — a coverage requirement. Conditional PPL needs the prior to put high local density on the thin gold tube of one specific response — a calibration requirement. Prop F.3 shows both can hold at once: good samples, yet $\mathcal{S}_{\text{resp}}\le B-\Delta$, an arbitrarily biased PPL.

Good reconstruction $R\to R_{\max}$ does not imply good PPL if the KL gap stays positive (Prop F.4). And even if the centers align ($\mu_p\approx\mu_q$), the scale/orientation/volume terms in the Gaussian KL keep PPL poor (Prop F.5) — a too-sharp posterior amplifies this. AR is immune because training = the object PPL evaluates = the object generation uses (F.25): $-\log p^{\text{AR}}=\sum_i-\log p(x_i\mid x_{<i})$. Continuous latents add latent-integration, posterior–prior matching, and decoder compatibility on top — which is why PPL behaves like a density-calibration metric, not a generation-quality metric.

Cola's prior is learned block-by-block as a conditional flow $p_\psi(z_0)=p_\psi(z_0^{(1)})\prod_{b\ge2}p_\psi(z_0^{(b)}\mid z_0^{(<b)})$, predicting a noisy block under clean historical conditions. Decompose the first block as $z^{(1)}=(z_K,z_U)$ (known / unknown). The mathematically correct task is:

The known part is a boundary condition, only the unknown part is transported by the flow.

Clean conditioning solves the transport under exactly the intended condition $(z_{\text{pre}},z_K)$. Partial repaint replaces the true known region with a degraded, time-varying surrogate $\tilde z_{K,t}$ — a different, noisier regression target.

Plain English: a noisier condition is compatible with more clean targets, so the irreducible variance of the velocity regression rises. Worse, it's a role mismatch: in Cola the flow path is for prior transport, and historical conditions are supposed to be stable anchors. Partial repaint demotes the known region from "fixed condition" to "partially-denoised state variable" — it changes the task from transport the unknown under a fixed condition to jointly maintain a noisy known part and transport the unknown.

Because inference integrates the learned field along an ODE, the condition-induced bias $\delta$ accumulates over the trajectory ($L$ = Lipschitz constant). This is why reducing the guided fraction $m$ hurts (more unguided interval = more accumulation) and why more repaint cycles $t$ don't help (repeated early corrections can't turn a transient condition into a persistent one). Left/right padding never re-noises the known region, so it avoids the worst failure — but it only rearranges layout, never locks the condition exactly, and it complicates the block-causal attention pattern. Hence the strict ordering: clean cond ≫ padding ≫ partial repaint, exactly as the table shows.