Fréchet Forcing Code ↗

Research project · 2026

Fréchet Forcing: Training Long-Horizon Streaming World Models from Short Clips

Fine-tuning streaming video world models on their own rollouts against real-data Fréchet statistics — sustaining coherent 4,000+ frame rollouts from 381-frame training clips with no teacher, no online critic, and no long videos.

Yian Zhao Yu Yan Hongcan Guo Ruochong Zheng Jian Zhang Jie Chen

Peking University

Paper Code Models Built on MiniWorld ↗
4,000+ Stable frames (>10× training horizon)
≤ 36h GPU-hours fine-tuning (225 steps)
25× / 2.7× Final-bin FVD reduction (DROID / Sekai)
Long-horizon 3×3 streaming rollouts
Action-conditioned robot manipulation (DROID) Nine 4,001-frame (~4.2 min) rollouts from a single real frame
Overview

Long-horizon stability without a teacher, critic, or long videos.

Streaming video world models generate the future chunk by chunk, conditioned on their own past and a stream of actions. When trained on short clips, they drift once a rollout exceeds the training horizon as prediction errors re-enter the context and compound — causing scenes on DROID to dissolve into saturated lattice textures and camera rollouts on Sekai to darken and fade to black.

A common remedy is distribution matching distillation (DMD) on self-generated rollouts. For action-conditioned world models, however, the teacher becomes a bottleneck: an external video model is agnostic to the actions, whereas the model's own pretrained network drifts in the same manner as the student. Teacher-free per-sample objectives avoid a teacher, yet supervise only states paired with recorded ground truth within short clips.

We observe that a long rollout need not match a teacher, provided it remains a realistic continuation of its own scene that follows the commanded actions. Fréchet Forcing fine-tunes a world model together with its deployed multi-step sampler under three complementary constraints derived solely from short real clips: generative anchoring, context fidelity, and reality matching.

Fine-tuning MiniWorld-1B (pretrained on 253-frame clips) for at most 36 GPU-hours on 381-frame clips yields MiniWorld-1B-FF, whose rollouts remain stable beyond 4,000 frames (>10× the fine-tuning horizon) and accurately follow robot and camera commands.

4,000-frame chunk-by-chunk rollouts from a single real frame. Two DROID manipulation episodes (top four rows) and one Sekai exploration scene (bottom two rows). Pretrained MiniWorld-1B (253-frame horizon) suffers from spreading lattice artifacts on DROID and fades to black on Sekai within 1,000 frames. MiniWorld-1B-FF, fine-tuned with Fréchet Forcing on 381-frame clips, stays coherent through frame 4,000.
Qualitative results

Long-horizon generated worlds

Explore 4,001-frame ultra-long rollouts and 60-second (961-frame) streaming rollouts under robot manipulation commands (DROID) and camera trajectories (Sekai). Switch between synchronized 3×3 grids and individual scene rollouts.

Method

Fréchet Forcing

A long rollout remains effective when it acts as a faithful conditional generator, preserves the identity of its initial scene, and produces realistic frames. Fréchet Forcing enforces these three properties with unit-weighted constraints.

ℒtotal = ℒanc Generative anchoring + ℒctx Context fidelity + ℒreal Reality matching

All distributional terms use running moment estimates (EMA decay β = 0.99) and self-normalised Fréchet distances FD / sg(FD), enabling batch-size-one training per GPU without storing past features.

01 · Regression on short window

Generative Anchoring (ℒanc)

Applies the flow-matching (FM) loss on W = 4 real chunks conditioned on a self-generated history of c ∼ U[1, N] chunks re-noised to ta = 0.7 and denoised with the deployed staircase sampler.

hc = [ sg S(G, x<c-1ta, a), xc-1 ]

Ending on real data eliminates seam flicker. Replacing the final history chunk with the clean real chunk xc-1 resolves the spatial mismatch across the boundary, preventing the model from periodically redrawing the scene every chunk.

02 · Clip-centred latent FD

Context Fidelity (ℒctx)

During a free rollout of N = 40 chunks from a single real latent, gradients flow through the final K = 8 chunks. Each latent is summarised by per-channel spatial mean and std concatenated across consecutive latents ψ(z), then centred on the clip's own ground truth:

h = ψ(zN-K:N) − mean(ψ(xN-K:N))

Preserving scene identity. Centring converts a marginal population metric into a clip-conditional one — preventing rollouts from collapsing toward a dataset-average appearance while scoring only the gradient-carrying tail.

03 · Perceptual frame FD

Reality Matching (ℒreal)

Decodes the last D = 4 rollout chunks (plus one preceding warm-up chunk whose frames are discarded) through the frozen causal VAE decoder and embeds frames with the CLS token of a frozen DINOv2 ViT-B/14.

ℒreal = FDEMA( φ(Dec(z)), refreal )

Ruling out degenerate textures. Matching high-level perceptual moments against VAE reconstructions of real training frames penalises the dark, washed-out, or fish-scale tiled artifacts that satisfy low-order latent moments.

Algorithm 1 · One training step of Fréchet Forcing PyTorch-like pseudocode 
# G: streaming world model;  S: deployed streaming sampler
# Dec: frozen VAE decoder;  phi: frozen DINOv2;  psi: latent moments
x, a = sample_clip()                    # 381-frame real latents & actions

# 1. Generative anchoring: FM after a self-denoised history
c = randint(1, N)
hist = S(G, renoise(x[:c], t=t_a), a).detach()
hist[-1] = x[c-1]                       # history ends on clean real chunk
L_anc = flow_matching(G, x[c:c+W], hist, a)

# 2. Free rollout from one real latent; last K chunks carry gradients
z = S(G, x[0], a)[-K:]
x_z = x[N-K:N]                          # GT over the same temporal span

# 3. Context fidelity & Reality matching via self-normalised EMA FD
L_ctx  = FD_ema(psi(z) - psi(x_z).mean(0), ref_ctx, ema_ctx)
L_real = FD_ema(phi(Dec(z)), ref_real, ema_real)

(L_anc + L_ctx + L_real).backward()
optimizer.step()
Comparison of training signals on self-generated rollouts Table 1
Property DMD / Long-DMD RMD Self-history Reversal Fréchet Forcing
Supervision target Teacher score Per-chunk teacher GT continuation GT start Real-data moments
Teacher-free ✗ ✗ ✓ ✓ ✓
Online critic-free ✗ ✗ ✓ ✓ ✓
Gradient into free rollout ✓ ✓ ✗ ✗ ✓
Original multi-step sampler kept ✗ ✗ ✓ ✓ ✓
Extra requirement Pretrained teacher Chunk teacher — Reversible dynamics Reference stats
Experiments

Quantitative evaluation over 4,000 frames

Evaluated on 32 held-out trajectories per domain up to frame 4,001 across six horizon bins, comparing pretrained MiniWorld-1B, an FM-only self-history baseline, and MiniWorld-1B-FF.

Realism and stability across 4,001 frames (Figure 2). Top: DROID robot manipulation. Bottom: Sekai camera-controlled exploration. While MiniWorld-1B and FM-only degrade sharply right after the 253-frame pretraining horizon, MiniWorld-1B-FF keeps CLIP-FD, FVD, brightness, and chunk-boundary flicker flat across the entire 4,000-frame rollout. The shaded grey band marks the 381-frame training clip length.

Realism over 4,001 frames on DROID

Across all six horizon bins, MiniWorld-1B-FF keeps CLIP-FD within 30–44 and FVD within 39–81, achieving 10× lower CLIP-FD and 25× lower FVD than MiniWorld-1B in frames 2,997–4,001. Manual inspection of 96 rollouts confirms 98% are artifact-free at frame 1,500 and 79% at frame 4,000 (vs. 0% for base).

Metric / Model 1–253 253–509 509–1k 1k–1.8k 1.8k–3k 3k–4k
CLIP-FD ↓ (lower is better)
GT (floor at same sample count)6105555
MiniWorld-1B (pretrained)3861105223348431
FM-only (self-history baseline)3866117217320401
MiniWorld-1B-FF (Ours)304238394044
FVD ↓ (I3D on 16-frame windows)
MiniWorld-1B (pretrained)45801777332,0002,058
FM-only (self-history baseline)44891945461,4201,873
MiniWorld-1B-FF (Ours)396657627081
Brightness (ratio to GT; 1.00 is ideal)
MiniWorld-1B (pretrained)1.001.001.001.021.031.00
FM-only (self-history baseline)0.990.980.960.940.940.94
MiniWorld-1B-FF (Ours)1.001.000.990.990.980.98
Chunk-boundary flicker (1.0 = no periodic redraw)
GT reference1.130.981.041.031.031.04
MiniWorld-1B (pretrained)1.081.121.121.221.351.46
FM-only (self-history baseline)1.061.091.071.131.251.37
MiniWorld-1B-FF (Ours)1.091.131.101.111.111.11
Training dynamics

Why naive distribution matching fails

Diagnostic probes over training steps reveal three distinct distributional failure modes on free rollouts — and show how each design choice in Fréchet Forcing prevents them.

Diagnostic probes across training steps (Figure 3). (a) Context dependence: Regression slope of each rollout's mean latent feature on its own ground truth in frames 765–933 (1 = full preservation). (b) Gradient–statistic alignment: Cosine similarity between the mean-matching correction and the true direction needed by the gradient-carrying tail chunks. (c) Held-out perceptual judge: Held-out CLIP-FD in frames 1,021–1,789 vs. training latent FD (dotted).

1. Context decoupling. Matching an unconditional marginal FD allows rollouts to ignore their initial frame and collapse toward a dataset-average reddish wash (slope drops from 0.92 to 0.50 in 50 steps). Clip centring fixes this.

2. Gradient–statistic misalignment. Pooling statistics over all N chunks while truncating gradients to the last K forces tail chunks to overshoot to compensate for the detached prefix (alignment cosine decays to 0.39). Scoring only the graded tail resolves it.

3. Feature-space exploitation. Optimising low-order latent moments alone drives training FD down while held-out CLIP-FD rises, producing dark or fish-scale tiled frames. Adding DINOv2 reality matching closes this loophole.

4. Chunk-boundary seam redraw. Re-rendering the anchoring history all the way to the seam teaches the model to redraw the scene every 8 frames (flicker 1.31). Ending the history on one clean real chunk lowers flicker to 1.07 (matching GT).

Citation

Cite Fréchet Forcing

@article{zhao2026frechetforcing,
  title   = {Fr{\'e}chet Forcing: Training Long-Horizon Streaming World Models from Short Clips},
  author  = {Zhao, Yian and Yan, Yu and Guo, Hongcan and Zheng, Ruochong and Zhang, Jian and Chen, Jie},
  journal = {arXiv preprint},
  year    = {2026}
}
 Expanded paper figure
Citation copied