A Fireworks AI report argues that reinforcement-learning runs can go off course when the system generating training examples and the system learning from them calculate probabilities differently. In two company experiments, the mismatch either destabilised a run or exposed a subtler problem in how a mixture-of-experts model combines its outputs.
Fireworks AI Watch analysis
What happened
Fireworks describes the issue in a technical report on numerical alignment and MoE routing. In a GLM 5.2 countdown-reasoning experiment, the company says its trainer and rollout engine disagreed by a KL divergence of about 0.013. Safeguards discarded roughly 45% of each batch, yet reward still fell from about 0.9 to below 0.2 around step 20 of a 25-step run. With the calculations aligned, Fireworks says KL was zero, no data was discarded and reward remained stable. The zero-KL result applies to the tested LoRA configuration.
A second investigation, using Qwen3.5-MoE, found a mismatch in how expert outputs were weighted and combined. Fireworks measured a k3 divergence of 0.296 on image tokens, then says swapping in reference MoE modules brought the measured values for text and image tokens to zero.
Key findings
- The training and rollout paths can disagree
Different precision, kernels or operation order can change token probabilities even when model weights match. - Standard safeguards may not fix the underlying problem
Fireworks says about 45% of batches were rejected in its misaligned GLM run, but reward still collapsed. - MoE routing is only part of the story
Matching which experts are selected does not guarantee that their outputs are combined in the same way. - Compare probabilities at identical weights
Fireworks recommends measuring training-inference KL to spot discrepancies that generated text alone may not reveal.
Why it matters
A training run can appear to have a data, reward or learning-rate problem when the underlying trouble is a disagreement between software paths. That can mean wasted accelerator time and experiments aimed at the wrong cause. The practical lesson for teams is to check probability agreement across the full model, including expert aggregation, before spending heavily on a changed training recipe.
Our read
This is a useful account of an unglamorous failure mode: the model can be chasing a slightly different mathematical target from the one its rollout engine actually generated. Fireworks’ experiments are company-reported, not independent validation, but they make the diagnostic concrete. If you run RL training, check the numbers between rollout and training, not just whether the output looks plausible. Floating-point arithmetic has once again found a way to make expensive work more expensive.
What to watch
- Whether Fireworks publishes further validation across models, workloads and hardware configurations.
- How training-inference KL behaves under changing batch conditions.
- Whether other teams report similar MoE aggregation mismatches or different fixes.
Discussion spark: Should AI training teams treat training-inference probability checks as a standard pre-flight test, or are those checks too workload-specific to be useful?
Sources and evidence
- Reinforcement learning: Why alignment of numerics and MoE routing matter (30 September 2026, 00:00 UTC)
not affiliated with or endorsed by Fireworks AI