Home BlogBlog Detail

FP8 Reinforcement Learning in SkyRL: Preserving Policy Consistency Across Training and Rollout

By Jinghan Yao, Eric Tang, Sumanth Hegde and Kourosh Hakhamaneshi | August 25, 2026

SkyRL now supports FP8-accelerated training and rollout with on-policy weight sync. In long-run RL experiments, the FP8 configuration closely tracks BF16 convergence while reducing end-to-end step time by up to 23%.

TL;DR. SkyRL now supports FP8 across the performance-critical parts of the reinforcement-learning stack: linear-layer GEMMs during training, model weights and the KV cache during rollout, and weight transfer between the trainer and vLLM. The key system change is a synchronization path that transfers the trainer-produced FP8 payloads and block scales directly to the rollout engine, avoiding a dequantize–requantize boundary that can change the rollout policy. In long RL runs, the FP8 configuration closely tracks BF16 convergence on Qwen3.5-9B using 8×H100 and Qwen3.5-35B-A3B using 8×B200, supporting both blockwise and MXFP8 recipes. End-to-end step time improves by up to approximately 19% and 23%, respectively, in the evaluated long-rollout configurations, and FP8 parameter storage additionally reduces per-GPU trainer weight memory by 39–42%.

Precision scope. “FP8” in this post refers to FP8-accelerated linear-layer computation, FP8 primary weights, FP8 rollout weights and KV cache, and FP8-aware weight synchronization. Precision-sensitive tensors (such as norms, embeddings, etc.), attention, and optimizer state remain in BF16 or FP32. Specifically, we use the blockwise recipe and the MXFP8 recipe for Hopper and Blackwell respectively.

Models. All experiments use the Qwen3.5 family (Qwen3.5-4B, Qwen3.5-9B, Qwen3.5-35B-A3B). These are hybrid-attention models in which some layers use gated-delta-network (GDN) linear attention.

LinkWhy FP8 matters for reinforcement learning

In many on-policy RL workloads, including the configurations evaluated here, rollout generation accounts for a substantial fraction of end-to-end time. Training then performs forward and backward passes over the generated tokens. Both phases can benefit from FP8 for different reasons: in training, lower-precision GEMMs increase peak arithmetic throughput, while in rollout, the smaller representation reduces memory traffic and KV-cache capacity requirements. On Hopper and Blackwell Tensor Cores, FP8 offers up to twice the peak matrix-multiplication throughput of BF16, while each FP8 value requires half the storage. To further reduce the memory footprint and speed up model weight loading, FP8 quantization can be applied to the policy model weights in the trainer, so that we only keep the FP8 model parameters and FP32 master weights, completely bypassing BF16 copies. This is often referred to as fp8_param.

Recent work has reached similar conclusions. LMSYS's Unified FP8 shows that using a consistent FP8 scheme for training and serving can reduce the train–rollout log-probability gap that arises in mixed-precision RL. NVIDIA NeMo RL reports 15–25% step-time reductions with closely matched accuracy in its evaluated Hopper configuration. Both build on the fine-grained block-scaling approach used by DeepSeek-V3, including 128×128 scaling blocks for weights rather than a single scale per tensor.

During our experiments, we found that the challenge of FP8 RL is not simply enabling FP8 independently in the trainer and rollout engine. Reinforcement learning couples the two systems through the policy: the trainer must optimize the same policy, to within controlled numerical differences, that generated the sampled trajectories.

LinkWhen independent quantization breaks policy consistency

A straightforward initial configuration is to enable Transformer Engine (TE) FP8 in Megatron using TE's per-tensor delayed-scaling recipe and set vLLM's quantization="fp8". In a Qwen3.5-4B DAPO run, this configuration becomes unstable within 70 steps. Reward remains below the BF16 baseline, policy entropy approaches zero, the mean importance-sampling ratio remains near 0.75 rather than 1.0, and the PPO clip ratio is more than an order of magnitude above the BF16 run.

Figure 1 Naive FP8 and BF16 on Qwen3.5-4B with DAPO. The FP8 configuration exhibits stalled reward, declining entropy, an importance-sampling ratio far from 1, and a substantially elevated clip ratio.

Root cause: quantization mismatch at weight synchronization. The failure originates at the interface between the two engines, rather than from either FP8 compute path in isolation:

Megatron and TE train with quantized FP8 weights and scale metadata — one scale per 128×128 block in the blockwise recipe or per 1x32 block in the MXFP8 recipe.
During weight synchronization, the original bridge dequantizes the weights to BF16 and transfers only the BF16 values, discarding the trainer's FP8 scale metadata.
vLLM independently quantizes the received BF16 values into FP8 values that are different from the FP8 weights used by Megatron due to the different quantization recipe.

This sequence composes two lossy transformations with independently selected scales. The resulting rollout weights match neither the trainer's quantized representation nor the higher-precision checkpoint exactly. This is an off-policy weight sync: the rollout engine samples from a policy the trainer never held.

For the blockwise recipe, across all quantized weights in the 4B model, the mean relative error between the trainer and rollout FP8 weights is 0.038. This is approximately 1.4× the quantization error of either representation relative to the BF16 checkpoint. Each engine's quantization error is tolerable on its own; the problematic quantity is the disagreement between engines. The trainer updates one numerical policy while the rollout engine samples from another, which is reflected in the importance-sampling and clipping diagnostics.

Figure 2 Off-policy weight sync, the original path. The bridge exports BF16 values without the trainer-s block-scale metadata, after which vLLM derives a new FP8 representation.

LinkOn-Policy FP8 weight sync

We redesigned weight synchronization in SkyRL so that vLLM consumes the FP8 representation produced by the trainer. We call this on-policy weight sync (OPWS): the rollout engine runs the same numerical policy the trainer updates, rather than a re-quantized approximation of it.

Figure 3 Quantized parameters are transferred as FP8 payloads with their block scales, while precision-sensitive tensors remain in BF16. vLLM installs the transferred representation without invoking an independent quantizer.

With this path, the trainer and rollout weights after synchronization are bitwise identical: the rollout engine receives the same FP8 values and scales produced by the trainer. The remaining train–rollout log-probability difference arises from activation quantization and kernel-level numerical differences. During the 400-step Qwen3.5-9B run, the mean absolute per-token log-probability gap remains between 0.02 and 0.03 without increasing over time. This is approximately three times the gap observed between the two BF16 engine paths, which provides a practical floor from kernel differences alone.

By comparison, a BF16 trainer paired with a rollout engine that independently quantizes weights to FP8 begins with a larger gap and exceeds 0.05 by step 400. This trend is consistent with a recurring quantization mismatch at each synchronization point.

Figure 4 Mean absolute per-token log-probability difference between the rollout engine and trainer on Qwen3.5-9B.

LinkExperimental scope

The core comparisons use matched seeds and data order.

Scope	Model	Hardware	Span	Comparison
Convergence and timing	Qwen3.5-9B dense	8xH100	400 steps	BF16 vs FP8 OPWS
Convergence and timing	Qwen3.5-35B-A3B MoE	8×B200	400 steps	BF16 vs. FP8 with OPWS
Scale-format ablation	Qwen3.5-9B dense	8×H100	200 steps	FP32 vs. power-of-two block scales

The DAPO configuration uses 32 prompts with eight samples per prompt, one optimizer step per batch, AdamW with a constant learning rate of 1e-6, and gradient-norm clipping at 1.0. Step-time comparisons use matched response lengths so that differences in policy behavior do not confound the system measurement.

With OPWS enabled, we evaluated FP8 training GEMMs, FP8 rollout weights and KV cache, and FP8-aware weight transfer in matched 400-step runs.

Qwen3.5-9B dense on 8xH100

Figure 5 Qwen3.5-9B on 8×H100. Reward and pass@8 track BF16 over 400 DAPO steps.

Qwen3.5-35B-A3B MoE on 8xB200

Figure 6 Qwen3.5-35B-A3B on 8×B200. Reward and pass@8 track BF16 over 400 DAPO steps.

Across both configurations, the FP8 and BF16 reward curves overlap closely over the 400-step horizon. The pass@8 trajectories likewise reach similar solve rates under repeated sampling, providing a complementary view to mean reward.

Response length is a sensitive indicator of policy divergence and also affects step time. It remains closely aligned between the FP8 and BF16 runs for both models:

Figure 7 Mean response length for both models. FP8 and BF16 stay aligned, so the step-time comparison is made at matched generation lengths.

LinkPerformance analysis

FP8 does not automatically reduce RL step time. Quantization, scale management, and dispatch introduce overhead, and generation and training respond differently to these costs.

Generation: lower memory traffic translates into lower latency

Autoregressive decoding is predominantly memory-bandwidth-bound. Each decoding step streams the model weights and reads the active KV cache while performing comparatively little arithmetic. FP8 reduces the representation size of both, lowering the generation phase at matched response lengths.

OPWS also removes rollout-time re-quantization. The weights and scales remain fixed until the next synchronization, activation quantization is fused into GEMM kernels, and decoding replays through a captured CUDA graph. Consequently, the host-side orchestration path is equivalent between the FP8 and BF16 rollout configurations.

Figure 8 Step decomposition at matched response lengths for Qwen3.5-9B. Rows labeled “KL on” include a reference-model forward pass.

Training: GPU savings are offset by host overhead

Training reuses each weight across thousands of tokens, making the measured phase more compute-intensive than decoding. The GPU kernels benefit from FP8: GEMM time decreases from 14.2 to 10.9 seconds, while FP8 parameter all-gather reduces NCCL time from 9.8 to 6.4 seconds. Total measured GPU kernel time decreases from 36.5 to 32.0 seconds per training phase.

The overhead is on the host. Unlike rollout weights, training weights change after every optimizer step. TE must derive block scales and maintain both the row-wise FP8 representation and the transposed representation used by backward computation. Repeating this work across 64 microbatches initially placed the trainer on a host-dispatch-limited path. Increasing the microbatch size and applying dispatch optimizations reduced the attributed host-side time from 52.9 to 16.9 seconds.

Figure 9 Training kernel time by category, FP8 versus BF16. FlashAttention and GDN times are unchanged because the evaluated blockwise recipe applies FP8 to linear-layer GEMMs; attention remains in BF16 in both configurations.

End to end, the training phase changes from 45.6 to 44.0 seconds, making training approximately time-neutral in the current H100 implementation. The end-to-end improvement therefore comes primarily from rollout generation. On 8×H100 with Qwen3.5-9B, total step time is 0.81–0.90× the BF16 time at matched response lengths. The largest observed reduction is approximately 19% for long rollouts.

Figure 10: End-to-end step time on 8×H100 with Qwen3.5-9B, by response length. The FP8 advantage grows as rollouts get longer.

On 8×B200 with Qwen3.5-35B-A3B, the largest observed end-to-end reduction is approximately 23%, again driven by generation. SkyRL therefore uses fp8_recipe="auto" to select blockwise FP8 on Hopper and architecture-native MXFP8 on Blackwell where supported.

Figure 11 End-to-end step time on 8×B200 with Qwen3.5-35B-A3B.

LinkDeep dive: FP8 quantization with FP32 and power-of-two block scales

On Hopper, TE's blockwise recipe can use full-precision FP32 scales or FP32 values constrained to powers of two. On Blackwell, only pow-2 scaling is supported by the tensor cores. The latter has an exponent-only effective representation and can be up to a factor of two coarser. With fp8_param, the optimizer updates an FP32 master copy and re-quantizes it into FP8 parameter storage after every step. Because RL updates can be small, we evaluated whether the coarser scale grid systematically suppresses those updates.

In the profiled DAPO runs, the largest measured per-element master-weight update is |ΔW| = 1.01 × 1e-6, close to the configured learning rate. The mean relative update, |ΔW|/|W|, is 0.6–0.9 × 1e-4 per optimizer step.

Weight-sync path

We conduct ablation experiments on the trainer-side TE 2.11 Float8BlockQuantizer update path on Qwen3.5-9B weights. After long RL optimizer steps, the cumulative displacement of FP8 parameter storage closely matches the displacement of the FP32 master weights under both scale formats:

Figure 12: Left: cumulative FP8 weight displacement over long RL steps, normalized by the FP32 master-s displacement.

The low single-step cosine similarity at RL-sized updates does not, by itself, imply that optimizer updates are lost. At step t, FP8 storage is recomputed as Q(W_t) from the current master weights. Its deviation from the master is therefore the current quantization error, rather than the sum of an independent rounding error introduced at every preceding step. The long-horizon displacement measurement shows that updates are not systematically eliminated even though each step only incurs a portion of the effective weight updates.

Training comparison

Matched 200-step DAPO runs on 8×H100 use identical seeds and data order and differ only in the scale representation:

Figure 13 Reward and pass@8 for FP32 and power-of-two block scales with -fp8 param-. The curves remain close throughout the run.

The per-token train–rollout log-probability gap is the one quantity that separates the two representations. The power-of-two run maintains a higher gap, 0.034 versus 0.026 averaged over the run, consistent with its coarser effective quantization. The two trajectories evolve in parallel and remain bounded, and the difference does not surface in reward or pass@8:

Figure 14 Mean absolute per-token log-probability difference between the rollout engine and trainer under the two scale representations. The power-of-two run carries a consistently higher but equally stable gap.

We observe no material convergence difference between the two scale representations in this configuration. SkyRL uses FP32 scales as the Hopper default.

There is a separate platform constraint. In the tested TE 2.11 integration, fp8_param requires Megatron's distributed optimizer to redirect FP8 parameter storage into packed buffers through TE's replace_raw_data. This path is implemented for the blockwise FP8 tensor used on Hopper, but not for the MXFP8Tensor used by the native Blackwell recipe. Until that integration is available, primary training weights remain in BF16 on the native Blackwell path even though FP8 still accelerates the supported GEMMs.

Memory savings

On Hopper, storing primary quantizable weights in one-byte E4M3 through fp8_param reduces Qwen3.5-9B policy-parameter storage from 8.3 to 5.1 GiB per GPU, a 39% reduction. The effect is larger for MoE models, where the routed experts hold most of the parameters and are all quantizable: for Qwen3.5-35B-A3B with 8-way expert parallelism, the same recipe reduces per-GPU parameter storage from 12.1 to 7.0 GiB, a 42% reduction. In both models, approximately 1.9 GiB remains in BF16 because it belongs to the precision-sensitive tensor set described above. The recovered HBM can be used for longer contexts, larger microbatches, or wider rollout batches.

Because fp8_param currently requires the Hopper blockwise path, these figures describe Hopper deployments; the B200 runs in this post train with BF16 primary weights.

Figure 15 Per-GPU policy parameter storage with and without -fp8 param- on Hopper. The BF16 remainder is the precision-sensitive tensor set.

LinkCurrent Scope and limitations

Performance depends on model architecture, response length, batch shape, kernels, and hardware. The largest gains in this study occur when rollout generation dominates step time.
FP8 does not cover every operation or state. Attention, selected parameters, optimizer state, and master weights use higher precision where required.

LinkPractical takeaways

Treat weight synchronization as part of numerical correctness. Training and rollout kernels can each be stable while independent quantization at their interface changes the data-generating policy.
Attribute performance by phase. FP8's end-to-end gain comes primarily from bandwidth-sensitive decoding in the measured configurations; host-side work currently limits training-phase speedup.
Keep the quantized representation consistent across engines. Values, scales, tensor exclusions, and layout conversion must be coordinated by one configuration rather than independently inferred by the trainer and rollout engine.

LinkReproduction Notes

Core configurations:

# Trainer: FP8 linear-layer GEMMs with an architecture-aware recipe
# Apply the corresponding keys to trainer.ref when a reference model is used.
trainer.policy.megatron_config.transformer_config_kwargs.fp8=e4m3
trainer.policy.megatron_config.transformer_config_kwargs.fp8_recipe=auto  
# Rollout: consume trainer-produced FP8 blocks and scales
generator.inference_engine.fp8_weight_sync_mode=auto
generator.inference_engine.engine_init_kwargs.kv_cache_dtype=fp8_e4m3