Step 5 Preview: Day-0 Inference and RL Support with SGLang and Miles

An AI agent often needs several steps to finish a job. When fixing a bug, it may read code, run tests, inspect errors, and try again, adding information that it may need later. As that history grows, the model has more to read before it can respond, while the GPUs must keep up with other users running their own tasks.

Step 5 Preview has already been served in production with SGLang’s inference engine and routing layer. Alongside this production serving work, SGLang and Miles provide day-0 inference and reinforcement learning support for Step 5 Preview. On four GB300 GPUs, at an MTP acceptance length of 2.85, FlashInfer TRT-LLM MoE decodes a single request at about 299 tokens/s, and MegaMoE reaches about 740 output tokens/s per GPU at 64 concurrent requests. Miles also demonstrates a 79-step GRPO training run, detailed below.

Get Started

Follow the SGLang Cookbook for setup instructions, then launch Step 5 Preview with:

sglang serve \
  --trust-remote-code \
  --model-path stepfun-ai/Step-5-Preview \
  --tp 4 \
  --moe-runner-backend flashinfer_trtllm \
  --attention-backend fa4 \
  --page-size 64 \
  --kv-cache-dtype bfloat16 \
  --mem-fraction-static 0.9 \
  --swa-full-tokens-ratio 0.03 \
  --chunked-prefill-size 16384 \
  --max-running-requests 64 \
  --cuda-graph-backend-prefill breakable \
  --flashinfer-allreduce-fusion-backend trtllm \
  --speculative-algorithm EAGLE \
  --speculative-num-steps 3 \
  --speculative-eagle-topk 1 \
  --speculative-num-draft-tokens 4 \
  --enable-multi-layer-eagle \
  --reasoning-parser step5 \
  --tool-call-parser step5 \
  --host 0.0.0.0 \
  --port 30000

For reinforcement learning and post-training, follow the Miles Cookbook.

Overview

Step 5 Preview combines 600B total parameters with 27B active per token, a 1M-token context window, and image input. Its main architectural choices address different parts of agent workloads: sparse experts limit the computation used for each token, hybrid attention controls how much history each layer reads, and three multi-token prediction (MTP) layers support speculative decoding.

Model feature	Configuration	What it enables
Sparse MoE	600B total parameters, 27B active; 8 of 352 routed experts plus one shared expert	Uses a subset of the model for each token
Long context	Up to 1M tokens	Room for documents, code, and accumulated task history
Image input	Text and images	Includes screenshots and visual material in a request
Hybrid attention	3 sliding-window layers followed by 1 sparse layer; 92 layers total	Combines recent context with selected older information
MTP	3 prediction layers	Proposes multiple tokens for verification together

Hybrid Attention

A long-running agent needs both the latest tool result and information from much earlier in the task. Reading the full history in every layer would add work as that history grows. Step 5 Preview instead combines a short recent window with selective access to older context, giving the serving system less attention work to perform than reading the entire history in every layer.

Three out of every four layers read the most recent 512 tokens. The fourth scores summaries of 8-token blocks and selects up to 512 historical blocks, then reads the individual tokens in those blocks and the current block. This is how the model reaches back into earlier context while keeping most layers focused on recent information.

SGLang Day-0 Support

For an application team, the serving challenge has two parts: keep responses fast as more users arrive, and avoid processing the same task history again when an agent takes its next step. SGLang addresses these through sparse attention kernels from StepFun, a choice of MoE backends, and routing that takes cached context into account.

Kernels from StepFun

StepFun developed optimized kernels for Step 5 Preview's sparse attention and contributed them to SGLang for this release. We thank the StepFun team for these kernels, for their support of the open-source community, and for working closely with us to bring them into SGLang to run optimally on NVIDIA GPUs from Day 0:

• Sparse prefill: Gathers the blocks selected by the indexer for a whole batch into one shared set, then runs attention over that set using per-query masks.
• Sparse decode: Runs decode attention over only the selected blocks, splitting long KV histories into partitions and merging the partial results.
• Indexer scoring: Scores each query directly against compressed history stored in SGLang's paged cache, for both prefill and decode.

Performance

SGLang offers two MoE backends for Step 5 Preview, each suited to a different traffic pattern. FlashInfer TRT-LLM MoE delivers strong low-latency performance at small concurrency: a single request decodes at about 299 tokens/s. MegaMoE delivers higher throughput as concurrency grows: at 64 concurrent requests it produces about 740 output tokens/s per GPU, and it prefills a 128K-token prompt at about 11,440 tokens/s per GPU, averaged over the full prompt. Decode results use an MTP acceptance length of 2.85.

Production Serving

Production traffic brings a different challenge from a fixed benchmark: requests arrive in bursts, input lengths vary, and workers carry different loads. The important questions are how long users wait for an answer, how smoothly it continues generating, and how much output each GPU can deliver while meeting those latency targets.

The routing layer helps by choosing where each request runs. An agent often sends much of the same context again after a tool call; a worker that already holds that prefix can reuse the cached computation. The router tracks both cached prefixes and worker load, so it can favor reuse without concentrating requests on an overloaded worker. It supports Kubernetes deployments and scales across replicas as the worker pool grows. The router is being prepared for a formal release; this section describes its use in the Step 5 Preview serving work.

Two optimizations developed for Step 5 Preview reduce repeated CPU work in this path:

Reuse tokenization across the router and engine. The router needs token IDs to match prefixes, so tokenizing the same request again in the engine duplicates work. For supported text chats, it forwards those IDs and the engine skips both tokenization and chat templating. A parity check enables this path only when the IDs match what the engine would produce, with fallback otherwise. Multimodal requests and unsupported formats keep their original inputs. This removes redundant processing while preserving the engine's input handling.

Keep cache updates lightweight. Workers send frequent events describing changes to their KV caches. Routing needs only the block hashes in those events, so the decoder skips the token-ID payloads without allocating arrays for them, reducing work as the event stream grows. Per-worker metrics track missing events, publisher resets, and decoding failures so operators can detect problems with the cache view used for routing.

Miles Day-0 Support

Once a model is serving an application, a team may want to train it for a particular task. Miles adds this option for Step 5 Preview, with a working RL run that demonstrates both learning progress and agreement between the training and rollout systems. The rollout system generates the responses used for training, so checking that the two systems score those responses similarly is an important integration check.

In a 79-step GRPO run on DAPO-Math-17K, the five-step moving mean of training reward rises from approximately 0.44 to 0.75. This shows that the model is learning on the math training workload. The run uses a 2K response cap and 256 samples per step.

The right-hand panel checks trainer/rollout agreement. Per-token KL stays roughly between 0.0006 and 0.0009, and the mean absolute log-probability difference remains low across the run. Together, the two panels show that this configuration can train the model while keeping the trainer and rollout engine closely aligned.

For teams planning their own post-training, this is evidence that the RL path has been exercised through a learning run. It is not yet evidence of better coding agents or a higher held-out math score. Those improvements need separate evaluations on the tasks the team wants to deploy.

Acknowledgments

This work was a collaboration between the SGLang & Miles team and the StepFun team, together with NVIDIA and Alibaba Bailian.

Alibaba Bailian: Peng Yu, Zhangheng Huang, Zhiyang Huang, Zhen Wang