DeepSeek-V4.1 Flash on SGLang: from 35 to 873 tokens/s

For launch coverage and recipes, see the day-0 support post and the LMSYS post.

Architecture changes and KV cache compression

V4.1 Flash has a larger backbone than V4 Flash but activates fewer parameters per input token, and its KV cache is far smaller. The rows that matter for serving:

Numbers are from the model config and the technical report.

The causal encoder-decoder structure cuts prefill compute. The 20 decoder layers get their global KV from the encoder's final output, so a long prompt is mostly processed by the first 20 layers. The decoder still needs a local window, which it rebuilds by replaying the last 128 tokens. That roughly halves prefill work in the backbone. Generated tokens are another matter: each one still runs through all 40 layers.

The KV cache compression comes from sharing, pooling and FP4 storage. Only four of the 40 layers produce global KV, layers 2, 8, 14 and 20. The rest read those caches, and a few recompute their own indices. The first three of those caches pool every two tokens into a single entry, and only the last keeps one entry per token. The entries themselves are stored in FP4, at 288 bytes for the main KV and 68 bytes for the indexer K. Per original token that comes to

about a quarter of V4 Flash. Since the local window can always be rebuilt by replay, less state needs to persist, and DeepSeek puts the SSD cache requirement at roughly an eighth of before. One caveat: 890 bytes is the logical size. Some SGLang paths still use a FlashMLA-compatible cache layout, so the memory actually allocated differs.

Two more pieces of the architecture show up throughout the kernel work. CSA2 narrows the indexer's search with a hierarchical candidate filter, so that in the end only the top 512 global positions take part in attention. Engram adds a form of conditional memory through n-gram lookups. They do different jobs in the model, and each brought new indexing, normalization and fusion work of its own.

Kernel optimizations for plain decode

First, a look back at how plain decode went from 35 tokens/s on our first working build to 203 tokens/s. The numbers in this section are the original measurements; the DSpark section switches to the random 4k/1k workload described at the end.

FP8 GEMM: 35 → 118 tokens/s

Some of the dense weights ship in FP8, but their quantization block and scale layout did not match what the backend expected, so those GEMMs were taking a slower fallback path. We now rearrange the scale layout once, when the weights are loaded, and they go straight into the Blackwell MXFP8 GEMM.

That alone took BS=1 from 35.2 to 117.8 tokens/s. When bringing up a new model, checking which kernel a GEMM actually dispatches to is usually worth more than tuning tiles.

Small-operator fusion and GEMV

Token-by-token decode runs a long chain of short kernels. RoPE, FP4 quantization, compressor pooling, RMSNorm and the cache write feed directly into one another, so fusing neighbors saves a launch and a trip through memory each time.

We also stopped rebuilding the per-step request indices and scratch buffers in every layer and shared them across layers instead, which cut out a fair amount of repeated conversion and initialization and took BS=1 from 146.5 to 148.4 tokens/s. Once the fast paths had been validated, we turned them on by default.

mHC: reduction fusion and overlap

mHC keeps four residual streams instead of one. For every attention and MoE sublayer it has to compute mixing coefficients for them and normalize those coefficients with Sinkhorn. None of it is expensive on its own, but it runs once for attention and once for MoE in every layer, and at small batch that adds up. Picking tile sizes based on the number of input rows, and then fusing the statistics reduction with Sinkhorn, took BS=1 from 133.5 to 141.1 and then to 146.5 tokens/s.

The larger gain comes from how V4.1's single-pass mHC is defined. The pre-mix uses coefficients produced by the previous sublayer, so the current attention or MoE can run in parallel with this sublayer's statistics. We run the two on separate streams and join them at the post-mix. Combined with the compressor and indexer fusion and overlap, this took BS=1 from about 152 to 186 tokens/s.

DSpark adaptation and optimization

DSpark ships with the official checkpoint, including three lightweight draft blocks. They take hidden states from the last few layers of the main model, produce logits for several positions at once, resolve the dependencies between draft tokens with a Markov head, and hand the block to the target model for batched verification.

We fix the block size at 5 and simulate acceptance with a target accept length of 5.5. Counting the anchor, target verify handles up to 6 rows for a single request. Plain decode handles one row per request, so the M=1 fast paths had to be reworked for these small batches.

The first group of optimizations went into verify and MoE. We wired the mHC overlap into verify and draft, and made the WO-A projection write directly into the layout the next stage needs. The candidate mask fuses the valid-length check with candidate handling, which cuts down on scans over large buffers. On the MoE side, the router now emits the layout the experts need, input quantization overlaps routing, and the expert-weighted reduction, shared-expert add and all-reduce run as one step, which cuts intermediate write-back.

Next came the small-batch projections and normalization. WO-A uses split-K so more thread blocks work at once, and mHC fuses the mixing of the four residual streams with RMSNorm. The draft's multiple KV projections now reuse the MXFP8 weights and scales instead of the old FP8 path.

Next we fused more of the indexer post-processing and the projections. The post-Top-K score check, invalid-position filtering and KV page address translation are now a single step, and chosen candidate blocks expand straight into a token mask. Q's RoPE is merged into the attention buffer write, and WO-A's split-K reduction does the following MXFP8 quantization itself, skipping an intermediate tensor.

Then we turned to verify-time compression in L2, L8 and L14 (layers numbered from 0). These three layers used to run a chain of small operators to find the previous token, handle the mask, then pool and write the cache. Verify positions are contiguous, so within a request the previous row can be read directly; only the first row needs the ring buffer. We fused pair pooling, RMSNorm, RoPE, quantization and the main KV write into a single kernel, and then reused the fused write for index-K.

Finally, MoE moved from EP4 to TP4 with padding. Split by TP4, the expert intermediate dimension is 576 per GPU, padded to 640 at load time to fit the kernel. Each GPU computes a different slice of the same experts, so uneven expert load causes less waiting. In a same-round comparison with all of the optimizations above in place, EP4 gives 854.64 tokens/s and TP4 gives 873.63 tokens/s, a 2.22% gain; in the trace, the median gap between ranks arriving at finalize dropped from 11.14 to 3.40 µs.

On random 4k/1k with the simulated accept length targeting 5.5, output speed went from 558.24 to 873.63 tokens/s, a gain of about 56.5%. The table keeps the 853.49 measured when C2 verify fusion landed; the 2.22% above comes from this round's EP4 / TP4 comparison.

None of these results use the new kernels DeepSeek released with V4.1.

How to reproduce: random 4k/1k, simulated accept length fixed at 5.5

This section reproduces the DSpark points 11 to 16 in the figure: 4,096 random input tokens, a fixed 1,024 output tokens, and a simulated accept length targeting 5.5. The server configuration controls the accept length, and every version uses the same input.

Use the SGLang code at BBuf/sglang@835c3909 and the official checkpoint at revision dba1be0a, on 4× GB300. We were on PyTorch 2.13.0+cu130, FlashInfer 0.6.18, Triton 3.7.1, sglang-kernel 0.4.6.post1, sgl-deep-gemm 0.1.7 and CUTLASS DSL 4.6.2.

Below is the TP4 serving command behind the 873.63 tokens/s, run from the root of that SGLang checkout. --tp 4 --ep-size 1 means both attention and MoE use TP4; this version applies the padding when it loads the weights.

export MODEL_PATH=/path/to/DeepSeek-V4.1-Flash
export SGLANG_RAGGED_VERIFY_MODE=static
export SGLANG_SIMULATE_ACC_LEN=5.5
export SGLANG_SIMULATE_ACC_METHOD=match-expected
CUDA_VISIBLE_DEVICES=0,1,2,3 PYTHONPATH="$PWD/python" MAX_JOBS=16 \
python -m sglang.launch_server \
  --model-path "$MODEL_PATH" \
  --served-model-name deepseek-ai/DeepSeek-V4.1-Flash \
  --tp 4 --ep-size 1 --trust-remote-code \
  --moe-a2a-backend none --moe-runner-backend flashinfer_mxfp4 \
  --mem-fraction-static 0.80 --max-total-tokens 33554432 \
  --chunked-prefill-size 4096 \
  --cuda-graph-bs-decode 1 2 4 8 16 32 64 \
  --max-running-requests 128 \
  --speculative-algorithm DSPARK --speculative-dspark-block-size 5 \
  --skip-server-warmup --reasoning-parser deepseek-v41 \
  --random-seed 42 --decode-log-interval 10 \
  --host 127.0.0.1 --port 30021

To reproduce the EP4 comparison, change --ep-size 1 to --ep-size 4 and keep everything else. The full TP4 launch script is available as launch-tp4.sh.

The input is generated with the fixed random seed 42: special tokens are excluded from the model vocabulary, and 4,096 token ids are drawn uniformly. The ids go in directly, with no chat template, and every configuration reuses them.

• The random input (prompt.json).
• Input construction, benchmark script and per-run results.

In a second terminal:

ASSET_URL=https://raw.githubusercontent.com/BBuf/how-to-optim-algorithm-in-cuda/master/large-language-model/sglang/assets/deepseek-v41-kernel-journey/random-dspark
curl -fL "$ASSET_URL/prompt.json" -o prompt.json
curl -fL "$ASSET_URL/benchmark.py" -o benchmark.py
python -m pip install requests
python benchmark.py bench --prompt prompt.json --max-tokens 1024 --out result --repeat 6

The script uses temperature=0 and ignore_eos=True, and checks on every run that the input was 4,096 tokens and the output 1,024 tokens. Inputs go through /generate; the script calls /freeze_gc after startup, clears the cache before each run, and discards one warm-up. Each server launch is measured for 6 runs. The small-batch projection, indexer post-processing / projection fusion, C2 verify and TP4 configurations were each launched twice independently, with the median taken over the 12 runs. For this round's EP4 / TP4 comparison the launches alternated TP, EP, TP, EP, all measurements were kept, and the profiler was off while measuring throughput.

match-expected accepts 5 or 6 tokens on each round, so the expected accept length is 5.5. A finite number of rounds and truncation at the last step move the measured value slightly, and the table keeps the actual values. Acceptance is simulated, so the generated text is not used to judge answer quality; simulation mode also disables the in-graph acceptance path.

Throughput is computed as the number of tokens added after the first streamed event divided by the time from the first event to the last, and does not include full prefill. Accept length counts the token the target model produces, so with block size 5 the ceiling is 6.

The table below puts DSpark off and on side by side, plus the MoE TP4 + padding configuration. All entries use the same code, the same random input, the same output length and the same timing method, with attention on TP4 throughout; EP4 / TP4 in the header refer to the MoE configuration.

Related links

• LMSYS Blog: SGLang and Miles Add Day-0 Support for DeepSeek-V4.1: the team's full day-0 write-up, covering inference and RL training.
• SGLang DeepSeek-V4.1 deployment guide: launch configuration, hardware support and tuning notes.
• SGLang DeepSeek-V4.1 code, the dsv4.1 branch.
• Miles DeepSeek-V4.1 Flash training guide: training environment, checkpoint preparation and RL launch configuration.
• Miles on GitHub.
• DeepSeek-V4.1 technical report.

For the kernel implementations, see C2 verify compression, indexer post-processing, Q RoPE / store and WO-A / MXFP8 directly.

Acknowledgments

Thanks to the DeepSeek team for open-sourcing DeepSeek-V4.1, and to everyone on the SGLang and Miles teams and in the community who helped with the model support, the kernels, testing and review. Some of the kernels were developed with the KDA 0.5 framework, and we are grateful to Humanize and Kernel Design Agents for the tools and the workflow.