Unified Radix Cache in SGLang: one tree for hybrid model prefix caching
Radix Cache is a major component in SGLang for cross-request cache reuse. Nowadays models apply hybrid attention mechanisms, but the traditional Radix Cache is only designed for full attention KV. Therefore we designed SGLang's Unified Radix Cache for hybrid attention mechanisms. Multi-turn hit rates stay near 98% with a Mooncake L3 tier. The session-aware configuration lowers SWE-bench TTFT by up to 16.6%. The Rust tree core lowers SWA TTFT by up to 42%.
Unified Radix Cache in SGLang
One radix tree, one slot per component, the same prefix identity across GPU, host, and external tiers
TP4 · 4×H200
Effective input throughput (tok/s) · DeepSeek-V4-Flash15.5× with L3
9.4K
L1
GPU only
14.3K
L1 + L2
+ host
145.5K
L1 + L2 + L3
+ Mooncake
Prefix cache hit rate, later rounds · DeepSeek-V4-Flashnear 98%
Prefix caching reuses KV when requests share a token prefix. Under full attention, a later request can reuse the cached prefix and compute only the new tokens.
Hybrid models combine three kinds of data with different reuse rules: FULL (full attention) needs KV along the matched prefix, SWA (sliding window attention) needs a continuous window immediately before the reuse boundary, and MAMBA needs a snapshot of the recurrent state, or checkpoint, at that boundary.
Reuse semantics diagramCandidate boundary t8 · SWA window W = 4 · gray cells are outside the required range
Matched token prefixt1t2t3t4t5t6t7t8t9t10t11t12
FULL · whole path
SWA · required window
MAMBA · exact checkpoint
DeepSeek-V4 pairs FULL with SWA, Kimi-K3 pairs FULL with MAMBA for its KDA recurrent state, and Inkling uses all three. SGLang handles these combinations on one shared tree, with each component defining its own reuse rule.
Class matrix vs components diagram
Before: one specialized cache class per combination × capability · 2 in total
FULLmatchinsertlockevict
FULL + SWAmatchinsertlockevict
After: one tree plus pluggable components · components ×2
UnifiedTreeCore: shared matching, split, insert, lock, evict mechanicsUnifiedRadixCache: pool orchestration
Components:FULLSWAsidecar ×N
Unified Radix Cache
Design and mechanism
Each radix node stores a token span and a slot for each component. FULL and SWA slots hold KV page indices; MAMBA holds a checkpoint at the prefix endpoint. SWA data outside the required window can stay cached or be evicted independently, leaving an empty slot called a tombstone.
Components handle their own splits, inserts, locks, and evictions. Reclaiming part of a node's SWA data starts by splitting the node at the retention boundary. A MAMBA checkpoint stays at its original prefix endpoint; a request reusing it obtains a private copy through copy-on-write, then continues updating its state.
Unified radix tree replayWindow W = 4 · one token per cell · earlier CSFA retained after request 2
t = 0/8
Request 1 · token streamABCSFA
The safe reuse boundary
Matching walks the matched path, and each component checks its data at every candidate boundary. Each candidate is evaluated independently. Computation resumes from the deepest boundary accepted by all components.
Reuse-boundary voting diagramScenario: n3 has tombstoned SWA window slots; n4 has no MAMBA checkpoint
HiCache extends Unified Radix Cache across GPU L1, host memory L2, and external storage L3. See HiCache in SGLang for its tiering and data transfers.
Components and sidecars
DeepSeek-V4's FULL and SWA use separate page indices, mapped by the allocator. Auxiliary pools for compressed KV, indexer buffers, and compressor states act as sidecars: they share their component's indices and transfers. Three follow FULL and two follow SWA.
Index reuse diagramThe blog's normalized six-page case · only the final two SWA pages retained here
FULL pages (component)
sidecars ×3 follow FULL
↓↓allocator translates
SWA window slots (component)
sidecars ×2 follow SWA
Page 4: FULL's F4 and its three sidecars share page number 4. The allocator translates F4 to SWA's S0, and SWA's two sidecars copy page number 0.
Multi-turn benchmark
Multi-turn conversations keep extending the reusable prefix. A comparison of L1 only, L1+L2, and an additional 500 GiB Mooncake L3 tier shows DeepSeek-V4-Flash reaching 145.5K effective input tokens/s with L3, up from 9.4K with L1 only, with average TTFT below 9 seconds.
Multi-turn benchmark results diagramEffective input throughput = total full prompt length ÷ wall-clock time; cache-hit prefix tokens count
DeepSeek-V4-FlashFULL+SWA · 4×H200 TP4 · 48 clients · 60 rounds · 4,096 in + 16 out per turnEffective input throughput (tokens/s)
L1
L1+L2
L1+L2+L3
Cache hit rate by round
Final metrics with L3hit rate~98%avg TTFT< 9 s
Inkling-SmallFULL+SWA+MAMBA · 8×H200 TP8 · 64 clients · 30 rounds · 1,216 in + 64 out per turnEffective input throughput (tokens/s)
L1
L1+L2
L1+L2+L3
Cache hit rate by round
Final metrics with L3hit rate96.8%avg TTFT1.23 s
Session-aware eviction
Applications use a stable session_id to identify requests from the same conversation. SGLang tracks that session's references to cached entries. Calling /close_session removes its references; shared prefixes keep those from other sessions.
FULL orders unlocked, eligible entries by reference presence, reference count, and then the base eviction policy, usually reclaiming unreferenced data first. Session retention applies to L1/L2; the storage backend manages L3 eviction.
Session-aware eviction diagramFULL entries · unlocked and eligible for eviction · AB is shared by A and B
t = 0/5
Ordinary LRU
A1ABB1C1C2recently used
Session-aware eviction
A1A · activeABA + B · 2B1B · activeC1unreferencedC2unreferenced
A and B retain session references, AB is shared by both, and C1/C2 are unreferenced.
SWE-bench results
On SWE-bench, enabling Unified Radix Cache together with session-aware eviction lowers TTFT by up to 11.0% on DeepSeek-V4-Pro and 16.6% on Qwen3.5-397B-A17B, compared with HiRadixCache and LRU.
SGLang runs tree traversal, lock bookkeeping, and LRU updates in Rust to reduce their CPU overhead on the scheduler. Python handles physical KV allocation and cache orchestration, applying the operations Rust returns to the pools.
Rust and Python ownership diagramOwnership of tree structure and pool operations
Python owns
request-to-token mappings
physical KV allocation
pool ops and orchestration
match / insert / evict calls→←deferred actions
Rust tree core owns
radix topology
per-component lock accounting
intrusive LRU lists
eviction walks
The L1-only Rust prototype #29074 reduced TTFT by 38% for SWA, 10% for full attention, and 5% for hybrid SSM against the Python tree in a 200-turn conversation benchmark.
Rust prototype benchmark diagramPrototype #29074 · L1 only · 200 turns · 100 in + 100 out per turn · 6 trials · sequential runs on the same GPUs
SWA gpt-oss-20b · TP2
All 200 turns
Turns 176–200
Full attention Qwen3-32B · TP2
All 200 turns
Last 25 turns
Hybrid SSM Qwen3-Next-80B-A3B · TP4
All 200 turns
Last 25 turns
The subsequently merged Rust TreeCore #32710 supports FULL, SWA, and MAMBA combinations and HiCache. Session-aware caching continues to use the Python tree core.
