Kimi K3: Architecture and SGLang Day-0 Support

The Kimi K3 architecture

K3 is a 2.8T-parameter MoE model with 104B parameters active per token, and it supports a native 1M-token context. K3 has 93 layers: 69 use KDA linear attention and 24 use gated global MLA. The first layer is a dense FFN; the remaining 92 use Stable LatentMoE. The 93 layers split into 8 blocks of 12, and Attention Residual aggregates across blocks. Click the diagram for the complete structure:

K3 at a glance

Click a component for detail

KV cache memory estimation

Start with memory: both MLA and MHA KV caches grow with context. The Kimi K3 day-0 support blog puts K3's compressed MLA KV cache at about 27KB per token per GPU across all 24 MLA layers, which works out to roughly 1.125KB per layer. If all 93 layers used MLA, a 1M-token request would need about 105GB on one GPU (1.125KB × 93 layers × 1M tokens). Uncompressed MHA at the same shape would store 48KB per layer per token (96 heads × 128 dims × K/V × bf16), more than 40× the MLA latent. Bringing that number down is the first problem to solve, which is why K3 cannot keep context-growing softmax attention in most layers.

K3 therefore interleaves 3 KDA + 1 MLA. KDA is a linear-attention variant whose recurrent state has a fixed size and does not grow with context; the mechanism comes later. It takes about 54MB per request per GPU and updates in place each step, while MLA keeps global attention. The model uses no explicit position encoding (NoPE); position comes implicitly from the KDA recurrence. The diagram below contrasts memory use between all-MLA and the KDA+MLA mix:

KV cache comparison

t = 0/16

KKKMKKKM3 KDA + 1 MLA interleaved, ×23 blocks, plus one final MLA layer: 93 total

Hypothetical: all 93 layers MLAcontext 0K token · cache 0.0 GB · per token 105 KB

K3: 24 MLA + 69 KDA layerscontext 0K token · cache 0.1 GB · per token 27 KB

The next sections walk from classic MHA to KDA and compare what changes at each step, which makes it clear why Kimi K3 picks KDA, a linear-attention variant.

Attention mechanisms in detail: classic MHA

In ordinary MHA, decoding token $t$ first appends its own KV to the cache, then reads all $t$ entries in three steps:

Score: compare against every token $i \le t$ with $z_{t,i} = \frac{q_t \cdot k_i}{\sqrt{d_k}}$ . The subscript $t,i$ reads “step $t$ 's query against key $i$ ”.
Normalize: compute $Z = \sum_{j=1}^{t} \exp(z_{t,j})$ , then assign attention weight $a_{t,i} = \exp(z_{t,i}) / Z$ .
Aggregate: take the weighted sum of values, $o_t = \sum_{i=1}^{t} a_{t,i} \cdot v_i$ .

A length- $N$ sequence therefore performs about $N^2/2$ dot products: $O(N^2)$ compute and $O(N)$ cache. In the diagram, every token has a line to each of the tokens before it.

MHA diagram

t = 0/9

cache 0 cells · dot products this step 0 · cumulative 0

Softmax attention buys per-position reads, which gives it the most complete global memory, but the cost is just as clear. Every new token rescans the whole history, so the whole history has to be stored and compute stays at $O(N^2)$ .

Attention mechanisms in detail: linear attention

Linear attention no longer keeps one KV entry per token. It accumulates history into a fixed-size matrix, $S = \sum_i k_i v_i^{\top}$ . S is a fixed-size memory matrix, and every new token writes into it once it is computed: k sets the write direction, meaning where the token points in vector space; v is the content written along that direction; and q reads with $o = S^{\top} q$ . No matter how long the context becomes, each step reads and writes only S.

The diagram below gives both a math view and the same sentence used earlier, so the difference is visible directly:

Naive linear attention diagram

t = 0/4

2D teaching slice: the real S is 128×128 per head

2D key space (teaching slice)

kₐ=(1/√2)[1,1]ᵀkᵦ=(1/√2)[1,−1]ᵀ

arrows = full keys; axes = channels

S entering this step

ch₁

ch₂

−0+

S is empty

→directly add kvᵀS←S+kvᵀ

S after this step

ch₁

ch₂

−0+

S is empty

old A contributionB contributionnew contribution from A=4Colors only trace provenance; the real S stores only the summed values

In MHA, "it" can point directly to "cat" because every historical token has its own KV position. The linear-attention state S has no token-position axis, so its readout mixes all historical writes in one fixed state. The diagram above shows the simplest form of naive linear attention: plain accumulation.

Attention mechanisms in detail: DeltaNet

Naive linear attention keeps adding to its state and never overwrites an old value. In the simplest case, if the same key direction receives 1 and then 4, both stay in S, and the next read returns $1+4$ . Different contexts usually give the same word different keys, but over a sequence as long as 1M tokens a fixed-capacity state can still fail to keep those writes apart. Say the text first says Zhang San lives in Beijing, so the state records Zhang San → Beijing, and later says Zhang San moved to Shanghai. Asked where Zhang San lives, the readout is likely a blend of Beijing and Shanghai rather than Shanghai cleanly overwriting Beijing.

DeltaNet's answer is an error-directed edit to the state. It introduces a dynamically produced scalar gate $\beta_t$ that sets how strong the current update is. Before writing, the model reads the prediction the state already holds along the current key, $r_{\text{old}} = S_{t-1}^{\top} k_t$ . It then writes the difference between the target value and that prediction instead of accumulating $v_t$ : $S_t = S_{t-1} + \beta_t k_t (v_t - r_{\text{old}})^{\top}$ . For a normalized key, the new readout is $r_{\text{new}} = (1-\beta_t)\, r_{\text{old}} + \beta_t v_t$ .

The coefficient $\beta_t$ controls rewrite strength: $\beta_t = 1$ fully replaces the old association, smaller $\beta_t$ retains more of the old value, and $\beta_t = 0$ does not rewrite it at all.

DeltaNet diagramAdjust β and see how A=4 rewrites A=1

t = 0/4

2D key space (teaching slice)

kₐ=(1/√2)[1,1]ᵀkᵦ=(1/√2)[1,−1]ᵀ

arrows = full keys; axes = channels

S entering this step

ch₁

ch₂

−0+

S is empty

→waiting for a token

S after this step

ch₁

ch₂

−0+

S is empty

old A contributionB contributionnew contribution from A=4Colors only trace provenance; the real S stores only the summed values

That fixes same-direction rewriting but leaves a capacity question: once unrelated history keeps entering a fixed-size S, what should decay? Even unrelated key vectors are rarely perfectly orthogonal in practice, so a write in one direction projects into another and creates crosstalk. The longer the context, the larger that effect gets.

Math details: A/B in the diagram and the crosstalk term

A and B are two unit directions $k_a, k_b$ in key space $\mathbb{R}^{d_k}$ ( $\lVert k_a \rVert = \lVert k_b \rVert = 1$ ), not token slots or individual channels. $A=1$ is shorthand for “the key points along $k_a$ and the value is the scalar 1.” Colors only trace where each write came from; the real S stores their sum.

Suppose S holds two associations, $S = k_a v_a^{\top} + k_b v_b^{\top}$ . Reading with $q_a = k_a$ gives $o_a = S^{\top} q_a = (k_a^{\top} k_a)\, v_a + (k_b^{\top} k_a)\, v_b = v_a + \cos\theta \cdot v_b$ , where $\theta$ is the angle between the two keys. The second term is B leaking into A, scaled exactly by the inner product $k_a^{\top} k_b$ .

For a general set of writes $\{(k_i, v_i)\}$ the readout is $o(q) = \sum_i (k_i^{\top} q)\, v_i$ , and interference between writes is governed by the Gram matrix $G_{ij} = k_i^{\top} k_j$ . The readout is crosstalk-free iff all keys are pairwise orthogonal — and $\mathbb{R}^{d_k}$ admits at most $d_k$ such directions, so once more associations are stored, crosstalk in a fixed-size state cannot be eliminated, only managed.

Crosstalk diagramKeep kₐ fixed and drag kᵦ; the example uses qₐ=kₐ and vₐ=vᵦ=1

or drag kᵦ in the figure

Attention mechanisms in detail: KDA

KDA's gating works per channel, so first, what a channel is. K3's hidden size is 7168; each of the 96 KDA heads uses its own learned projection to compress the model-wide 7168-d hidden state down to 128 dimensions, and a channel is one dimension of that projected 128-d space. Key channel $j$ sets the coefficient with which content is written into row $j$ of S:

Channel diagram

hidden statexₜ ∈ ℝ⁷¹⁶⁸

→Wₖ ∈ ℝ¹²⁸ˣ⁷¹⁶⁸

k for one head

chⱼ…

kₜ ∈ ℝ¹²⁸the highlighted dim = one channel

A fixed state does not grow with context, which is exactly why it saves memory; the cost is that more and more history has to share one S. The delta rule can rewrite the same or a nearby key direction, but it does not proactively clear irrelevant old signal, which is the crosstalk above. KDA therefore adds channel-wise gating: for every token and head, the model produces a vector $\alpha_t$ from the current hidden state, applies $\mathrm{Diag}(\alpha_t)$ to scale the channels of the old state, and then performs the DeltaNet rewrite.

Math details: how KDA gates and rewrites the same S

The full update is $S_t = \big(I - \beta_t k_t k_t^{\top}\big)\,\mathrm{Diag}(\alpha_t)\, S_{t-1} + \beta_t k_t v_t^{\top}$ . Term by term: $\alpha_t \in (0,1)^{d_k}$ is a vector gate computed from the current hidden state, and $\mathrm{Diag}(\alpha_t)$ scales row $j$ of S by $\alpha_t[j]$ — every historical contribution in that row decays together; it neither picks A or B apart nor rotates keys toward orthogonality. $\beta_t \in (0,1)$ is a scalar write strength.

Read one update in two steps. Gate first: $\tilde S = \mathrm{Diag}(\alpha_t)\, S_{t-1}$ . Then delta-rewrite along the current key: with $r_{\text{old}} = \tilde S^{\top} k_t$ (the decayed state's prediction for this key), $S_t = \tilde S + \beta_t k_t (v_t - r_{\text{old}})^{\top}$ . For a unit $k_t$ , the post-update readout along the same key is $S_t^{\top} k_t = (1-\beta_t)\, r_{\text{old}} + \beta_t v_t$ : $\beta_t = 1$ replaces fully, $\beta_t = 0$ keeps everything.

Unrolling the recurrence shows where position comes from: $S_t = \sum_{s \le t} \Big[\prod_{u=s+1}^{t} \big(I - \beta_u k_u k_u^{\top}\big)\mathrm{Diag}(\alpha_u)\Big]\, \beta_s k_s v_s^{\top}$ . Earlier writes pass through more gates and rewrites, so decay accumulates with distance — the mechanism that lets K3 drop RoPE. K3 also pins a lower bound on $\alpha$ 's log-decay with a scaled sigmoid ( $g_{\min} = -5$ , so every step's retention exceeds $e^{-5} \approx 6.7\times10^{-3}$ ): chunkwise computation has to rescale keys by the reciprocal cumulative decay, and bounding that numerical range lets both diagonal and off-diagonal tiles run as dense Tensor Core matmuls, removing the position-pair diagonal path Kimi Linear needed. The output gate moves from Kimi Linear's low-rank parameterization to an input-dependent full-rank projection.

The diagram compresses the real 128-d key space into a 2D teaching slice: $k_a = \tfrac{1}{\sqrt{2}}[1,1]^{\top}$ and $k_b = \tfrac{1}{\sqrt{2}}[1,-1]^{\top}$ , both spanning $\mathrm{ch}_1$ and $\mathrm{ch}_2$ . The gate scales both rows first, then the delta rule writes along the full $k_a$ the residual needed to reach target value 4.

KDA diagramAdjust α₁, α₂, and β to see how the state changes

t = 0/4α₁α₂β

2D key space (teaching slice)

kₐ=(1/√2)[1,1]ᵀkᵦ=(1/√2)[1,−1]ᵀ

arrows = full keys; axes = channels

S entering this step

ch₁

ch₂

−0+

S is empty

→Diag(α)α=1

② S after Diag(α)

ch₁

ch₂

−0+

S is empty

→delta along full kₐfirst write

③ S after delta along kₐ

ch₁

ch₂

−0+

S is empty

old A contributionB contributionthis step's residual update along kₐColors only trace provenance; the real S stores only the summed values

So Kimi K3 needs no RoPE, because the KDA state update already carries position. An earlier token passes through more decay steps and state transforms before it reaches the current position, while a nearer token passes through fewer, and the model reads order and distance from that. Where RoPE encodes position explicitly with a fixed rotation, KDA models it implicitly through learnable, input-dependent decay. K3 also lower-bounds that decay to keep the chunkwise numerical range in check, so every causal tile in the KDA kernel can run on Tensor Cores; the output gate becomes a full-rank projection.

Attention Residual

An ordinary Transformer's residual stream is like one draft passed from the bottom layer to the top, with every layer editing the same page. This keeps information moving upward, but by layer 90, shallow details are mixed with dozens of later updates; a deep layer cannot reopen an earlier version directly.

Attention Residual keeps a few historical versions on the side. K3 splits its 93 layers into 8 blocks — 12 layers each for the first seven, 9 for the last — and each block sums its own layer outputs into one block representation. No layer has to rely only on the current draft: each one uses its own learned pseudo-query to weight the embedding, every preceding block's summary, and its own block's partial sum so far, and the mixture is that layer's input — the once-accumulating residual is taken over inside a block by that partial sum:

Attention Residual diagram

Standard residualhℓ = hℓ₋₁ + Fℓ(hℓ₋₁)

Shallow information must survive every intervening addition; a deep layer cannot retrieve one earlier output on its own.

Attention Residualhℓ = Σᵢ αℓᵢ bᵢ

A deep layer forms a softmax-weighted mixture of saved summaries rather than selecting one exact layer, creating shorter paths for information and gradients.

B₁–B₇ hold 12 layers each; B₈ is the 9-layer L85–93 tail. Retrieval happens per layer: from the block's second layer onward the candidate set also holds the block's partial sum so far — that is the ordinary residual inside the block.

Every layer can retrieve earlier representations by content instead of squeezing everything into one accumulated stream; one summary per 12 layers shrinks the candidate set from 93 to 9 and keeps the cost down.

More summaries mean finer retrieval but more state, bandwidth, and pipeline traffic.

Math and cost: why K3 saves one summary every 12 layers

Let $b_0$ be the embedding, $b_n$ the summary of block $n$ (the accumulated outputs of its layers), and $b_n^{\,i-1}$ the partial sum over the first $i-1$ layers of block $n$ . Retrieval is done per layer: layer $i$ of block $n$ has its own learned pseudo-query $q$ , and its candidate set is $\{b_0, b_1, \ldots, b_{n-1}\}$ when $i = 1$ , plus $b_n^{\,i-1}$ when $i \ge 2$ . Scoring $\alpha = \mathrm{softmax}\big(s(q, \cdot)\big)$ with a dot-product-style score $s$ , the mixture $h = \sum \alpha \cdot b$ is that layer's input. Note that AttnRes does not add a term on top of a residual; it replaces layer-by-layer accumulation itself. Weight can concentrate on one summary, but the choice is never hard.

Cost: full AttnRes stores every layer output — $L$ vectors of width $d$ per token, i.e. $O(Ld)$ . The block version stores $N = \lceil L/12 \rceil$ summaries plus the embedding, i.e. $O(Nd)$ . With $L = 93$ and block size 12, K3 keeps $N = 8$ summaries, i.e. $9 \times 7168$ numbers per token. These summaries also travel with activations through the pipeline, so $N$ enters communication volume directly. Note that blocking shrinks the candidate set (93 → 9), not the number of retrievals: all 93 layers still retrieve once each. The kernel just batches the cross-block reads per block and lets each layer merge in its own intra-block partial sum with an online softmax.

Granularity vs. cost: smaller blocks (one layer per block in the limit) give finer depth-wise retrieval but grow state, memory bandwidth, and pipeline traffic linearly in $N$ ; larger blocks are cheaper, but layers inside a block are already summed and cannot be retrieved separately. The AttnRes paper reports the cost as: end-to-end inference latency overhead below 2% on typical inference workloads; on the training side, negligible without pipeline parallelism and a measured end-to-end overhead below 4% with it.

LatentMoE

Every FFN except the first uses MoE: each token selects 16 of 896 routed experts, while 2 shared experts are always active. The routed-pool selection rate is therefore $16/896 \approx 1.8\%$ ; the separate whole-model ratio is $104\text{B}/2.8\text{T} \approx 3.7\%$ active parameters per token.

Routing still scores the full 7168-d hidden state, but the selected experts compute in a 3584-d latent space and project back to full width. This halves both the 16-way all-to-all dispatch traffic and expert weight volume:

LatentMoE diagram

16/ 8961.8%

At this point we have seen how KDA, MLA, Attention Residual, and LatentMoE form K3's hybrid architecture. But the model design only answers how to compute. In production, SGLang still has to answer how these different states fit in memory and how prefill is parallelized. The next half starts from those system questions.

SGLang's day-0 adaptations

K3's hybrid architecture creates two inference states with completely different lifecycles:

	MLA (24 layers)	KDA (69 layers)
State shape	KV cache, appends per token	Recurrent state, fixed size
Growth	~27KB per token per GPU, append-only	~54MB per request per GPU (`TP=8`), overwritten in place each step

MLA appends KV per token, while KDA reserves a fixed amount per request and overwrites it every step. That difference propagates into memory allocation, prefill parallelism, and decode scaling—the three parts of SGLang's day-0 work.

RadixAttention: prefix caching for KDA state

Traditional RadixAttention can share a prefix across requests because attention KV only appends as tokens arrive and never rewrites history. KDA's recurrent state $S$ is different: every token it reads overwrites the state in place. Suppose D and E both hit the prefix ABC. If they shared one $S(\text{ABC})$ , D's next step would turn it into $S(\text{ABCD})$ , so E could no longer start from the state after ABC — the mechanism rules prefix caching out. So how do we keep RadixAttention's cache tree?

SGLang's answer is to never let a request mutate the shared state in the radix tree. A KDA state stored in the tree is a read-only checkpoint: after a prefix hit, copy-on-write restores it into the request's private working slot, and the forward pass mutates only that copy. Copying and forking are what keep an overwrite from destroying the prefix. Below, the first view compares how the KDA and MLA radix cache grow under two workload changes; the second steps through copy-on-write, snapshot, donate, and checkpoint eviction:

KDA prefix cache diagramKV can share one prefix in place; KDA must restore a shared checkpoint into a private working slot before mutation

Which workload dimension makes each state grow?The four plots compare growth direction only; they do not share a y-axis.

Workload changeKDA stateMLA KV cache

One request gets longer

~54MB / active request (TP=8), independent of token count

~27KB / token, linear in context length

Same cached-token total, more active branches

Every active branch needs its own mutable working state

Depends on total cached tokens; branch topology itself adds no KV

Concurrent branches4 branches

One prefix reuse follows 1 → 2 → 3 → 4. As generation continues, 3 → 4 repeats at later checkpoint boundaries.

Every branch shown hits ABC. S(ABC) in the tree is a read-only checkpoint, not a live request's working state.

S(ABC) in the tree is read-only; every active request starts from this shared checkpoint.

Memory management for two state types

Because K3 mixes KDA and MLA layers, memory management has to carve out two kinds of region as well. The old approach preallocates a KDA region and an MLA region at startup. KDA demand follows concurrency while MLA demand follows total context length, so once the workload differs from the estimate, one region fills while the other sits idle. The fix is direct: SGLang puts both in one pool, with fixed KDA blocks allocated from the left, MLA pages from the right, and the free space in the middle shared. When a request finishes, aborts or is retracted and leaves a hole in the middle, a block from the end is moved into it, so the free region stays contiguous and both ends stay packed — which is what lets a 54MB KDA block and a 27KB MLA page draw from the same bytes with no common page size forced on them. Day-0 makes this opt-in with --enable-unified-memory:

Memory pool comparisonBoth states allocate on admission; KDA then stays fixed while MLA grows with tokens

t = 0/32

Static split pools (fixed at startup)

active requests 0

KDA 0= 0 requests × 3 pagesMLA 0accumulates page by page with tokensfree 44 pagesunused capacityfailures 0

Unified pool (SGLang)

active requests 0

KDA 0= 0 requests × 3 pagesMLA 0accumulates page by page with tokensfree 44 pagesunused capacityfailures 0

In the baseline, allocations pick free slots inside two statically sized regions. When the MLA region fills, free KDA slots cannot be borrowed. The unified layout instead packs KDA from the left and MLA from the right, so every request keeps running.

Chunked pipeline prefill

Prefill processes many prompt tokens at once, which gives it enough work to cut into chunks and fill a pipeline. TP8 puts all eight GPUs on the same prefill work, but the problem is that every layer needs a lockstep AllReduce before the full result can enter the next layer. Chunked PP8 works differently: PP8 assigns ranges of the 93 layers to G1–G8 and lets different GPUs process different chunks. GPUs pass only the activation from the preceding stage, and those P2P transfers overlap with computation on the next chunk. As the diagram shows, G1–G8 each run a run of complete layers rather than splitting one layer eight ways, so no AllReduce is needed between layers. The cost is that filling and draining the pipeline takes some steps, so it cannot run full from the start; over a long-context prefill that overhead is diluted, so throughput still comes out ahead.

Chunked pipeline prefill diagramThe same 8 GPUs; TP8 above, chunked PP8 below

ProblemTP8: all 8 GPUs sync after every layer

Every layer is split eight ways. After compute, every rank must finish an AllReduce before any rank can enter the next layer.

G1computeAllReducecomputeAllReducecomputeAllReduce

G2computeAllReducecomputeAllReducecomputeAllReduce

G3computeAllReducecomputeAllReducecomputeAllReduce

G4computeAllReducecomputeAllReducecomputeAllReduce

G5computeAllReducecomputeAllReducecomputeAllReduce

G6computeAllReducecomputeAllReducecomputeAllReduce

G7computeAllReducecomputeAllReducecomputeAllReduce

G8computeAllReducecomputeAllReducecomputeAllReduce

✓ A barrier after each of 93 layers✓ Communication stays on the critical path✓ Eight-way slicing makes GEMMs narrower

SolutionChunked PP8: 8 layer stages, prompt in chunks

G1–G8 each execute a run of complete layers. The long prompt becomes multiple chunks; after one chunk, a GPU sends its activation directly to the next GPU.

Prompt chunks5 chunkspipeline-slot utilization 42%bubble share 58%Equal-stage teaching model: more chunks amortize the fill and drain bubbles.

G1L1–12C1C2C3C4C5

G2L13–24C1C2C3C4C5

G3L25–36C1C2C3C4C5

G4L37–48C1C2C3C4C5

G5L49–60C1C2C3C4C5

G6L61–72C1C2C3C4C5

G7L73–84C1C2C3C4C5

G8L85–93C1C2C3C4C5

Measured 8K prefill (2×4 GB300; topology is the only variable)

Prefill capacity per node

TEP81.00×

PP8×TP11.45–1.72× (representative point: 1.64×)

Exposed critical-path communication / 1K tokens

TP8 · 9.38 ms

PP8 · 0.88 msabout 91% lower

Decode is another story. It usually has only one new token per step, not enough work to fill a pipeline as deep as PP8, so TP8 remains the better fit. A common PD-disaggregated split is PP8 prefill → TP8 / DCP8 decode. As noted above, PP also pays fill and drain overhead, so it does not win when requests or chunks are too few.

Decode context parallelism (DCP)

MLA has multiple attention heads, but they all share one compressed KV latent, so unlike MHA there is no clear axis to shard when TP splits by head. Every TP8 rank therefore has to replicate the complete MLA KV latent. That means the same MLA KV latent sits on every GPU, and adding GPUs does not increase logical context capacity. SGLang uses DCP to shard by token position instead: each GPU computes partial attention over its local $1/N$ KV, split as the diagram shows, then one packed all-to-all merges the result exactly:

DCP diagramThe same 12-token MLA context; naive TP above, DCP below

Parallel GPUs NEach GPU scans about 1/4 of KV

ProblemNaive TP: every GPU stores the full KV

MLA has multiple attention heads, but they all share the same compressed KV latent, leaving no cache head axis for TP to shard. TP4 therefore replicates this KV latent on all 4 GPUs: more GPUs do not increase logical context capacity.

T1T2T3T4T5T6T7T8T9T10T11T12

GPU 1

GPU 2

GPU 3

GPU 4

The same 12-token contextPhysical KV cells: 48 (4× replicated)

SolutionDCP: KV striped across GPUs by token position

The small query is replicated to all GPUs; the long, memory-heavy KV is striped round-robin by token position, with each position stored once.

T1T2T3T4T5T6T7T8T9T10T11T12

GPU 1

GPU 2

GPU 3

GPU 4

Physical KV cells: 12 (one copy per position)With the same 48-cell memory: 12 → 48 logical tokens

Why one MLA decode step remains exact

① Project the full q locallyq is small; no broadcast→② Local attention per GPUscan only 1/N of KV→③ One packed all-to-alleach segment is 1/N of o→④ LSE-weighted mergematches the full softmax result

filled = stored on this GPUoutline = owned by another GPU

DCP primarily increases capacity and concurrency, not latency for one short request. Keeping more long sessions on the GPU avoids host offload, re-prefill, and throughput collapse. DCP groups are built inside the TP group, so TP8 with DCP8 is still just 8 GPUs: on K3 logical capacity goes from about 1.5M to 12.2M tokens (roughly 7.9×) and reaches 541 tok/s at 48 agent sessions. It costs one all-to-all per MLA layer. The KDA state S, meanwhile, does not grow with context and has no token-position axis, so it stays TP/head-sharded.

Performance numbers and benchmark results

BS=1 decode, aggregate system throughput, and per-user speed are three distinct metrics. Single-request speed first: 15 kernel and communication optimizations raise non-speculative BS=1 decode from 44.3 to 112.5 tok/s, and DSpark's speculative decoding is a separate line at ~423 tok/s:

BS=1 decode optimization8×GB300 · TP8 · BF16 KV cache · non-speculative

Optimization roundP15current speed 112.5 tok/svs. P0 +154%this round +1.1

Add DSpark112.5 tok/s→~423 tok/s

Cluster metrics are a different question: under PD disaggregation, aggregate throughput trades against per-user speed, and each deployment goal maps to a different parallel layout:

Deployment trade-offsGB300 · 8K input / 1K output · PD disaggregation

Deployment goal

System efficiency (tok/s/GPU)

Total throughputPP8 prefill → TP8 decode · FP42,808

Long context2×PP8 prefill → 2×DCP8 decode2,633

Per-user speed (tok/s/user)

Total throughputthroughput configuration18.7

Per useradd independent TP8 decode instances116+

throughput firstadd decode instances →per-user speed first

Throughput: 2,808 tok/s/GPU; 18.7 tok/s/user.

Acknowledgments

Kimi K3 day-0 support was a collaboration between the SGLang & Miles team at RadixArk and the Moonshot AI team, together with NVIDIA, AMD, Approaching AI, Baseten, and Modal.

AMD: Wun-guo Huang, Xinyi Song, Hai Xiao, Soga Lin, Duyi Wang, Thomas Wang
Approaching AI: Huanming Shen, Xiaohao Zhang, Nan Li, Mingxing Zhang

Thanks to DigitalOcean for providing AMD instances for testing. Thanks also to Google Cloud, DigitalOcean, Nebius, fal, RunPod, DeepInfra, and GMI Cloud for serving Kimi K3 on SGLang.

The Kimi K3 architecture

KV cache memory estimation

Attention mechanisms in detail: classic MHA

Attention mechanisms in detail: linear attention

Attention mechanisms in detail: DeltaNet

Attention mechanisms in detail: KDA

Attention Residual

LatentMoE

SGLang's day-0 adaptations

RadixAttention: prefix caching for KDA state

Memory management for two state types

Chunked pipeline prefill

Decode context parallelism (DCP)

Performance numbers and benchmark results

System efficiency (tok/s/GPU)

Per-user speed (tok/s/user)

Acknowledgments

Further reading