SGLang and Miles Add Day-0 Support for Inkling 2 Small

Decode throughput and per-user speed at batch sizes 1 to 192Inkling 2 Small · GB300 and VR200, 4 GPUs each · random 8K in / 1K out · DSpark + MegaMoE · TP4 / EP4

Selected: GB300

1batch size

441tok/s per user

110tok/s per GPU

Decode throughput, tok/s per GPU

Overview

Thinking Machines Lab has released the open weights for Inkling 2 Small, a mixture-of-experts model with 594B total parameters and 17B active parameters. Built for agentic coding, visual reasoning and knowledge work, it accepts text and images and supports a 1M-token context window. SGLang and Miles provide day-0 support for serving and reinforcement learning with the released weights.

Model weights · MXFP8 weights · Model card · Thinking Machines Lab announcement

What's New vs. Inkling-Small

Inkling 2 Small adds Block Attention Residuals while retaining hybrid local and global attention and a 1M-token context window. The table compares it with the previous generation's Inkling-Small.

	Inkling-Small	Inkling 2 Small
Parameters	276B total12B active	594B total17B active
Context window	1M tokens	1M tokens
Modalities	Text, image and audio inputText output	Text and image inputText output
Attention	Hybrid local and global attention	Hybrid local and global attentionBlock Attention Residuals

SGLang Day-0 Support

SGLang supports Inkling 2 Small from day 0 on NVIDIA, AMD and Google hardware. It serves both released checkpoints, the mixed NVFP4 / MXFP8 checkpoint and the MXFP8 checkpoint, with DSpark speculative decoding, prefix caching, HiCache, a unified memory pool, image input and native TML rendering.

Features

Feature	Status	Coverage
API	Yes	Chat Completions, Responses and Anthropic Messages. Native TML rendering, tool calling and streaming
Vision	Yes	Image input, including vision with tool use
Quantization	Yes	Mixed NVFP4 / MXFP8 checkpoint and MXFP8 checkpoint
Parallelism	Yes	Tensor and expert parallelism
Speculative Decoding	Yes	DSpark with the bundled MTP head
Unified Radix Cache	Yes	Prefix caching for hybrid attention
HiCache	Yes	L2 host memory and L3 Mooncake storage, compatible with speculative decoding
Unified Memory	Yes	Unified memory pool for hybrid attention
MXFP8 KV Cache	Yes	Block-scaled MXFP8 KV cache on Blackwell

Performance

We measured SGLang decode throughput for Inkling 2 Small on four GPUs per platform, using tensor and expert parallelism (TP4 / EP4) with random 8K-token inputs and 1K-token outputs. At batch 192, VR200 reaches 5,895 decode tokens/s per GPU at 123 tokens/s per user, and GB300 reaches 3,642 decode tokens/s per GPU at 76 tokens/s per user. The chart above shows both platforms across batch sizes 1 to 192. These are decode-only measurements with DSpark acceptance simulated at 3.5 tokens per step.

These results build on DSpark speculative decoding and the following kernel work:

MegaMoE: we extended DeepGEMM’s persistent MoE kernel to Inkling 2 Small, so one kernel per layer covers both NVFP4 and MXFP8 experts in the model’s weight layout. Compared with the grouped-GEMM path, it cuts time to first token (TTFT) by 20–27% and DSpark decode time per token by 17% at batch 64.
Decode attention: a dedicated relative-bias decode kernel replaces FA4. It splits the KV cache across up to 64 CTAs, so even a single request fills the GPU, and reads a compact distance-indexed bias table. Attention drops from 25 to 9 µs per layer at batch 1.
Kernel fusion: everything before attention is one fused prologue kernel, and the router’s sigmoid, bias and top-k are another. Quantization is folded into the expert GEMMs. Nearly every launch uses programmatic dependent launch (PDL).

Hardware

Deployment recipes cover NVIDIA Blackwell (B300, GB300, B200, GB200), NVIDIA Vera Rubin (VR200), NVIDIA Hopper (H200), AMD Instinct (MI350X, MI355X) and Google TPU (TPU v7x), with these Docker images:

NVIDIA Blackwell and Hopper: lmsysorg/sglang:dev-inkling-2-small
NVIDIA Vera Rubin: lmsysorg/sglang:dev-inkling-2-small-cu134
AMD Instinct: lmsysorg/sglang:dev-inkling-2-small-mi35x
Google TPU: lmsysorg/sglang:dev-inkling-2-small-tpu

Launch recipes are in the SGLang Cookbook. For questions, join the SGLang Slack.

Miles Day-0 Support

Miles combines a Megatron trainer with SGLang rollouts for Inkling 2 Small. It supports full-parameter and LoRA training on text and image prompts, with colocated rollout engines. The rollout engines serve the released NVFP4 / MXFP8 experts.

Area	Status	Coverage
Training	Yes	Full-parameter and LoRA on MegatronLoRA on attention, routed experts and the output projection
Rollout	Yes	SGLang, colocated with training
Parallelism	Yes	Trainer EP8, with TP4 / EP4 or TP8 / EP8 rollout engines
Quantization	Yes	BF16 trainer, while rollout serves the released NVFP4 / MXFP8 experts
TITO	Yes	Token-in/token-out multi-turn sessions over OpenAI and Anthropic APIs, tool calls included
Vision	Yes	Image prompts through the frozen vision tower
Validated Run	Yes	DAPO-Math-17k LoRA, 8× GB30093 GRPO steps

We validated LoRA on DAPO-Math-17k on 8× GB300 (two 4-GPU nodes, with trainer EP8 and one TP4 / EP4 rollout engine per node). The run used 32 prompts × 8 samples per step, a 4,096-token response cap and rank-32 adapters on attention, routed experts and the output projection. The run completed 93 GRPO steps. Over the first 85 steps (charted below), the smoothed reward rose from 0.72 to about 0.85, and train–rollout KL divergence remained around 1.1–1.4 × 10⁻³.

Raw and smoothed rollout reward over the first 85 GRPO steps; the smoothed trend rises from 0.72 to about 0.85 — Inkling 2 Small LoRA on DAPO-Math-17k, 8× GB300, steps 0–84. Rollout reward is shown first, followed by train–rollout KL divergence. In the reward plot, the light line shows raw values and the dark line shows the smoothed trend.

Train–rollout KL divergence over the first 85 GRPO steps, remaining around 1.1e-3 to 1.4e-3 — Inkling 2 Small LoRA on DAPO-Math-17k, 8× GB300, steps 0–84. Rollout reward is shown first, followed by train–rollout KL divergence. In the reward plot, the light line shows raw values and the dark line shows the smoothed trend.

We also validated multimodal LoRA training on Geometry3K image prompts with 1,024-token responses. Across 15 GRPO steps, raw rollout reward rose from 0.50 to 0.94, while train–rollout KL divergence remained around 1.4–2.1 × 10⁻³.

Multimodal RL rollout raw reward over 15 GRPO steps, rising from 0.50 to 0.94 — Inkling 2 Small LoRA on Geometry3K, steps 0–14. Rollout reward is shown first, followed by train–rollout KL divergence.

Multimodal RL train–rollout KL over 15 GRPO steps, between 1.4e-3 and 2.1e-3 — Inkling 2 Small LoRA on Geometry3K, steps 0–14. Rollout reward is shown first, followed by train–rollout KL divergence.

The recipe, conversion and launch commands are in the Miles docs.

Acknowledgments

Thanks to the Thinking Machines Lab team and the NVIDIA team for collaborating with the SGLang and Miles teams on day-0 support for Inkling 2 Small.