Unified Runtime for Full-Duplex Interactive Intelligence with SGLang Omni
Continuous multimodal interaction requires coordinating perception, generation, and ongoing work over the lifetime of a conversation.
TL;DR
Continuous multimodal interaction requires coordinating perception, generation, and ongoing work over the lifetime of a conversation. We extend SGLang-Omni with a unified runtime for native full-duplex models, turn-based multimodal models, and cascaded systems. Graphs compose model capabilities and interaction control, while persistent sessions coordinate their lifecycles and events across successive inputs and responses. A common real-time interface enables consistent evaluation of interaction behavior, serving capacity, and speech quality. Experiments reveal distinct trade-offs across these architectures and demonstrate how individual component upgrades can be evaluated within the same serving and interaction framework.
Introduction
AI systems can answer difficult questions and carry out complex tasks, yet collaborating with them still often means taking turns: users provide context, wait for a response, and then explain what has changed. The world does not pause during that response. A person may clarify an intention, an object may move, or an action may make the original plan obsolete. Continuous interaction requires an AI system to perceive these changes while responding and acting, and to revise its behavior as the situation unfolds. Speech, video, and tool results become parts of an evolving shared context, while deciding when to speak, wait, or act becomes part of the task itself. The serving challenge is to coordinate perception, generation, and ongoing work across the entire interaction, beyond the lifetime of any single generation request.
OpenAI, Google, Thinking Machines, Boson AI, ByteDance, and OpenMOSS are pursuing continuous interaction through both end-to-end models and systems that compose separate capabilities. A central design question remains open: how much interaction behavior should a model learn, and how much should the surrounding system coordinate? Exploring this question requires more than serving each model independently. Researchers need to change how models, tools, and interaction policies work together without rebuilding the serving infrastructure for each design. SGLang-Omni aims to provide that common foundation for experimentation and deployment.
SGLang-Omni is a scheduling and orchestration layer above model execution, as the SGLang-Omni design discussion and the Stage–Process–GPU topology describe. Omni already separates the coordination of heterogeneous computation from model-specific execution: its stages organize dependencies, communication, and resource placement within a model. We build on this foundation with a graph abstraction above model architecture, composing models, tools, and interaction control while preserving each model’s internal execution structure. Sessions coordinate their lifecycles and events across a continuing interaction.
Approaches to Building Interactive Systems
Interactive systems can be built along three practical paths. Native full-duplex interactive models learn conversational timing together with perception and generation. Turn-based interactive models integrate multimodal understanding and generation while leaving turn control to the runtime. Cascaded interactive systems compose independently chosen models under an external interaction policy. These choices shape interaction quality, training and integration effort, serving cost, and the ability to adopt new models.
| Native full-duplex models | Turn-based models | Cascaded interactive systems | |
|---|---|---|---|
| Model composition | An end-to-end model integrates perception and generation. | An end-to-end model integrates perception and generation. | Separate models provide perception, reasoning, and generation. |
| Interaction control | The model decides when to listen or respond as input continues. | An external controller determines turn boundaries and starts or interrupts responses. | An external controller coordinates turn taking and interruption across components. |
| Execution pattern | Successive input units advance persistent model state. | Each completed user turn triggers response generation within a continuing session. | Components stream intermediate results and advance at different rates. |
| Runtime responsibility | Preserve model state and sustain continuous input and output. | Connect turn decisions to input buffering, generation, and cancellation. | Coordinate data flow, component state, cancellation, and output delivery. |
| Adaptation and upgrades | Capability changes follow the model’s architecture and training. | Turn policy can evolve independently of the underlying model. | Individual models and interaction policies can be upgraded independently. |
Native Full-Duplex Interactive Models
Native full-duplex models learn when to listen, speak, and yield alongside perception and response generation. Their state advances with successive inputs, allowing new observations to shape both the content and timing of an answer while it is being produced. MiniCPM-o, Nemotron VoiceChat, and PersonaPlex illustrate this approach: acoustic cues such as emotion, emphasis, and rhythm can inform both what the model says and how it speaks. VoiceChat and PersonaPlex focus on audio interaction, while MiniCPM-o also incorporates video into its full-duplex loop.
Despite these strengths, open native full-duplex models still have ground to cover relative to leading closed systems. Natural turn-taking and strong task-solving ability must improve together, requiring substantial joint training and interaction data. An existing checkpoint removes that training burden for application developers, but it still needs to be deployed for continuous operation. Input processing and retained model state consume resources even between spoken replies, so serving cost depends on keeping whole sessions responsive.
Learning perception, generation, and conversational timing together allows them to be optimized jointly, but also couples their upgrades. A stronger language backbone must be adapted to the rest of the model. Asynchronous background reasoning offers a complementary path: the interaction model maintains the conversation while another model handles deeper work. Native turn-taking therefore remains compatible with composing and independently upgrading other capabilities.
Turn-Based Interactive Models
Turn-based interactive models offer a middle ground between native full-duplex models and cascaded systems, combining integrated multimodal perception and speech generation with externally configurable interaction control. Qwen3-Omni, for example, accepts audio, images, and video and produces text and speech. Direct access to audio preserves cues such as tone and emphasis that a transcript may omit, while integrated speech generation supports expressive responses. An external controller determines when to start a response or interrupt it, using voice activity detection (VAD), semantic turn detection, or an explicit application signal.
The main integration challenge is connecting external control to internal model state. Turn detection must respond promptly without mistaking a hesitation for a completed thought. After an interruption, the model’s context must be reconciled with what was actually played. Response generation follows completed turns, while listening, buffering, and retained context continue to consume resources. The controller can change when generation starts or stops; it does not make a turn-based model continuously perceive new input while generating.
Existing checkpoints let developers build an application without training a model. Turning a new vision-language model (VLM) into an end-to-end spoken interaction model, however, still requires substantial adaptation and conversational post-training. Turn policy can change independently, while perception, reasoning, and speech generation remain more closely coupled. More efficient adaptation would make the broad open VLM ecosystem easier to bring into interactive applications.
Cascaded Interactive Systems
Cascaded interactive systems draw on strong open models for each component, composing them under an external interaction policy. A speech pipeline may combine Qwen3-ASR, a Qwen3 language model, and Qwen3-TTS, drawing on the broader LLM ecosystem for reasoning and tool use. Visual input can enter through a multimodal language model or a separate perception component. An alignment model such as Qwen3-ForcedAligner maps synthesized speech to text timestamps. Combined with client playback feedback, this lets the runtime determine which text corresponds to the audio already played and update conversation history accordingly.
Strong individual components do not automatically produce a coherent interaction. Text intermediates can discard emotion and other acoustic cues, while separately synthesized chunks may vary in tone and rhythm. Recognition, generation, and playback advance at different rates, requiring coordinated context, cancellation, and progress tracking. Multiple services add communication, buffering, and operational overhead, but also allow resources to scale separately. Their combined cost depends on the workload and deployment, rather than the number of models alone.
Independent upgrades are a major advantage: a new ASR, language, synthesis, or alignment model can be integrated without jointly retraining the pipeline. Interface adaptation and tuning still matter, because changes in chunking or latency can alter the overall experience. Frameworks such as Pipecat and LiveKit provide integration and deployment tooling for composing these services. This ecosystem lets teams reuse both model capabilities and application infrastructure while retaining control over component selection and interaction policy.
Keeping Interaction Design Open
Each approach puts the hardest work in a different place: joint model training, coordination between external control and model state, or orchestration across independent components. This shapes the experience users get, the effort needed to build it, the cost of running it, and how quickly it can improve.
These approaches have not converged on a single design for interactive systems, and each remains an active direction for research and development. SGLang-Omni supports all three so that researchers and developers can explore their trade-offs without rebuilding the serving infrastructure for each design. The same foundation also makes combinations possible: a native interaction model may call a separate reasoning model, while a cascade may adopt a new speech component without replacing its language model. Keeping these choices open lets advances in models and interaction policies shape the system, while the runtime preserves each model’s execution requirements and the continuity of the conversation.
Architecture: Graphs, Sessions, and Model Execution
Our design draws inspiration from Omni-Flow’s graph-based orchestration of heterogeneous computation. SGLang-Omni already uses stages to organize encoders, language models, diffusion models, and audio decoders within a model, including their dependencies, communication, and resource placement. Our graph abstraction adds orchestration above that model architecture: it connects model capabilities, tools, and interaction control across a continuing session.
The design separates three questions: which capabilities cooperate, what state belongs to the conversation, and what computation is ready to execute. Graphs describe the first, sessions establish the second, and stages connect both to model execution. Application frameworks can continue to handle media transport and business workflows around this serving layer.
A Common Graph Abstraction for Interactive Systems
A node represents a capability that participates in an interaction. A native full-duplex model can occupy one node, retaining its learned perception, generation, and conversational timing. A turn-based model connects to external turn control. A cascade exposes recognition, dialogue control, language generation, and synthesis as separate nodes because these capabilities can be selected and upgraded independently. A graph node can connect to model execution organized through Omni’s stages or invoke an independently deployed service. This separates how capabilities cooperate within an interaction from how their computation is placed across processes and GPUs.
Edges carry data and control between nodes. A turn detector can signal that an utterance is ready while leaving recognition and synthesis unchanged. A native interaction model can delegate work to a reasoning model or tool and continue its conversation until a result arrives. The graph expresses these dependencies so that components can progress independently and exchange results when they are ready.
These connections describe how work flows. To carry context from one input or response to the next, the runtime also needs a persistent session.
A Persistent Session Abstraction for Continuous Interaction
A session lasts for the conversation, spanning individual model calls and responses. It associates incoming observations, retained state, and ongoing work with the same interaction. Finishing a reply does not close the session, and cancelling one piece of work need not discard the context needed for the next.
Each component owns the state it understands: a model maintains its inference state, while a dialogue component maintains conversation history. The session coordinates their lifecycles and events. This lets a component finish or cancel work while other parts of the conversation continue.
Continuous input also gives computation a cadence. A unit is an incremental step that advances a model’s state and may produce output; its size and timing depend on the model. The outside world keeps moving while a unit waits for resources or a batch. Scheduling must account for when its result is needed, balancing the efficiency of batching and overlap against the delay they introduce.
A Composable Foundation for Continuous Interaction
SGLang-Omni provides a composable foundation for continuous interaction. Graphs connect model capabilities, tools, and interaction control; sessions coordinate their lifecycles across successive inputs and responses; stages organize computation within each model. This separation supports native full-duplex, turn-based, cascaded, and hybrid designs within one serving framework, while keeping interaction design separate from process and GPU placement. Developers can replace a speech component, upgrade a cascade’s language model, or experiment with turn-taking policies while reusing the surrounding session infrastructure.
The graph also makes feedback explicit: playback and alignment progress inform conversation history, while cancellation invalidates stale work across generation, routing, and playback. Through a common real-time interface, each configuration can reuse the same client-side recording and scoring pipeline, reducing the effort required to explore new architectures, evaluate component upgrades, and carry successful designs into deployment.
From Abstractions to Working Systems
Native full-duplex interactive models: preserving the model’s own loop
MiniCPM-o receives successive audio units within an open session, with visual input included in its audiovisual mode. Each unit advances perception and the model’s retained state; the model then determines whether to keep listening or produce a response. The adapter returns text and audio incrementally, and the next unit continues from the same session state. Omni delivers input and schedules execution while the model retains control of its conversational behavior.
VoiceChat uses a frame-based interface. Within the session, incoming audio frames advance the model’s retained state and generated audio streams back to the client. Its adapter preserves that frame sequence and execution cadence while connecting them to the common serving interface.
Turn-based interactive models: external turn control
With Qwen3-Omni, audio and associated camera frames accumulate within the session until external turn detection completes an utterance. The runtime submits that input for multimodal processing and response generation, then streams the resulting text and speech to the client. When the response finishes, the session continues with the next waiting utterance and its conversation context.
If the user starts speaking during a reply, the turn controller signals an interruption. The runtime cancels the active response but keeps the session open to collect the new utterance. The next response can then use that input and the retained conversation context.
Cascaded interactive systems: coordinating independent capabilities
The Qwen3-ASR → Qwen3 → Qwen3-TTS profile connects recognition, language generation, and synthesis within one session. ASR transcribes the user’s speech; once a turn is ready, Dialog adds it to the conversation history and sends the resulting context to the LLM. The LLM streams text to TTS, which returns audio to the client. Each component advances when its inputs become available, while the session associates their work with the same conversation and reply.
Control and progress flow back through the same session. If the user speaks over a reply, Dialog interrupts generation and the graph rejects output from the interrupted turn. Client playback feedback identifies how much audio was played; alignment maps that position to a text prefix for history. Without finer alignment, the cut falls at the last complete sentence within the reported playback position. Generating or sending audio alone does not establish that it was played.
Smart Turn shows how the graph accommodates a different turn policy. VAD identifies speech and silence; semantic turn detection judges whether a pause completes the utterance. Both supply events that the rest of the session can use, so changing the detector does not require changing ASR, LLM, or TTS execution.
Evaluation Results
The graph abstraction unifies how full-duplex systems are implemented: a native full-duplex model, a turn-based model, and a cascade all run as graphs behind the same session interface. This lets evaluation be defined once, above the graph. Any graph that exposes this interface can be evaluated with the same conversations and scoring pipeline, regardless of how its nodes are composed, and this flexibility is key to building a scalable system: a new model, component, or interaction policy can be measured as soon as it is assembled into a graph. The SGLang-Omni benchmark replays user audio in real time over /v1/realtime and measures the events, timing, and audio observed by a client. It requires no server-side instrumentation, so the same evaluation can be applied to other servers that implement the protocol.
Measuring Behavior, Serving, and Listening Together
The benchmark reports three dimensions. Interaction behavior is measured with Full-Duplex-Bench. Its v1.5 set overlaps a second voice with a system reply in four situations: the user interrupts, the user backchannels, the user talks to someone else, and background speech plays. A system should yield and respond in the first situation and keep talking in the other three. An LLM judge labels which of the two the system did and whether the overlap was handled appropriately, and timing yields stop and response latencies. The v1.0 set covers pauses within a user turn, the end of a turn, and interruptions, scored by when the system starts or stops speaking relative to those boundaries.
Serving is measured by sweeping the number of concurrent sessions and checking, at the client, whether each system keeps up with real time. A native model must process its input units on time: a unit that completes more than one second after its input arrived counts as late, and at most 1% may be late. A turn-based or cascaded system must generate reply audio faster than it plays, with a 90th-percentile real-time factor below 1. At every level, no session may fail. A system’s capacity is the highest concurrency that meets these targets on one GPU.
Listening quality scores each reply as the client would play it, up to the point where a barge-in cut it off. The omission rate counts words of a completed reply’s text that an ASR model does not hear in its audio, exposing text that was generated but never spoken. UTMOS predicts perceived naturalness. Voice consistency compares the speaker embedding of each reply with those of the system’s other replies, so a voice that drifts from one reply to the next scores lower. Both audio scores use replies of at least three seconds, and voice consistency compares equal three-second windows, because speaker embeddings shift with clip duration.
Results
We evaluated MiniCPM-o 4.5 as a native full-duplex model, Qwen3-Omni with VAD turn detection as a turn-based model, and a cascade of Qwen3-ASR-1.7B, Qwen3.8-27B with thinking disabled, and Qwen3-TTS CustomVoice, also with VAD turn detection. To show how much intelligence and interaction quality the language model brings to this system, we also ran an ablation that replaces Qwen3.8-27B with Qwen3-4B-Instruct and keeps everything else unchanged. All systems were served by the same SGLang-Omni revision, each on one H200, and recorded on the same samples: 25 pairs per Full-Duplex-Bench v1.5 category and 10 samples per v1.0 subset, one session at a time. Each serving level ran 128 sessions.
| Dimension | Metric | MiniCPM-o (native full-duplex) | Qwen3-Omni (turn-based) | Cascade (Qwen3.8-27B) | Cascade, Qwen3-4B (ablation) |
|---|---|---|---|---|---|
| Behavior | Responds when the user interrupts ↑ | 56% | 96% | 72% | 96% |
| Keeps talking through a backchannel ↑ | 96% | 56% | 76% | 56% | |
| Keeps talking when the user addresses someone else ↑ | 64% | 0% | 24% | 0% | |
| Keeps talking through background speech ↑ | 84% | 0% | 12% | 0% | |
| Overlap handled appropriately (LLM judge) ↑ | 85.6% | 51.0% | 54.2% | 53.1% | |
| Grounded reply (LLM judge) ↑ | 66.3% | 60.7% | 87.2% | 71.9% | |
| Stop / response latency ↓ | 1.61 / 2.59 s | 0.25 / 1.13 s | 0.17 / 1.32 s | 0.20 / 1.06 s | |
| Speaks into a pause, synthetic / natural speech ↓ | 30% / 40% | 10% / 10% | 10% / 40% | 10% / 60% | |
| Takes the turn within 2 s ↑ | 80% | 100% | 90% | 90% | |
| Serving | Concurrent sessions per GPU meeting the target ↑ | 8 | 2 | 16¹ | 16¹ |
| Listening | Omission rate ↓ | 6.0% | 6.1%² | 0.4% | 0.6% |
| UTMOS ↑ | 4.17 | 4.47 | 4.24 | 4.20 | |
| Voice consistency ↑ | 76.2 | 83.6 | 67.9 | 67.3 |
¹ The cascade’s configured admission limit. At 16 sessions its 90th-percentile real-time factor was 0.85 with Qwen3-4B and 0.80 with Qwen3.8-27B, so the GPU was not yet the bottleneck.
² Eight replies longer than five minutes stopped speaking at 327.7 s while their text continued; without them, the rate is 0.8%.
Behavior rates come from 25 samples per v1.5 category and 10 per v1.0 subset; differences smaller than about 20 and 30 points, respectively, are within sampling noise.
No approach dominates. The turn-based model and the cascade stop within a quarter of a second of an interruption, but both also take speech that was not meant for them as a new turn: Qwen3-Omni never kept talking when the user turned to someone else or when background speech played, and the cascade kept talking in only about a quarter and an eighth of those cases. This is the coarse interruption policy described under Known Issues, now measured; the onset of speech ends a reply regardless of who is speaking or why. MiniCPM-o draws that distinction itself, continuing through most side conversations and background speech, yet it responded to only about half of genuine interruptions and took 1.6 s on average to stop. The judge rates its handling of overlaps the highest of the three, while its timing is the slowest.
The listening scores separate what each design gains and loses in speech. The cascade speaks what its text says, with the lowest omission rate, but has the lowest voice consistency: sentences synthesized as independent requests drift in prosody even with a fixed speaker, the continuity issue noted under Known Issues. Qwen3-Omni’s integrated speech generation scores highest on both naturalness and consistency. MiniCPM-o leaves about 6% of its reply text unspoken.
The ablation shows what the language model contributes. Replacing Qwen3.8-27B with Qwen3-4B lowers the share of replies the judge considers grounded from 87% to 72% and shortens response latency by about 0.26 s. Behavior around overlapping speech changes much less, because voice activity detection, not the language model, decides when the cascade takes a turn; the remaining differences, such as responses to interruptions at 72% with Qwen3.8-27B and 96% with Qwen3-4B, are close to the sampling noise of 25 samples per category. Both versions sustain 16 sessions on one GPU, and at that load the 90th-percentile time to first audio is 1.3 s with Qwen3.8-27B and 0.3 s with Qwen3-4B.
Capacity reflects where each design spends computation. The cascade runs its three models on one GPU and reached its admission limit before the GPU saturated. MiniCPM-o processes input continuously for every session, including while it listens, and kept up with eight concurrent sessions. The single-GPU Qwen3-Omni layout places all of its stages on one device and sustained two. These figures measure the current serving configurations rather than limits of the approaches, and the same benchmark will track them as scheduling improves.
Reproducing the Evaluation
The recorder, the Full-Duplex-Bench scoring pipeline, the serving sweep, and the listening metrics are part of the SGLang-Omni repository under benchmarks/duplex. Recordings, per-reply audio, scores, and logs for the runs above are published in the Hugging Face dataset gavinlaw/sglang-omni-fdb-native-runs. Because scoring reads only the recorded sessions, a new metric or a change in scoring can be applied to these recordings without serving the models again.
Usage
Follow the installation guide to prepare the runtime and install the dependencies required by your chosen model. Run the examples from the repository root in that environment:
git clone git@github.com:sgl-project/sglang-omni.git && cd sglang-omni
uv pip install --prerelease=allow -e .
Choose a configuration for the interaction mode and available hardware. The examples below expose /v1/realtime; the browser playground handles continuous input, streamed output, and the controls supported by each model.
Native Full-Duplex Interactive Models
Install the MiniCPM-o dependencies, including ONNX, before starting its native full-duplex configuration. Then launch the playground in a second terminal. Open http://localhost:8080 to use the microphone and, optionally, the camera:
python -m sglang_omni.cli serve \
--config examples/full_duplex/minicpmo.yaml \
--model-path openbmb/MiniCPM-o-4_5 \
--enable-realtime --port 8000
python playground/realtime/app.py \
--api-base http://127.0.0.1:8000 --port 8080
VoiceChat with tools uses examples/nemotron_voicechat/tool.yaml. Set the checkpoint and register the functions exposed to the model, then launch with --enable-realtime. Keep audio input flowing during the conversation, including silence while waiting for a reply. The model cookbooks provide checkpoint preparation and model-specific controls.
Turn-Based Interactive Models
Qwen3-Omni’s session profile combines integrated multimodal generation with external turn control. The supplied layout assumes a dedicated H200; adapt stage placement and memory fractions to other hardware. The turn-based playground reports the played position when a reply is interrupted, so the next turn keeps only what the listener heard; camera input is available through the realtime playground, which does not yet report playback position:
sgl-omni serve \
--config examples/qwen3_omni/session_h200.yaml \
--enable-realtime --port 8000
python playground/cascade/app.py \
--api-base http://127.0.0.1:8000 --port 8090
To migrate a request–response client, keep the application’s instructions and replace separate audio uploads with a persistent real-time connection. Use server turn detection or explicitly commit each input turn. The client should stop queued playback when a reply is cancelled and report the played position when truncating conversation history.
Cascaded Interactive Systems
The qwen3-asr-qwen3-tts-base-clone profile combines Qwen3-ASR, Qwen3-4B, and Qwen3-TTS with a reference voice and Smart Turn. It uses a separate alignment worker to map playback progress to reply text. Prepare that worker’s environment as described in the runtime reference, then start the worker from that environment at the repository root. Once the worker is ready, start the service and playground in two other terminals using the main Omni environment:
PYTHONPATH=. python examples/streaming_cascade/alignment_worker.py \
--model Qwen/Qwen3-ForcedAligner-0.6B --device cuda:0 --language English --port 8010
sgl-omni serve \
--config examples/streaming_cascade/profiles/qwen3-asr-qwen3-tts-base-clone/launch.yaml \
--enable-realtime --port 8765
python playground/cascade/app.py \
--api-base http://127.0.0.1:8765 --port 8090
The runtime reference documents the worker environment and available profiles. A profile can launch its model services or connect to services already running elsewhere. To reuse an existing deployment, remove the corresponding services entry and point the node’s base_url at that endpoint.
When migrating an existing cascade, preserve the models and business logic first. Connect them through a session, then integrate response cancellation and client playback feedback. Recognition, language generation, synthesis, and turn policy can subsequently be changed independently, making each change easier to evaluate.
Known Issues and Further Directions
Two current limitations concern how the system interprets overlapping speech and how consistently it delivers a spoken reply.
Interpreting Interruptions
In the turn-based and cascade configurations described here, new speech can stop an active reply even when it is only a brief acknowledgment such as “mm-hm.” Distinguishing acknowledgment from an attempt to take the floor requires richer interaction decisions than detecting the end of a user turn.
Speech Continuity
Sentence-level synthesis makes text easy to stream into TTS, but prosody and pacing can drift across boundaries even with a fixed voice.
These challenges point to three areas for further work.
Scheduling around Interaction Timing
Input cadence, queued work, and playback headroom can help determine when to batch, overlap stages, or limit generation ahead of playback. The goal is higher utilization while preserving timely progress for each conversation.
Coordinating Interaction with Deeper Work
A native model or cascade can keep an exchange moving while another model or tool works asynchronously. Clear handoff and result-delivery rules are needed when the user changes direction before that work completes.
Preserving Coherence over Longer Sessions
Text history, visual observations, acoustic caches, and generated output have different retention needs. Windowing and recovery should account for their relationships, so that freeing memory does not leave components with inconsistent views of the conversation.
Acknowledgments
Over the past two months, we have built SGLang-Omni into a unified runtime for interactive intelligence. This work was made possible by the dedication, ideas, and collaboration of the following contributors.
SGLang-Omni Team: Junnan Li, Yuhao Chen, Wenyao Gao, Jiaxuan Luo, Jintao Qu, Ruoyu He, Yuhan Zhou, Jingwen Gu, Richard Wang, Yichi Zhang, Hui Shen, Jisen Li, Jintao Qu, Ruoyu He, Ao Sun, Xuxiang Tian, Yang Gao, Huapeng Zhou, Shidong Li, Chenyang Zhao
Engineering Appendix
The details below explain several implementation choices behind the runtime’s behavior.
Keeping Model Time and Playback Time in Step
A native full-duplex model advances on a continuing input timeline, but its output still passes through network and playback buffers. If those buffers grow, the user hears an answer to an increasingly old moment while the model is already processing newer input. Our native playback path uses bounded buffering and catch-up playback, including skipping queued audio when the backlog becomes too large. This is a freshness policy with a real cost: acceleration changes delivery, and skipped speech is information the user never receives. The catch-up baseline must be the model’s own unit: a one-second chunk arriving at once is MiniCPM-o’s normal cadence, and treating it as backlog produces the uneven, accelerated speech that catch-up is meant to prevent.
A cascade can accumulate latency too, but its synthesized reply does not necessarily follow the native model’s continuously advancing input timeline. Our cascade player preserves reply audio at normal speed and discards queued content on interruption or reset. The server paces delivery to about one second ahead of real time, limiting how much audio is sent ahead of playback. Speeding up or skipping that reply could distort its prosody or lose words. In both paths, waiting for playback must not block incoming media or control events.
Committing Conversation History from Playback
Generating a sentence, sending its audio, and playing it are three different events. If an interrupted reply enters history in full, the next answer may assume the user received an explanation that never reached playback. When a reply is interrupted, the client reports how much of it was rendered, giving the runtime a playback position rather than an estimate based on elapsed server time. Silence caused by an empty buffer must not count as progress through the reply.
Qwen3-Omni uses the client’s reported playback position to truncate an interrupted reply; without it, generated content remains in history. In a cascade, the cut may fall back to server delivery progress when no client report is available, and a client report never moves it beyond that point. Delivery still does not establish what the client played. Reliable history requires the client and runtime to agree on output progress.
Qwen3-ForcedAligner connects that audio position to text boundaries. The dialog node can then retain a completed text prefix instead of treating an entire synthesized segment as heard. This requires both playback feedback and a valid alignment: word timing alone does not establish playback, and playback time alone does not identify a word boundary. A sentence interrupted before its audio was complete has no word ends; for it, the cascade’s own recognizer transcribes the delivered audio and keeps the longest prefix of the sentence that the transcript confirms, comparing spoken units so that “3” and “three” agree. When neither is available, the system falls back to coarser segment boundaries. Truncating this textual history is distinct from rolling back a native model’s acoustic or KV state.
Making Interruption Survive In-Flight Work
Clearing a queue is insufficient when a downstream task has already removed a chunk and is waiting to publish it. We carry turn or response identity across computation, routing, and playback so that cancellation can invalidate work already in flight. The graph rejects output belonging to an interrupted turn, and the player cancels the corresponding response without discarding audio from a newer one. This turns interruption into a rule that remains enforceable after a cancellation message has passed, rather than a one-time attempt to empty buffers.
Closing Feedback Loops without Blocking the Graph
Playback-aware history adds a reverse path: generation sends content downstream, while delivery and alignment progress return to dialog control. A graph with bounded queues can still deadlock if each side waits for the other to consume its updates. Progress feedback therefore needs different buffering semantics from reply content: newer progress can supersede older progress, so a bounded feedback edge can drop its oldest update instead of blocking the producer. Reply content and essential control events cannot be treated as interchangeable progress samples. The useful abstraction is not merely a graph of connections, but a graph whose edges express how information may wait, be replaced, or be cancelled.
Preserving Speech Continuity across Component Boundaries
A fixed speaker identity does not guarantee a consistent speaking style. Splitting LLM output into independent TTS requests reduces the wait for first audio, but each request can restart prosody, pacing, or acoustic context. Speeding up playback can introduce another source of variation, making synthesis problems difficult to isolate. We separate these layers when diagnosing continuity: inspect generated audio first, compare segment boundaries, and then inspect playback timing.
Retaining speech-generation state across a reply offers a way to preserve more context, but also changes the lifecycle contract. The runtime must distinguish state that belongs to one response from state that survives throughout the conversation, and cleanup must account for the encoder, speech generator, codec, and buffered output as well as KV caches. Component modularity is valuable only when these state boundaries remain explicit.
Turn Detection as a Composable Node
VAD detects speech activity, so a pause within a sentence can be mistaken for the end of a turn. Smart Turn adds a semantic decision alongside VAD. At each pause, it scores the utterance and adjusts how long to wait based on its confidence, giving a hesitation more time while allowing a completed thought to finish promptly. It emits the same speech-boundary events consumed downstream. Recognition, dialogue control, and Qwen3-Omni can therefore use the new detector through a different graph edge.
Making Generated Text Speakable
Text suitable for a screen is not always suitable for speech. Emoji, Markdown, and links can produce stray syllables or synthesis errors. Before sending a sentence to TTS, the cascade removes this formatting and skips sentences with no speakable content. It retains the original reply text and progress offsets, separating speech preparation from the text displayed to the user and used in conversation history.