• NVIDIA Triton Python backend fronting a vLLM-served speech LLM, taken from prototype to production.
• Derives confidence, sentiment and spoken-language ID from the model’s own token-level outputs. No additional models to train or serve.
• Orchestrates per-signal LLM queries over gRPC alongside ONNX Runtime inference for confidence calibration and forced alignment.
• p50 latency of approximately 5 to 10 ms per signal.
• Technical lead for a three-engineer team. A later concurrency refactor took sustained throughput from roughly 8 to 80 requests per second.

A voice assistant needs more from its speech model than a transcript. Downstream systems want to know how confident the recognizer is, what language was spoken, and what tone the request carried, and they want those signals cheaply enough to run on every utterance. The classical ASR engine produced them as by-products of decoding. When the speech stack moved to an LLM, they had to come from somewhere else.

Reading labels off the model

The approach that came out of our applied science partners was to ask the LLM itself. After the transcript is generated, the backend teacher-forces a task token into the sequence and reads the answer directly off the softmax over the next token: the probability mass on a small set of label tokens gives a calibrated confidence score, a sentiment class or a language ID in a single forward step. Nothing new is trained and nothing new is deployed alongside the LLM, which matters a great deal when the LLM is already the largest thing in the fleet.

From prototype to service

My team’s job was to turn that idea into a service. The result is a two-container topology. The first container is a Triton Python backend that owns request handling, orchestrates the per-signal queries over gRPC, and runs ONNX Runtime models for confidence calibration and forced alignment. It sits in front of a second Triton container that serves the LLM through vLLM and exposes the token-level logits the first one needs. I led the design and implementation of the first container and its production rollout, and authored the AWS CDK infrastructure it runs on: ECS deployments, IAM, S3 and Route 53 DNS firewall protections.

In production the backend returns each signal at a p50 of approximately 5 to 10 ms. Getting it to sustain production load on the same hardware is its own story, told in Triton Backend Throughput Refactor. The same model also picked up a retrieval-augmented personalization layer, described in RAG Personalization for Speech Recognition.