• NVIDIA Triton Python backend fronting a vLLM-served speech LLM, taken from prototype to production.
• Derives confidence, sentiment and spoken-language ID from the LLM’s own token-level outputs: sentiment and language ID are read straight off the softmax, confidence by an ONNX model over those outputs. A forced-alignment model runs alongside on the delivered transcript.
• Orchestrates per-signal LLM queries over gRPC alongside ONNX Runtime inference for confidence calibration and forced alignment.
• p50 latency of approximately 5 to 10 ms per signal.
• Technical lead for a three-engineer team. A later concurrency refactor took sustained throughput from roughly 8 to 80 requests per second.

A voice assistant needs more from its speech model than a transcript. Downstream systems want to know how confident the recognizer is, what language was spoken, and what tone the request carried, and they want those signals cheaply enough to run on every utterance. The classical ASR engine produced them as by-products of decoding. When the speech stack moved to an LLM, they had to come from somewhere else.

Reading labels off the model

The approach that came out of our applied science partners was to ask the LLM itself. After the transcript is generated, the backend teacher-forces a task token into the sequence and reads the answer directly off the softmax over the next token: the probability mass on a small set of label tokens gives a sentiment class or a language ID in a single forward step, with no new model to train for those signals. Confidence takes one more step: a small ONNX model consumes the LLM’s token-level outputs and produces the calibrated score. Forced alignment is the one signal that does not read the LLM’s internals; its model runs alongside, on the transcript the LLM delivers. Both ONNX models are small enough to sit in the same backend as the LLM queries.

From prototype to service

My team’s job was to turn that idea into a service. The result is a two-container topology. The first container is a Triton Python backend that owns request handling, orchestrates the per-signal queries over gRPC, and runs ONNX Runtime models for confidence calibration and forced alignment. It sits in front of a second Triton container that serves the LLM through vLLM and exposes the token-level logits the first one needs. I led the design and implementation of the first container and its production rollout, and authored the AWS CDK infrastructure it runs on: ECS deployments, IAM, S3 and Route 53 DNS firewall protections.

In production the backend returns each signal at a p50 of approximately 5 to 10 ms. Getting it to sustain production load on the same hardware is its own story, told in Triton Backend Throughput Refactor. The same model also picked up a retrieval-augmented personalization layer, described in RAG Personalization for Speech Recognition.