Paralinguistic Signals from a Speech LLM

Thu, 01 Jan 1970 00:33:45 +0000

NVIDIA Triton Python backend fronting a vLLM-served speech LLM, taken from prototype to production.
Derives confidence, sentiment and spoken-language ID from the model’s own token-level outputs. No additional models to train or serve.
Orchestrates per-signal LLM queries over gRPC alongside ONNX Runtime inference for confidence calibration and forced alignment.
p50 latency of approximately 5 to 10 ms per signal.
Technical lead for a three-engineer team. A later concurrency refactor took sustained throughput from roughly 8 to 80 requests per second.

gRPC Bidirectional Streaming for a C++ Node-Graph Framework

Thu, 01 Jan 1970 00:33:44 +0000

Nova Sonic needed the speech framework’s pipelines delivered as a containerized gRPC service: audio and system prompts streaming in, inference requests streaming out. The team’s code had only ever been called through JNI.
Mapped gRPC C++’s bidirectional-streaming reactor onto the framework’s pipe abstraction once, as a reusable layer covering session setup and teardown, signal handling, graceful error handling and logging.
A new service needs under 300 lines of integration code: which pipe input receives request messages, and which pipe outputs become response messages. Adopted org-wide as the standard way to deploy a pipeline.
Established Protobuf and gRPC generated code as first-class CMake libraries in the framework’s build, so the same generated types are consumed by the pipeline’s nodes and by the server without duplicate-symbol conflicts.