Tech-Lead on jdcsen

Paralinguistic Signals from a Speech LLM

Thu, 01 Jan 1970 00:33:45 +0000

NVIDIA Triton Python backend fronting a vLLM-served speech LLM, taken from prototype to production.
Derives confidence, sentiment and spoken-language ID from the model’s own token-level outputs. No additional models to train or serve.
Orchestrates per-signal LLM queries over gRPC alongside ONNX Runtime inference for confidence calibration and forced alignment.
p50 latency of approximately 5 to 10 ms per signal.
Technical lead for a three-engineer team. A later concurrency refactor took sustained throughput from roughly 8 to 80 requests per second.

Thu, 01 Jan 1970 00:33:45 +0000

Retrieval-augmented personalization for the same speech LLM behind the paralinguistic backend.
A user’s contacts, device names and music library are retrieved from their catalogs with approximate-nearest-neighbor search.
Retrieved entries are injected into the LLM’s prefill context, so the model transcribes the user’s own vocabulary instead of guessing at it.
The original design needed three round trips through the LLM per request. Working with the applied science team, we replaced LLM-generated query embeddings with an index keyed on voice features, eliminating one of them, a roughly 20 ms LLM round trip per request.
Keying retrieval on voice features also decoupled the lookup from the LLM’s context request, so retrieval runs speculatively, in parallel with the model, instead of waiting on it.
Led the effort end to end, from retrieval design to serving.

Thu, 01 Jan 1970 00:33:45 +0000

Load testing showed latency growing linearly with concurrency: the server was serializing, capping throughput at about 8 requests per second.
Three structural fixes: parallelize independent ONNX inference calls, replace a reference-counted five-thread dispatcher with single-threaded cooperative multitasking, and run several independent backend instances.
Latency curve went from linear to roughly square-root in concurrency; sustained throughput reached 64 to 80 requests per second on the same hardware, an 8 to 10x improvement.
Single-request latency barely moved (about 90 to 70 ms). This was a contention fix, not a per-request speedup.

Thu, 01 Jan 1970 00:33:39 +0000

A new L3Harris radio needed far more output dynamic range than the fielded RF front end it had to work with was designed to give.
The front end’s transmit chain had two gain stages: a coarse driver with three setpoints (low, medium, high) and a downstream fine stage adjustable in dB. The radio commanded a single power level. I designed and implemented the control software that turns that level into a safe setpoint pair for both stages.
Reaching the required range meant disabling the equipment’s built-in protection interlocks, so the software became the only thing standing between a command and hardware damage. Both stages maxed, or either past its limit, could destroy the amplifier and drop a pilot’s radio link.
Acted as de-facto software lead: direction for a small team, and project status reporting to organization management.