Projects with the tag Triton

Paralinguistic Signals from a Speech LLM

python LLM-inference Triton vLLM ONNX gRPC tech-lead

• NVIDIA Triton Python backend fronting a vLLM-served speech LLM, taken from prototype to production.
• Derives confidence, sentiment and spoken-language ID from the model’s own token-level outputs. No additional models to train or serve.
• Orchestrates per-signal LLM queries over gRPC alongside ONNX Runtime inference for confidence calibration and forced alignment.
• p50 latency of approximately 5 to 10 ms per signal.
• Technical lead for a three-engineer team. A later concurrency refactor took sustained throughput from roughly 8 to 80 requests per second.

Triton Backend Throughput Refactor

python LLM-inference Triton ONNX performance tech-lead

• Load testing showed latency growing linearly with concurrency: the server was serializing, capping throughput at about 8 requests per second.
• Three structural fixes: parallelize independent ONNX inference calls, replace a reference-counted five-thread dispatcher with single-threaded cooperative multitasking, and run several independent backend instances.
• Latency curve went from linear to roughly square-root in concurrency; sustained throughput reached 64 to 80 requests per second on the same hardware, an 8 to 10x improvement.
• Single-request latency barely moved (about 90 to 70 ms). This was a contention fix, not a per-request speedup.