Projects with the tag vLLM
- NVIDIA Triton Python backend fronting a vLLM-served speech LLM, taken from prototype to production.
- Derives confidence, sentiment and spoken-language ID from the LLM’s own token-level outputs: sentiment and language ID are read straight off the softmax, confidence by an ONNX model over those outputs. A forced-alignment model runs alongside on the delivered transcript.
- Orchestrates per-signal LLM queries over gRPC alongside ONNX Runtime inference for confidence calibration and forced alignment.
- p50 latency of approximately 5 to 10 ms per signal.
- Technical lead for a three-engineer team. A later concurrency refactor took sustained throughput from roughly 8 to 80 requests per second.