Projects with the tag tech-lead
- NVIDIA Triton Python backend fronting a vLLM-served speech LLM, taken from prototype to production.
- Derives confidence, sentiment and spoken-language ID from the model’s own token-level outputs. No additional models to train or serve.
- Orchestrates per-signal LLM queries over gRPC alongside ONNX Runtime inference for confidence calibration and forced alignment.
- p50 latency of approximately 5 to 10 ms per signal.
- Technical lead for a three-engineer team. A later concurrency refactor took sustained throughput from roughly 8 to 80 requests per second.
- Retrieval-augmented personalization for the same speech LLM behind the paralinguistic backend.
- A user’s contacts, device names and music library are retrieved from their catalogs with approximate-nearest-neighbor search.
- Retrieved entries are injected into the LLM’s prefill context, so the model transcribes the user’s own vocabulary instead of guessing at it.
- The original design needed three round trips through the LLM per request. Working with the applied science team, we replaced LLM-generated query embeddings with an index keyed on voice features, eliminating one of them, a roughly 20 ms LLM round trip per request.
- Keying retrieval on voice features also decoupled the lookup from the LLM’s context request, so retrieval runs speculatively, in parallel with the model, instead of waiting on it.
- Led the effort end to end, from retrieval design to serving.
- Load testing showed latency growing linearly with concurrency: the server was serializing, capping throughput at about 8 requests per second.
- Three structural fixes: parallelize independent ONNX inference calls, replace a reference-counted five-thread dispatcher with single-threaded cooperative multitasking, and run several independent backend instances.
- Latency curve went from linear to roughly square-root in concurrency; sustained throughput reached 64 to 80 requests per second on the same hardware, an 8 to 10x improvement.
- Single-request latency barely moved (about 90 to 70 ms). This was a contention fix, not a per-request speedup.
- A new L3Harris radio needed far more output dynamic range than the fielded RF front end it had to work with was designed to give.
- The front end’s transmit chain had two gain stages: a coarse driver with three setpoints (low, medium, high) and a downstream fine stage adjustable in dB. The radio commanded a single power level. I designed and implemented the control software that turns that level into a safe setpoint pair for both stages.
- Reaching the required range meant disabling the equipment’s built-in protection interlocks, so the software became the only thing standing between a command and hardware damage. Both stages maxed, or either past its limit, could destroy the amplifier and drop a pilot’s radio link.
- Acted as de-facto software lead: direction for a small team, and project status reporting to organization management.