- One OpenAI- and Anthropic-compatible endpoint fronting 36 model keys on a single RTX 5090 (32 GB): 14 LLM keys (Qwen3 coder, thinking and instruct tiers, a vision model, a captioner), 4 Whisper variants, 11 image generators (Flux, SDXL, Chroma, Qwen-Image), 4 Wan video models and 3 GPU feature-extraction sidecars.
- Built on llama-swap, a Go router that starts and stops upstream inference processes on demand. Anything that speaks HTTP can be an upstream, which is what lets llama.cpp, whisper.cpp, a patched stable-diffusion.cpp and three PyTorch services share one card behind one API.
- Co-residency is declared with set-algebra rules and eviction costs, but llama-swap does not measure VRAM, so I did: a sweep script that measures resident and peak footprints per model and per combination, which turned up a 6.7 GB transient VAE-decode spike as the binding constraint.
- Every workhorse LLM has two keys: an exclusive full-context key and a co-resident “lite” twin, so a 256k-context 30B model and an image generator never fight for the card.
- Heavy upstreams run as sibling containers launched on demand, so the router image rebuilds in seconds instead of recompiling sd-server and three multi-gigabyte venvs.
- Fork of stable-diffusion.cpp adding per-request reference-image identity conditioning (PhotoMaker v2 on SDXL bases, PuLID on Flux) to the
sd-server HTTP surface. Upstream registered the flags but only the CLI ever populated them. - Eleven commits, about 1,900 lines added over 19 files. Roughly 89% lives in
examples/server/; the core engine changes total 87 lines. sd-cli and the core library stay Python-free. - Reference-image encoding runs in-process through an embedded CPython interpreter (pybind11), behind two CMake flags that default to OFF so the vanilla build is unchanged.
- Identity embeddings can be extracted once and re-injected: the round trip reproduces the image-path generation byte for byte at a fixed seed.
- “No identity images” is proven to mean “no effect”: generations are md5-identical to the bare base model.
- One upstream-worthy bug fix in core: an off-by-one in
clip_preprocess center-cropping that crashed any CLIP-vision path on odd input dimensions.
- NVIDIA Triton Python backend fronting a vLLM-served speech LLM, taken from prototype to production.
- Derives confidence, sentiment and spoken-language ID from the model’s own token-level outputs. No additional models to train or serve.
- Orchestrates per-signal LLM queries over gRPC alongside ONNX Runtime inference for confidence calibration and forced alignment.
- p50 latency of approximately 5 to 10 ms per signal.
- Technical lead for a three-engineer team. A later concurrency refactor took sustained throughput from roughly 8 to 80 requests per second.
- Retrieval-augmented personalization for the same speech LLM behind the paralinguistic backend.
- A user’s contacts, device names and music library are retrieved from their catalogs with approximate-nearest-neighbor search.
- Retrieved entries are injected into the LLM’s prefill context, so the model transcribes the user’s own vocabulary instead of guessing at it.
- The original design needed three round trips through the LLM per request. Working with the applied science team, we replaced LLM-generated query embeddings with an index keyed on voice features, eliminating one of them, a roughly 20 ms LLM round trip per request.
- Keying retrieval on voice features also decoupled the lookup from the LLM’s context request, so retrieval runs speculatively, in parallel with the model, instead of waiting on it.
- Led the effort end to end, from retrieval design to serving.
- Load testing showed latency growing linearly with concurrency: the server was serializing, capping throughput at about 8 requests per second.
- Three structural fixes: parallelize independent ONNX inference calls, replace a reference-counted five-thread dispatcher with single-threaded cooperative multitasking, and run several independent backend instances.
- Latency curve went from linear to roughly square-root in concurrency; sustained throughput reached 64 to 80 requests per second on the same hardware, an 8 to 10x improvement.
- Single-request latency barely moved (about 90 to 70 ms). This was a contention fix, not a per-request speedup.
- Nova Sonic needed the speech framework’s pipelines delivered as a containerized gRPC service: audio and system prompts streaming in, inference requests streaming out. The team’s code had only ever been called through JNI.
- Mapped gRPC C++’s bidirectional-streaming reactor onto the framework’s pipe abstraction once, as a reusable layer covering session setup and teardown, signal handling, graceful error handling and logging.
- A new service needs under 300 lines of integration code: which pipe input receives request messages, and which pipe outputs become response messages. Adopted org-wide as the standard way to deploy a pipeline.
- Established Protobuf and gRPC generated code as first-class CMake libraries in the framework’s build, so the same generated types are consumed by the pipeline’s nodes and by the server without duplicate-symbol conflicts.
- Python provisioning system that stands up the complete Nova Sonic inference stack on one developer machine.
- Deploys several Triton model containers plus the C++ node-graph orchestrator, wired together and ready to take speech-to-speech traffic.
- Took a fifteen-person team from sharing a couple of hand-built demo environments to every engineer having their own.
- Same pattern as L3Dockerize: make the right environment the cheap one.
- Nodes in the C++ stream-processing framework behind Nova Sonic register themselves in a global registry through static initialization. No central list to edit, no explicit dependency from tools on the nodes they might load.
- Registry is a thread-safe, function-local static constructed on first use, so registration is safe regardless of static initialization order across translation units.
- Registration works by linking a node library or by
LD_PRELOADing it. The development CLI can assemble a graph from a JSON definition using nodes it was never compiled against. - Pulled double duty as a dependency-inversion mechanism: consumers depend on the node interface, not on implementing libraries. Adopted across the organization and later picked up for embedded speech processing.
- The legacy ASR engine, roughly 500k lines of C++, was built by its owners with GCC and plain CMake for the cloud. The on-device team consumed it as a library across about 7 ARM toolchains from the Android NDK plus 4 x86 gcc/clang variants, through a Conan-based build layer.
- Before: on-device releases were hand-curated snapshots of the upstream engine, maintained in forks of every ASR package and re-merged periodically. Each release took days at minimum and often weeks, as toolchain-specific breakage surfaced and needed patches.
- Fix: point all engine code at a single virtual build-system package. In the cloud dependency universe it resolves to bare CMake; in the device universe it resolves to the Conan layer, which drives the same CMake underneath. About 40 packages converted; one source tree builds for all 12 toolchains.
- Kept it that way with a pre-merge analyzer that compiles every pull request against the device toolchains before it can land, so device compatibility is checked at merge time rather than discovered at release time.
- Haskell library for distributing Monte Carlo Tree sampling among AWS Lambda instances.
- Demonstrates the graceful manner with which functional paradigms align with a distributed and serverless model.
- Deployed with ECR, API Gateway and AWS Lambda, making use of CloudFormation for simple deployment