Python on jdcsen

Self-Hosted Multi-Modal Inference on One GPU

Thu, 01 Jan 1970 00:33:46 +0000

One OpenAI- and Anthropic-compatible endpoint fronting 36 model keys on a single RTX 5090 (32 GB): 14 LLM keys (Qwen3 coder, thinking and instruct tiers, a vision model, a captioner), 4 Whisper variants, 11 image generators (Flux, SDXL, Chroma, Qwen-Image), 4 Wan video models and 3 GPU feature-extraction sidecars.
Built on llama-swap, a Go router that starts and stops upstream inference processes on demand. Anything that speaks HTTP can be an upstream, which is what lets llama.cpp, whisper.cpp, a patched stable-diffusion.cpp and three PyTorch services share one card behind one API.
Co-residency is declared with set-algebra rules and eviction costs, but llama-swap does not measure VRAM, so I did: a sweep script that measures resident and peak footprints per model and per combination, which turned up a 6.7 GB transient VAE-decode spike as the binding constraint.
Every workhorse LLM has two keys: an exclusive full-context key and a co-resident “lite” twin, so a 256k-context 30B model and an image generator never fight for the card.
Heavy upstreams run as sibling containers launched on demand, so the router image rebuilds in seconds instead of recompiling sd-server and three multi-gigabyte venvs.

Thu, 01 Jan 1970 00:33:46 +0000

Fork of stable-diffusion.cpp adding per-request reference-image identity conditioning (PhotoMaker v2 on SDXL bases, PuLID on Flux) to the sd-server HTTP surface. Upstream registered the flags but only the CLI ever populated them.
Eleven commits, about 1,900 lines added over 19 files. Roughly 89% lives in examples/server/; the core engine changes total 87 lines. sd-cli and the core library stay Python-free.
Reference-image encoding runs in-process through an embedded CPython interpreter (pybind11), behind two CMake flags that default to OFF so the vanilla build is unchanged.
Identity embeddings can be extracted once and re-injected: the round trip reproduces the image-path generation byte for byte at a fixed seed.
“No identity images” is proven to mean “no effect”: generations are md5-identical to the bare base model.
One upstream-worthy bug fix in core: an off-by-one in clip_preprocess center-cropping that crashed any CLIP-vision path on odd input dimensions.

Thu, 01 Jan 1970 00:33:45 +0000

NVIDIA Triton Python backend fronting a vLLM-served speech LLM, taken from prototype to production.
Derives confidence, sentiment and spoken-language ID from the model’s own token-level outputs. No additional models to train or serve.
Orchestrates per-signal LLM queries over gRPC alongside ONNX Runtime inference for confidence calibration and forced alignment.
p50 latency of approximately 5 to 10 ms per signal.
Technical lead for a three-engineer team. A later concurrency refactor took sustained throughput from roughly 8 to 80 requests per second.

Thu, 01 Jan 1970 00:33:45 +0000

Retrieval-augmented personalization for the same speech LLM behind the paralinguistic backend.
A user’s contacts, device names and music library are retrieved from their catalogs with approximate-nearest-neighbor search.
Retrieved entries are injected into the LLM’s prefill context, so the model transcribes the user’s own vocabulary instead of guessing at it.
The original design needed three round trips through the LLM per request. Working with the applied science team, we replaced LLM-generated query embeddings with an index keyed on voice features, eliminating one of them, a roughly 20 ms LLM round trip per request.
Keying retrieval on voice features also decoupled the lookup from the LLM’s context request, so retrieval runs speculatively, in parallel with the model, instead of waiting on it.
Led the effort end to end, from retrieval design to serving.

Thu, 01 Jan 1970 00:33:45 +0000

Load testing showed latency growing linearly with concurrency: the server was serializing, capping throughput at about 8 requests per second.
Three structural fixes: parallelize independent ONNX inference calls, replace a reference-counted five-thread dispatcher with single-threaded cooperative multitasking, and run several independent backend instances.
Latency curve went from linear to roughly square-root in concurrency; sustained throughput reached 64 to 80 requests per second on the same hardware, an 8 to 10x improvement.
Single-request latency barely moved (about 90 to 70 ms). This was a contention fix, not a per-request speedup.

Thu, 01 Jan 1970 00:33:44 +0000

Python provisioning system that stands up the complete Nova Sonic inference stack on one developer machine.
Deploys several Triton model containers plus the C++ node-graph orchestrator, wired together and ready to take speech-to-speech traffic.
Took a fifteen-person team from sharing a couple of hand-built demo environments to every engineer having their own.
Same pattern as L3Dockerize: make the right environment the cheap one.

Thu, 01 Jan 1970 00:33:40 +0000

Python-based Docker container builder.
Dynamically resolves and fetches software component dependencies to build Docker images.
Intended to serve as a low/zero cost tool for developers to convert existing software installs to more easily monitorable, scalable, and extensible docker-based installations.
Concept evaluated in a system-wide CIT test of an L3Harris hardware product.

Thu, 01 Jan 1970 00:33:39 +0000

Python/Bash Scripting Toolkit for cross-project refactoring.
Uninvasive, agile solution to deal with problem of poorly-isolated dependencies impacting many components.
Core application written in Python, per-project scripts writable in any language.
Designed according to Unix ”simple, clear, modular, extensible” philosophy to facilitate integration into existing toolchains and processes.
Successfully deployed to automatically upgrade the dependencies of approx. 60 components without user intervention.

Thu, 01 Jan 1970 00:33:39 +0000

Augmented Reality Racing Line System running on an NVidia Jetson TX2.
Microservice-based architecture comprised of Python, C++, Javascript, and MySQL components.
Winner of 2019 Graduating Class Senior Design Competition.
Honors Thesis was an attempt at replacing L-BFGS-B Gradient Descent optimizer with a more reliable, machine-learning based approach.