Projects with the tag python
- One OpenAI- and Anthropic-compatible endpoint fronting 36 model keys on a single RTX 5090 (32 GB): 14 LLM keys (Qwen3 coder, thinking and instruct tiers, a vision model, a captioner), 4 Whisper variants, 11 image generators (Flux, SDXL, Chroma, Qwen-Image), 4 Wan video models and 3 GPU feature-extraction sidecars.
- Built on llama-swap, a Go router that starts and stops upstream inference processes on demand. Anything that speaks HTTP can be an upstream, which is what lets llama.cpp, whisper.cpp, a patched stable-diffusion.cpp and three PyTorch services share one card behind one API.
- Co-residency is declared with set-algebra rules and eviction costs, but llama-swap does not measure VRAM, so I did: a sweep script that measures resident and peak footprints per model and per combination, which turned up a 6.7 GB transient VAE-decode spike as the binding constraint.
- Every workhorse LLM has two keys: an exclusive full-context key and a co-resident “lite” twin, so a 256k-context 30B model and an image generator never fight for the card.
- Heavy upstreams run as sibling containers launched on demand, so the router image rebuilds in seconds instead of recompiling sd-server and three multi-gigabyte venvs.
- Fork of stable-diffusion.cpp adding per-request reference-image identity conditioning (PhotoMaker v2 on SDXL bases, PuLID on Flux) to the
sd-server HTTP surface. Upstream registered the flags but only the CLI ever populated them. - Eleven commits, about 1,900 lines added over 19 files. Roughly 89% lives in
examples/server/; the core engine changes total 87 lines. sd-cli and the core library stay Python-free. - Reference-image encoding runs in-process through an embedded CPython interpreter (pybind11), behind two CMake flags that default to OFF so the vanilla build is unchanged.
- Identity embeddings can be extracted once and re-injected: the round trip reproduces the image-path generation byte for byte at a fixed seed.
- “No identity images” is proven to mean “no effect”: generations are md5-identical to the bare base model.
- One upstream-worthy bug fix in core: an off-by-one in
clip_preprocess center-cropping that crashed any CLIP-vision path on odd input dimensions.
- NVIDIA Triton Python backend fronting a vLLM-served speech LLM, taken from prototype to production.
- Derives confidence, sentiment and spoken-language ID from the model’s own token-level outputs. No additional models to train or serve.
- Orchestrates per-signal LLM queries over gRPC alongside ONNX Runtime inference for confidence calibration and forced alignment.
- p50 latency of approximately 5 to 10 ms per signal.
- Technical lead for a three-engineer team. A later concurrency refactor took sustained throughput from roughly 8 to 80 requests per second.
- Retrieval-augmented personalization for the same speech LLM behind the paralinguistic backend.
- A user’s contacts, device names and music library are retrieved from their catalogs with approximate-nearest-neighbor search.
- Retrieved entries are injected into the LLM’s prefill context, so the model transcribes the user’s own vocabulary instead of guessing at it.
- The original design needed three round trips through the LLM per request. Working with the applied science team, we replaced LLM-generated query embeddings with an index keyed on voice features, eliminating one of them, a roughly 20 ms LLM round trip per request.
- Keying retrieval on voice features also decoupled the lookup from the LLM’s context request, so retrieval runs speculatively, in parallel with the model, instead of waiting on it.
- Led the effort end to end, from retrieval design to serving.
- Load testing showed latency growing linearly with concurrency: the server was serializing, capping throughput at about 8 requests per second.
- Three structural fixes: parallelize independent ONNX inference calls, replace a reference-counted five-thread dispatcher with single-threaded cooperative multitasking, and run several independent backend instances.
- Latency curve went from linear to roughly square-root in concurrency; sustained throughput reached 64 to 80 requests per second on the same hardware, an 8 to 10x improvement.
- Single-request latency barely moved (about 90 to 70 ms). This was a contention fix, not a per-request speedup.
- Python provisioning system that stands up the complete Nova Sonic inference stack on one developer machine.
- Deploys several Triton model containers plus the C++ node-graph orchestrator, wired together and ready to take speech-to-speech traffic.
- Took a fifteen-person team from sharing a couple of hand-built demo environments to every engineer having their own.
- Same pattern as L3Dockerize: make the right environment the cheap one.
- Python-based Docker container builder.
- Dynamically resolves and fetches software component dependencies to build Docker images.
- Intended to serve as a low/zero cost tool for developers to convert existing software installs to more easily monitorable, scalable, and extensible docker-based installations.
- Concept evaluated in a system-wide CIT test of an L3Harris hardware product.
- Python/Bash Scripting Toolkit for cross-project refactoring.
- Uninvasive, agile solution to deal with problem of poorly-isolated dependencies impacting many components.
- Core application written in Python, per-project scripts writable in any language.
- Designed according to Unix ”simple, clear, modular, extensible” philosophy to facilitate integration into existing toolchains and processes.
- Successfully deployed to automatically upgrade the dependencies of approx. 60 components without user intervention.
- Augmented Reality Racing Line System running on an NVidia Jetson TX2.
- Microservice-based architecture comprised of Python, C++, Javascript, and MySQL components.
- Winner of 2019 Graduating Class Senior Design Competition.
- Honors Thesis was an attempt at replacing L-BFGS-B Gradient Descent optimizer with a more reliable, machine-learning based approach.