Containerization on jdcsen

Self-Hosted Multi-Modal Inference on One GPU

Thu, 01 Jan 1970 00:33:46 +0000

One OpenAI- and Anthropic-compatible endpoint fronting 36 model keys on a single RTX 5090 (32 GB): 14 LLM keys (Qwen3 coder, thinking and instruct tiers, a vision model, a captioner), 4 Whisper variants, 11 image generators (Flux, SDXL, Chroma, Qwen-Image), 4 Wan video models and 3 GPU feature-extraction sidecars.
Built on llama-swap, a Go router that starts and stops upstream inference processes on demand. Anything that speaks HTTP can be an upstream, which is what lets llama.cpp, whisper.cpp, a patched stable-diffusion.cpp and three PyTorch services share one card behind one API.
Co-residency is declared with set-algebra rules and eviction costs, but llama-swap does not measure VRAM, so I did: a sweep script that measures resident and peak footprints per model and per combination, which turned up a 6.7 GB transient VAE-decode spike as the binding constraint.
Every workhorse LLM has two keys: an exclusive full-context key and a co-resident “lite” twin, so a 256k-context 30B model and an image generator never fight for the card.
Heavy upstreams run as sibling containers launched on demand, so the router image rebuilds in seconds instead of recompiling sd-server and three multi-gigabyte venvs.

Thu, 01 Jan 1970 00:33:44 +0000

Python provisioning system that stands up the complete Nova Sonic inference stack on one developer machine.
Deploys several Triton model containers plus the C++ node-graph orchestrator, wired together and ready to take speech-to-speech traffic.
Took a fifteen-person team from sharing a couple of hand-built demo environments to every engineer having their own.
Same pattern as L3Dockerize: make the right environment the cheap one.

Thu, 01 Jan 1970 00:33:41 +0000

Haskell library for distributing Monte Carlo Tree sampling among AWS Lambda instances.
Demonstrates the graceful manner with which functional paradigms align with a distributed and serverless model.
Deployed with ECR, API Gateway and AWS Lambda, making use of CloudFormation for simple deployment

Thu, 01 Jan 1970 00:33:40 +0000

Python-based Docker container builder.
Dynamically resolves and fetches software component dependencies to build Docker images.
Intended to serve as a low/zero cost tool for developers to convert existing software installs to more easily monitorable, scalable, and extensible docker-based installations.
Concept evaluated in a system-wide CIT test of an L3Harris hardware product.