Projects with the tag containerization
- One OpenAI- and Anthropic-compatible endpoint fronting 36 model keys on a single RTX 5090 (32 GB): 14 LLM keys (Qwen3 coder, thinking and instruct tiers, a vision model, a captioner), 4 Whisper variants, 11 image generators (Flux, SDXL, Chroma, Qwen-Image), 4 Wan video models and 3 GPU feature-extraction sidecars.
- Built on llama-swap, a Go router that starts and stops upstream inference processes on demand. Anything that speaks HTTP can be an upstream, which is what lets llama.cpp, whisper.cpp, a patched stable-diffusion.cpp and three PyTorch services share one card behind one API.
- Co-residency is declared with set-algebra rules and eviction costs, but llama-swap does not measure VRAM, so I did: a sweep script that measures resident and peak footprints per model and per combination, which turned up a 6.7 GB transient VAE-decode spike as the binding constraint.
- Every workhorse LLM has two keys: an exclusive full-context key and a co-resident “lite” twin, so a 256k-context 30B model and an image generator never fight for the card.
- Heavy upstreams run as sibling containers launched on demand, so the router image rebuilds in seconds instead of recompiling sd-server and three multi-gigabyte venvs.
- Python provisioning system that stands up the complete Nova Sonic inference stack on one developer machine.
- Deploys several Triton model containers plus the C++ node-graph orchestrator, wired together and ready to take speech-to-speech traffic.
- Took a fifteen-person team from sharing a couple of hand-built demo environments to every engineer having their own.
- Same pattern as L3Dockerize: make the right environment the cheap one.
- Haskell library for distributing Monte Carlo Tree sampling among AWS Lambda instances.
- Demonstrates the graceful manner with which functional paradigms align with a distributed and serverless model.
- Deployed with ECR, API Gateway and AWS Lambda, making use of CloudFormation for simple deployment
- Python-based Docker container builder.
- Dynamically resolves and fetches software component dependencies to build Docker images.
- Intended to serve as a low/zero cost tool for developers to convert existing software installs to more easily monitorable, scalable, and extensible docker-based installations.
- Concept evaluated in a system-wide CIT test of an L3Harris hardware product.