Projects with the tag homelab
- One OpenAI- and Anthropic-compatible endpoint fronting 36 model keys on a single RTX 5090 (32 GB): 14 LLM keys (Qwen3 coder, thinking and instruct tiers, a vision model, a captioner), 4 Whisper variants, 11 image generators (Flux, SDXL, Chroma, Qwen-Image), 4 Wan video models and 3 GPU feature-extraction sidecars.
- Built on llama-swap, a Go router that starts and stops upstream inference processes on demand. Anything that speaks HTTP can be an upstream, which is what lets llama.cpp, whisper.cpp, a patched stable-diffusion.cpp and three PyTorch services share one card behind one API.
- Co-residency is declared with set-algebra rules and eviction costs, but llama-swap does not measure VRAM, so I did: a sweep script that measures resident and peak footprints per model and per combination, which turned up a 6.7 GB transient VAE-decode spike as the binding constraint.
- Every workhorse LLM has two keys: an exclusive full-context key and a co-resident “lite” twin, so a 256k-context 30B model and an image generator never fight for the card.
- Heavy upstreams run as sibling containers launched on demand, so the router image rebuilds in seconds instead of recompiling sd-server and three multi-gigabyte venvs.