• One OpenAI- and Anthropic-compatible endpoint fronting 36 model keys on a single RTX 5090 (32 GB): 14 LLM keys (Qwen3 coder, thinking and instruct tiers, a vision model, a captioner), 4 Whisper variants, 11 image generators (Flux, SDXL, Chroma, Qwen-Image), 4 Wan video models and 3 GPU feature-extraction sidecars.
• Built on llama-swap, a Go router that starts and stops upstream inference processes on demand. Anything that speaks HTTP can be an upstream, which is what lets llama.cpp, whisper.cpp, a patched stable-diffusion.cpp and three PyTorch services share one card behind one API.
• Co-residency is declared with set-algebra rules and eviction costs, but llama-swap does not measure VRAM, so I did: a sweep script that measures resident and peak footprints per model and per combination, which turned up a 6.7 GB transient VAE-decode spike as the binding constraint.
• Every workhorse LLM has two keys: an exclusive full-context key and a co-resident “lite” twin, so a 256k-context 30B model and an image generator never fight for the card.
• Heavy upstreams run as sibling containers launched on demand, so the router image rebuilds in seconds instead of recompiling sd-server and three multi-gigabyte venvs.

This is the inference stack I run at home. It started as a place to keep a coding-assistant LLM resident and grew into the backend for everything: transcription, image generation, video generation, and the feature extractors that feed a couple of downstream personal projects. Clients like opencode and ComfyUI point at one port and name a model; the stack loads what they asked for, evicts what is in the way, and answers.

One scheduler owns the card

The core is llama-swap, a small Go server that routes OpenAI and Anthropic API requests by the model field and starts or stops the upstream process that serves each one. It is a generalization of llama.cpp’s own multi-model router mode, with the difference that any HTTP-speaking server can be an upstream. Here that means llama-server and whisper-server in-process, plus sd-server and three PyTorch feature services as separate images. Surfaces that are not under /v1, like sd-server’s native async job API, are reached through a per-model passthrough where the model key in the URL drives load and evict.

Which models may share the card is declared as a set expression over single-letter tags, roughly “any light LLM with any light image generator, or one of these heavy models alone.” Every key carries a TTL (sticky for LLMs, five minutes for Whisper and the sidecars, ten for video) and a relative eviction cost so the router prefers to evict a 4B LLM before a 14B Wan model. Anything left out of every set is card-owning: requesting it evicts everything else and takes the whole 32 GB.

The scheduler is a description, not a measurement

llama-swap does not look at VRAM before it schedules. The rules are a promise you make on its behalf; if you promise that two models fit and they don’t, you get an out-of-memory 500 in the middle of a request. And measuring the truth is harder than running nvidia-smi: with sticky models and lazy eviction, naive sequential readings accumulate the previous model’s footprint, and eviction returns before CUDA has actually freed anything.

So the numbers come from a sweep script with a defined protocol: unload everything and wait until the card is actually below 800 MiB and holding, load a model, settle until three consecutive readings agree, record the resident figure, then run a representative request under a 100 ms sampler and record the peak. The sampler interval mattered. At 500 ms it read a co-resident peak of 24.4 GB; at 100 ms the same run showed 31.1 GB, because the binding constraint is not weights at all but a roughly 6.7 GB transient spike during VAE decode at 1024 squared. That measurement produced the rule the config is built on: a co-resident LLM has to stay under about 12.4 GB, which is why the 4B and 7B-class models are the ones allowed alongside image generation, and why every larger LLM gets an exclusive key and a lite twin.

The same measurements pay off in smaller ways. The original rules mandated Whisper in every combination, reserving its 4.3 GB whether or not anyone was transcribing; dropping that mandate freed exactly enough to promote one more image model into the co-resident tier. And KV cache, not weights, is what limits context length on this card: quantizing the cache to q8_0 is what lets the 30B mixture-of-experts models reach 256k tokens at a measured 31.1 GB.

Sidecars over the socket

Only the LLM and Whisper servers live in the router’s image. The patched sd-server and the three feature services (speaker diarization and embeddings; ControlNet-style image feature extractors such as depth, edges, pose and segmentation; and shot detection, body and camera estimation for video) are standalone images launched on demand as sibling containers sharing the router’s network namespace, with a per-key stop command so eviction is a real docker stop. Getting signal semantics right took a round: without an init process as PID 1 the stop signal is swallowed and eviction hangs. The payoff is that a base-image bump rebuilds the router in seconds instead of recompiling sd-server and re-baking three venvs. The cost is a mounted Docker socket, which is fine on a single-user box and would not be on anything else.

A few numbers that came out of the sidecar work: the video service decodes 1080p h264 through NVDEC at roughly 3,000 frames per second, about 100x realtime, so decode is never the bottleneck once it is overlapped with the batched forward pass, and the pose-estimation path runs at about 27x realtime. For the captioning model, continuous batching takes throughput from 157 to 807 tokens per second at concurrency 8, a 5.2x gain, without helping single-request latency at all, which is what you expect from a memory-bandwidth-bound decode.

Notes from the sharp edges

Running this many upstreams surfaces a lot of upstream behavior, and the repository keeps a list of twelve defects with workarounds: Whisper uploads that fail without ffmpeg in the image, an sd-server flag parser that exits silently on hyphen-versus-underscore mistakes, Flux-distilled models that need a CFG scale of 1.0 rather than the default, Wan’s 3D VAE decode needing CPU offload for a 14 GB buffer, a segfault in verbose transcription output when VAD filters a file to zero speech. Most of the stack’s value is in those notes and the measured tables rather than in any single piece of code, which is roughly half the reason it exists.