Self-Hosted Multi-Modal Inference on One GPU

Thu, 01 Jan 1970 00:33:46 +0000

One OpenAI- and Anthropic-compatible endpoint fronting 36 model keys on a single RTX 5090 (32 GB): 14 LLM keys (Qwen3 coder, thinking and instruct tiers, a vision model, a captioner), 4 Whisper variants, 11 image generators (Flux, SDXL, Chroma, Qwen-Image), 4 Wan video models and 3 GPU feature-extraction sidecars.
Built on llama-swap, a Go router that starts and stops upstream inference processes on demand. Anything that speaks HTTP can be an upstream, which is what lets llama.cpp, whisper.cpp, a patched stable-diffusion.cpp and three PyTorch services share one card behind one API.
Co-residency is declared with set-algebra rules and eviction costs, but llama-swap does not measure VRAM, so I did: a sweep script that measures resident and peak footprints per model and per combination, which turned up a 6.7 GB transient VAE-decode spike as the binding constraint.
Every workhorse LLM has two keys: an exclusive full-context key and a co-resident “lite” twin, so a 256k-context 30B model and an image generator never fight for the card.
Heavy upstreams run as sibling containers launched on demand, so the router image rebuilds in seconds instead of recompiling sd-server and three multi-gigabyte venvs.

stable-diffusion.cpp: Identity Conditioning in sd-server

Thu, 01 Jan 1970 00:33:46 +0000

Fork of stable-diffusion.cpp adding per-request reference-image identity conditioning (PhotoMaker v2 on SDXL bases, PuLID on Flux) to the sd-server HTTP surface. Upstream registered the flags but only the CLI ever populated them.
Eleven commits, about 1,900 lines added over 19 files. Roughly 89% lives in examples/server/; the core engine changes total 87 lines. sd-cli and the core library stay Python-free.
Reference-image encoding runs in-process through an embedded CPython interpreter (pybind11), behind two CMake flags that default to OFF so the vanilla build is unchanged.
Identity embeddings can be extracted once and re-injected: the round trip reproduces the image-path generation byte for byte at a fixed seed.
“No identity images” is proven to mean “no effect”: generations are md5-identical to the bare base model.
One upstream-worthy bug fix in core: an off-by-one in clip_preprocess center-cropping that crashed any CLIP-vision path on odd input dimensions.