• Fork of stable-diffusion.cpp adding per-request reference-image identity conditioning (PhotoMaker v2 on SDXL bases, PuLID on Flux) to the sd-server HTTP surface. Upstream registered the flags but only the CLI ever populated them.
• Eleven commits, about 1,900 lines added over 19 files. Roughly 89% lives in examples/server/; the core engine changes total 87 lines. sd-cli and the core library stay Python-free.
• Reference-image encoding runs in-process through an embedded CPython interpreter (pybind11), behind two CMake flags that default to OFF so the vanilla build is unchanged.
• Identity embeddings can be extracted once and re-injected: the round trip reproduces the image-path generation byte for byte at a fixed seed.
• “No identity images” is proven to mean “no effect”: generations are md5-identical to the bare base model.
• One upstream-worthy bug fix in core: an off-by-one in clip_preprocess center-cropping that crashed any CLIP-vision path on odd input dimensions.

stable-diffusion.cpp is a ggml-based diffusion engine in the same family as llama.cpp, and its sd-server is what my self-hosted inference stack uses for image generation. Upstream supports two identity-conditioning methods, PhotoMaker v2 and PuLID, that let a generation be steered toward a reference image. Both were effectively CLI-only. PhotoMaker’s flags were registered for the server, but nothing on any request path populated the identity images, so --photo-maker on a server was inert. PuLID’s engine consumed a precomputed embedding file and had no way to produce one. This fork closes both gaps.

Keep the patch out of the engine

The first design decision was where the code should live. The engine already knew how to consume identity conditioning; what was missing was everything upstream of it: decoding reference images off an HTTP request, running the reference encoders, and handing the engine an embedding in the format it expected. So almost all of the work sits in examples/server/. The core library gets two new capability queries in the public C API, an extension hook, and a ten-line bug fix, and nothing else. That is deliberate. It keeps sd-cli and the core untouched by Python, and it keeps the fork mergeable against a fast-moving upstream (an upstream merge landed mid-project and was absorbed without conflict in the engine).

The reference encoders are Python models, and the server is C++. Rather than shell out to a subprocess per request, the server embeds a CPython interpreter with pybind11, warms it once at startup, and calls the encoders in-process. Encoding runs on CPU by design so it never contends with the diffusion model for VRAM; it costs tens of milliseconds against a generation measured in tens of seconds. Both interpreters sit behind CMake flags (SD_SERVER_PHOTOMAKER_PY, SD_SERVER_PULID_PY) that default to OFF. With the flags off every Python-backed call is a stub that reports the feature as unavailable, so the vanilla build compiles, links and runs exactly as before; the stack’s Dockerfile turns them on.

The engine’s only ingestion path for an embedding is a file, so the server writes one per request through mkstemp and points the generation parameters at it. That seam is the whole integration, which is why the core patch is so small.

Making the feature honest

Two invariants took more work than the feature itself.

No identity images must mean no effect. PhotoMaker v2’s weights carry a required UNet LoRA that upstream merged on every request, so a request with no reference images on a PhotoMaker model did not produce the same image as the bare base. I added a begin_request hook on the generation-extension interface, dispatched before LoRAs are applied, and gated the merge on whether identity images are actually pending. Separately, the PhotoMaker encoder is now only loaded on UNet architectures, so on DiT bases like Flux or Qwen-Image the flag costs no backend, no weights and no VRAM. Verified by md5: a request with identity weight zero is byte-identical to a generation with the feature off at the same seed.

Partial input must not silently degrade. The PhotoMaker loader requires the identity-image count to match the embedding row count. If the encoder found nothing usable in one of several images, the mismatch quietly disabled the feature for the whole request with only a warning. The fix reports which inputs were kept and prunes the image list to match, so a partial set still generates, and only an unusable set returns a 400 with a reason. Requests on an incompatible model, or missing the prompt trigger the method requires, are bounced the same way instead of producing an unconditioned image.

Extract once, inject anywhere

The final commit adds a synchronous POST /sdcpp/v1/extract_id endpoint that returns the canonical embedding (shape, dtype, base64 payload) and lets a client send it back on later requests in place of the reference images. The read and write paths for embeddings are plain C++, outside the Python guards, so a client-supplied embedding injects in any build. PuLID’s embedding fully replaces its reference images; PhotoMaker’s replaces only the encoder vector because its CLIP tower still consumes pixels, and the capabilities endpoint advertises that difference per method. Extract-then-inject reproduces the image-path generation byte for byte at a fixed seed.

For PuLID, the reference implementation encodes exactly one image. Fusing several is done in the extractor by porting the reference pipeline up to the point before its IDFormer stage, averaging there, and running IDFormer once; single-image input takes the untouched reference path so its output stays identical.

The bug fix I’d send first

clip_preprocess computed its resize scale as max(width_scale, height_scale), which guarantees both scaled dimensions reach the target in exact arithmetic. Truncating the float product does not: a 201-pixel edge scaled to 224 comes out as 223.999985, truncates to 223, and the center crop reads one past the short edge and throws. Rounding, then flooring at the target, makes the crop safe. It is a general crash fix for any CLIP-vision path on awkward input sizes, ten lines in src/core/util.cpp, and the natural first pull request upstream, ahead of the pure-C++ server plumbing and, last and least likely to be accepted, the embedded-Python extractor.

Known debt, so nobody has to find it: the per-request temp files are not unlinked after generation, the validation is a self-test entrypoint plus live smoke suites rather than an automated test target, and the reference encoders’ pretrained packs are research-licensed, which keeps this a personal-use feature.