Projects on jdcsen

Self-Hosted Multi-Modal Inference on One GPU

Thu, 01 Jan 1970 00:33:46 +0000

One OpenAI- and Anthropic-compatible endpoint fronting 36 model keys on a single RTX 5090 (32 GB): 14 LLM keys (Qwen3 coder, thinking and instruct tiers, a vision model, a captioner), 4 Whisper variants, 11 image generators (Flux, SDXL, Chroma, Qwen-Image), 4 Wan video models and 3 GPU feature-extraction sidecars.
Built on llama-swap, a Go router that starts and stops upstream inference processes on demand. Anything that speaks HTTP can be an upstream, which is what lets llama.cpp, whisper.cpp, a patched stable-diffusion.cpp and three PyTorch services share one card behind one API.
Co-residency is declared with set-algebra rules and eviction costs, but llama-swap does not measure VRAM, so I did: a sweep script that measures resident and peak footprints per model and per combination, which turned up a 6.7 GB transient VAE-decode spike as the binding constraint.
Every workhorse LLM has two keys: an exclusive full-context key and a co-resident “lite” twin, so a 256k-context 30B model and an image generator never fight for the card.
Heavy upstreams run as sibling containers launched on demand, so the router image rebuilds in seconds instead of recompiling sd-server and three multi-gigabyte venvs.

stable-diffusion.cpp: Identity Conditioning in sd-server

Thu, 01 Jan 1970 00:33:46 +0000

Fork of stable-diffusion.cpp adding per-request reference-image identity conditioning (PhotoMaker v2 on SDXL bases, PuLID on Flux) to the sd-server HTTP surface. Upstream registered the flags but only the CLI ever populated them.
Eleven commits, about 1,900 lines added over 19 files. Roughly 89% lives in examples/server/; the core engine changes total 87 lines. sd-cli and the core library stay Python-free.
Reference-image encoding runs in-process through an embedded CPython interpreter (pybind11), behind two CMake flags that default to OFF so the vanilla build is unchanged.
Identity embeddings can be extracted once and re-injected: the round trip reproduces the image-path generation byte for byte at a fixed seed.
“No identity images” is proven to mean “no effect”: generations are md5-identical to the bare base model.
One upstream-worthy bug fix in core: an off-by-one in clip_preprocess center-cropping that crashed any CLIP-vision path on odd input dimensions.

Paralinguistic Signals from a Speech LLM

Thu, 01 Jan 1970 00:33:45 +0000

NVIDIA Triton Python backend fronting a vLLM-served speech LLM, taken from prototype to production.
Derives confidence, sentiment and spoken-language ID from the model’s own token-level outputs. No additional models to train or serve.
Orchestrates per-signal LLM queries over gRPC alongside ONNX Runtime inference for confidence calibration and forced alignment.
p50 latency of approximately 5 to 10 ms per signal.
Technical lead for a three-engineer team. A later concurrency refactor took sustained throughput from roughly 8 to 80 requests per second.

RAG Personalization for Speech Recognition

Thu, 01 Jan 1970 00:33:45 +0000

Retrieval-augmented personalization for the same speech LLM behind the paralinguistic backend.
A user’s contacts, device names and music library are retrieved from their catalogs with approximate-nearest-neighbor search.
Retrieved entries are injected into the LLM’s prefill context, so the model transcribes the user’s own vocabulary instead of guessing at it.
The original design needed three round trips through the LLM per request. Working with the applied science team, we replaced LLM-generated query embeddings with an index keyed on voice features, eliminating one of them, a roughly 20 ms LLM round trip per request.
Keying retrieval on voice features also decoupled the lookup from the LLM’s context request, so retrieval runs speculatively, in parallel with the model, instead of waiting on it.
Led the effort end to end, from retrieval design to serving.

Triton Backend Throughput Refactor

Thu, 01 Jan 1970 00:33:45 +0000

Load testing showed latency growing linearly with concurrency: the server was serializing, capping throughput at about 8 requests per second.
Three structural fixes: parallelize independent ONNX inference calls, replace a reference-counted five-thread dispatcher with single-threaded cooperative multitasking, and run several independent backend instances.
Latency curve went from linear to roughly square-root in concurrency; sustained throughput reached 64 to 80 requests per second on the same hardware, an 8 to 10x improvement.
Single-request latency barely moved (about 90 to 70 ms). This was a contention fix, not a per-request speedup.

gRPC Bidirectional Streaming for a C++ Node-Graph Framework

Thu, 01 Jan 1970 00:33:44 +0000

Nova Sonic needed the speech framework’s pipelines delivered as a containerized gRPC service: audio and system prompts streaming in, inference requests streaming out. The team’s code had only ever been called through JNI.
Mapped gRPC C++’s bidirectional-streaming reactor onto the framework’s pipe abstraction once, as a reusable layer covering session setup and teardown, signal handling, graceful error handling and logging.
A new service needs under 300 lines of integration code: which pipe input receives request messages, and which pipe outputs become response messages. Adopted org-wide as the standard way to deploy a pipeline.
Established Protobuf and gRPC generated code as first-class CMake libraries in the framework’s build, so the same generated types are consumed by the pipeline’s nodes and by the server without duplicate-symbol conflicts.

One-Command Nova Sonic Dev Environments

Thu, 01 Jan 1970 00:33:44 +0000

Python provisioning system that stands up the complete Nova Sonic inference stack on one developer machine.
Deploys several Triton model containers plus the C++ node-graph orchestrator, wired together and ready to take speech-to-speech traffic.
Took a fifteen-person team from sharing a couple of hand-built demo environments to every engineer having their own.
Same pattern as L3Dockerize: make the right environment the cheap one.

Static Service Registry for a C++ Node-Graph Framework

Thu, 01 Jan 1970 00:33:43 +0000

Nodes in the C++ stream-processing framework behind Nova Sonic register themselves in a global registry through static initialization. No central list to edit, no explicit dependency from tools on the nodes they might load.
Registry is a thread-safe, function-local static constructed on first use, so registration is safe regardless of static initialization order across translation units.
Registration works by linking a node library or by LD_PRELOADing it. The development CLI can assemble a graph from a JSON definition using nodes it was never compiled against.
Pulled double duty as a dependency-inversion mechanism: consumers depend on the node interface, not on implementing libraries. Adopted across the organization and later picked up for embedded speech processing.

One Build for Cloud and Device: Consolidating a 500k-Line C++ Engine

Thu, 01 Jan 1970 00:33:42 +0000

The legacy ASR engine, roughly 500k lines of C++, was built by its owners with GCC and plain CMake for the cloud. The on-device team consumed it as a library across about 7 ARM toolchains from the Android NDK plus 4 x86 gcc/clang variants, through a Conan-based build layer.
Before: on-device releases were hand-curated snapshots of the upstream engine, maintained in forks of every ASR package and re-merged periodically. Each release took days at minimum and often weeks, as toolchain-specific breakage surfaced and needed patches.
Fix: point all engine code at a single virtual build-system package. In the cloud dependency universe it resolves to bare CMake; in the device universe it resolves to the Conan layer, which drives the same CMake underneath. About 40 packages converted; one source tree builds for all 12 toolchains.
Kept it that way with a pre-merge analyzer that compiles every pull request against the device toolchains before it can land, so device compatibility is checked at merge time rather than discovered at release time.

DMCTS: Distributed Monte Carlo Tree Search

Thu, 01 Jan 1970 00:33:41 +0000

Haskell library for distributing Monte Carlo Tree sampling among AWS Lambda instances.
Demonstrates the graceful manner with which functional paradigms align with a distributed and serverless model.
Deployed with ECR, API Gateway and AWS Lambda, making use of CloudFormation for simple deployment

L3Dockerize: Python-based containerization tool

Thu, 01 Jan 1970 00:33:40 +0000

Python-based Docker container builder.
Dynamically resolves and fetches software component dependencies to build Docker images.
Intended to serve as a low/zero cost tool for developers to convert existing software installs to more easily monitorable, scalable, and extensible docker-based installations.
Concept evaluated in a system-wide CIT test of an L3Harris hardware product.

BUPDATE: Batch Updating tool

Thu, 01 Jan 1970 00:33:39 +0000

Python/Bash Scripting Toolkit for cross-project refactoring.
Uninvasive, agile solution to deal with problem of poorly-isolated dependencies impacting many components.
Core application written in Python, per-project scripts writable in any language.
Designed according to Unix ”simple, clear, modular, extensible” philosophy to facilitate integration into existing toolchains and processes.
Successfully deployed to automatically upgrade the dependencies of approx. 60 components without user intervention.

DRAGLINE: Dynamic RAcinG LINE

Thu, 01 Jan 1970 00:33:39 +0000

Augmented Reality Racing Line System running on an NVidia Jetson TX2.
Microservice-based architecture comprised of Python, C++, Javascript, and MySQL components.
Winner of 2019 Graduating Class Senior Design Competition.
Honors Thesis was an attempt at replacing L-BFGS-B Gradient Descent optimizer with a more reliable, machine-learning based approach.

Honors Undergraduate Thesis

Thu, 01 Jan 1970 00:33:39 +0000

Intended to provide a machine-learning based replacement for the L-BFGS-B gradient descent approach used by DRAGLINE
Evaluated the performance of simple feed-forward neural networks when tasked with predicting optimal paths through racecourses.
Provided the author an excuse to naively apply newly acquired machine-learning knowledge to a completely inappropriate problem.

Safety-Critical Power Control for a Fielded RF Front End

Thu, 01 Jan 1970 00:33:39 +0000

A new L3Harris radio needed far more output dynamic range than the fielded RF front end it had to work with was designed to give.
The front end’s transmit chain had two gain stages: a coarse driver with three setpoints (low, medium, high) and a downstream fine stage adjustable in dB. The radio commanded a single power level. I designed and implemented the control software that turns that level into a safe setpoint pair for both stages.
Reaching the required range meant disabling the equipment’s built-in protection interlocks, so the software became the only thing standing between a command and hardware damage. Both stages maxed, or either past its limit, could destroy the amplifier and drop a pilot’s radio link.
Acted as de-facto software lead: direction for a small team, and project status reporting to organization management.

Formula SAE Controls Team Lead

Thu, 01 Jan 1970 00:33:38 +0000

Led engineering teams to complete critical systems of a year-long engineering effort to develop an electric Formula race car.
Participated in and led design reviews for critical vehicle components.
Architected and implemented vehicle low-voltage systems, including safety circuits, vehicle networks, telemetry, and control.
Designed and built vehicle CCAs, working with EDA software (Eagle/KiCAD), from requirement collection to delivery.
Worked in close-knit, cross-functional teams to ensure effective and efficient concurrent development.