- Load testing showed latency growing linearly with concurrency: the server was serializing, capping throughput at about 8 requests per second.
- Three structural fixes: parallelize independent ONNX inference calls, replace a reference-counted five-thread dispatcher with single-threaded cooperative multitasking, and run several independent backend instances.
- Latency curve went from linear to roughly square-root in concurrency; sustained throughput reached 64 to 80 requests per second on the same hardware, an 8 to 10x improvement.
- Single-request latency barely moved (about 90 to 70 ms). This was a contention fix, not a per-request speedup.
The paralinguistic backend worked, and its latency at concurrency one was fine. Its first real load test told a different story.
Diagnosis
Plotting latency against offered concurrency gave a nearly straight line:
| Concurrent requests | Before (ms, approx.) | After (ms, approx.) |
|---|---|---|
| 1 | 90 | 70 |
| 4 | 500 | 120 |
| 8 | 1200 | 150 |
| 16 | 2000 | 200 |
| 32 | 4000 | 500 |
The “before” column fits latency proportional to about concurrency to the first power. Latency growing linearly with concurrency means each request is waiting its turn behind the others: the server is serializing somewhere, and sustained throughput was pinned at approximately 8 requests per second no matter how much load was offered. That shape, more than any absolute number, is what told me where to look.
Three fixes
- Parallelize the serialized inference. Inside each request the backend ran its ONNX Runtime inference calls, confidence calibration and forced alignment, one after the other. They were independent, so I made them run concurrently.
- Replace reference counting with cooperative multitasking. Request lifetime had been managed by a reference-counted dispatcher spread across five threads. I replaced it with a yield-based cooperative design on a single application thread per instance. That eliminated a whole class of refcounting bugs, made the request path far easier to read, and cut four of the five threads, and with them the GIL contention within a request.
- Run independent model instances. With the dispatcher no longer shared state, Triton could be configured with several independent instances of the backend, removing the GIL contention across requests as well.
Result
Afterwards the “after” column fits latency proportional to roughly the square root of concurrency. Sustained throughput landed between 64 and 80 requests per second on the same hardware, an 8 to 10x improvement in both latency under load and throughput.
The number I like to point at is the concurrency-one latency, which barely moved. This was never a per-request speedup; it was a contention fix, and measuring it as a curve rather than a single number is what pointed at the right changes.