• Retrieval-augmented personalization for the same speech LLM behind the paralinguistic backend.
• A user’s contacts, device names and music library are retrieved from their catalogs with approximate-nearest-neighbor search.
• Retrieved entries are injected into the LLM’s prefill context, so the model transcribes the user’s own vocabulary instead of guessing at it.
• The original design needed three round trips through the LLM per request. Working with the applied science team, we replaced LLM-generated query embeddings with an index keyed on voice features, eliminating one of them, a roughly 20 ms LLM round trip per request.
• Keying retrieval on voice features also decoupled the lookup from the LLM’s context request, so retrieval runs speculatively, in parallel with the model, instead of waiting on it.
• Led the effort end to end, from retrieval design to serving.

A general-purpose speech model is very good at general-purpose speech. It is much worse at the words that matter most in a voice assistant: the name of your cousin, the nickname you gave the kitchen speaker, the band nobody outside your household has heard of. Classical ASR engines handled this with per-user language-model biasing. An LLM has no such knob, but it has something better: a context window.

Retrieve, then prefill

The design is retrieval-augmented generation applied to transcription. Each user’s catalogs, contacts, registered devices and music, are indexed for approximate-nearest-neighbor search. At request time the backend retrieves the entries most likely to be relevant and injects them into the LLM’s prefill context ahead of the audio, so that when the model decodes a name it has already seen that name spelled the way the user spells it. The model itself is untouched; personalization lives entirely in what it is shown.

Latency was the whole problem

How much context to inject, and how strongly it should steer generation, were questions for our applied science partners. We worked with them to benchmark and tune both, but the serving side owned a harder constraint: latency. The original RAG design needed three round trips through the LLM for every request, using the model itself to produce the embeddings that drove retrieval before it could produce the transcript. Each pass through the largest model in the fleet costs about 20 ms, and three of them do not fit comfortably inside a voice assistant’s response budget.

The fix was to change what the index is keyed on. Instead of embeddings generated by the LLM, we indexed each user’s catalog against voice features, so the embedding pass disappears entirely. The LLM still requests context and then transcribes with the candidates in its prefill, but the round trip that existed only to produce a query embedding is gone. That was an applied science decision as much as an engineering one, and making the case for it, with latency numbers from the serving side, is the part of this project I’m proudest of. It took the personalized path from three LLM round trips to two, saving roughly 20 ms per request.

It also bought a second win that was not on the original list. With the LLM no longer producing the query, the lookup no longer depends on the LLM asking for context at all. The backend can retrieve speculatively, kicking off the catalog search from the voice features as soon as they are available and having the candidates ready before the model requests them. The retrieval latency moves off the critical path entirely, overlapping with work the LLM was going to do anyway.