RAG Personalization for Speech Recognition

Thu, 01 Jan 1970 00:33:45 +0000

Retrieval-augmented personalization for the same speech LLM behind the paralinguistic backend.
A user’s contacts, device names and music library are retrieved from their catalogs with approximate-nearest-neighbor search.
Retrieved entries are injected into the LLM’s prefill context, so the model transcribes the user’s own vocabulary instead of guessing at it.
The original design needed three round trips through the LLM per request. Working with the applied science team, we replaced LLM-generated query embeddings with an index keyed on voice features, eliminating one of them, a roughly 20 ms LLM round trip per request.
Keying retrieval on voice features also decoupled the lookup from the LLM’s context request, so retrieval runs speculatively, in parallel with the model, instead of waiting on it.
Led the effort end to end, from retrieval design to serving.

RAG on jdcsen