Projects with the tag context-engineering
- Retrieval-augmented personalization for the same speech LLM behind the paralinguistic backend.
- A user’s contacts, device names and music library are retrieved from their catalogs with approximate-nearest-neighbor search.
- Retrieved entries are injected into the LLM’s prefill context, so the model transcribes the user’s own vocabulary instead of guessing at it.
- The original design needed three round trips through the LLM per request. Working with the applied science team, we replaced LLM-generated query embeddings with an index keyed on voice features, eliminating one of them, a roughly 20 ms LLM round trip per request.
- Keying retrieval on voice features also decoupled the lookup from the LLM’s context request, so retrieval runs speculatively, in parallel with the model, instead of waiting on it.
- Led the effort end to end, from retrieval design to serving.