• Nova Sonic needed the speech framework’s pipelines delivered as a containerized gRPC service: audio and system prompts streaming in, inference requests streaming out. The team’s code had only ever been called through JNI.
• Mapped gRPC C++’s bidirectional-streaming reactor onto the framework’s pipe abstraction once, as a reusable layer covering session setup and teardown, signal handling, graceful error handling and logging.
• A new service needs under 300 lines of integration code: which pipe input receives request messages, and which pipe outputs become response messages. Adopted org-wide as the standard way to deploy a pipeline.
• Established Protobuf and gRPC generated code as first-class CMake libraries in the framework’s build, so the same generated types are consumed by the pipeline’s nodes and by the server without duplicate-symbol conflicts.

The C++ framework behind Amazon’s speech services is built from a small set of abstractions. Nodes and pipes are instances; node builders and pipe builders are the factories that make them (the static service registry is what holds the builders). A pipe has well-defined inputs and outputs, and a stream-processing implementation is a pipe wired from nodes.

For the Nova program our team was asked to expose that as a service: a container with a gRPC interface, where a client streams user audio and system prompts in and receives inference requests out, to be routed onward. This was new ground. Until then the framework had been consumed in-process through JNI, and nothing in it knew how to be a server.

A reactor is a pipe

Reading through gRPC’s C++ API, the bidirectional-streaming reactor stood out as the same shape as one of our pipes: an object that receives a stream of messages on one side, emits a stream on the other, and has a lifecycle with a beginning, an end, and errors in between. So rather than writing a server for one pipeline, I wrote the mapping between the two abstractions once.

The layer owns everything that is the same for every service. A client’s stream becomes a session; the pipe is built on session start and torn down on completion or cancellation; process signals shut sessions down cleanly instead of dropping them; pipe errors propagate to the client as gRPC status rather than crashing the server; and logging is consistent across services because it lives in the layer. What remains per service is the part that is actually different: which pipe input the incoming request messages feed, and which pipe outputs are turned into response messages. That comes to fewer than 300 lines of integration code for a new service, and most of those are declarations.

It worked because of one decision made early: the pipe’s input and output message types are the gRPC service definition’s message types. The pipe begins with a node that unpacks the request Protobuf and ends with a node that packs the response, so the reactor never translates, it only forwards. Container boundaries ended up aligning with team boundaries, since a team that owns a pipe can ship it as a service without involving anyone else, and that is what made the pattern the organization’s standard.

Protobuf as a library

That decision has a build-system consequence. The generated Protobuf and gRPC code is needed in two places: the server, which serves the service, and the pipe’s boundary nodes, which unpack and pack the messages. In a large codebase built across many toolchains, letting each consumer generate and compile its own copy is a fast route to duplicate symbols and mismatched versions. So I set up the generated code as proper CMake library targets, generated once, versioned with the .proto files, and linked by anyone who needs the types. Downstream teams consume the service stubs the same way they consume any other library, which is what the resume line about Protobuf and gRPC integration refers to, and it was essential to the reactor approach rather than incidental to it.