Streaming Text Generation With Server-Sent Events

A text-generation endpoint waited for the complete response before sending anything to the browser. Longer requests could hit the gateway timeout while the generation process kept going.

Send chunks as they become available

I replaced the single JSON response with Server-Sent Events. The endpoint could send text chunks while generation continued, followed by a completion event or an error the browser could handle.

validated prompt
→ generation iterator
→ SSE chunks
→ complete, fail, or disconnect

The event names gave the frontend a way to distinguish an unfinished response from a completed one. If the stream ended without a completion event, the partial text should remain marked incomplete.

When a client disconnects

The loop below checks for a disconnect after the generation iterator has supplied a chunk. On that path, it records cancellation metadata and prepares a cancelled event. The check does not interrupt an iterator that is still waiting for its next chunk.

The local stream and the generation provider have separate lifetimes. Stopping work requires leaving the loop, closing the iterator, and cancelling the upstream request where supported. A cancelled event written to a disconnected connection is not an acknowledgement from the browser.

Log the request state without copying the text

The displayed log record contains the request ID, prompt size, estimated output-token count, duration, model version, validation result, and cancellation reason. It calculates sizes from the text without placing the prompt or generated response in that record.

That record supports investigating an interrupted request without copying its content into logs. Apply the same restriction to other log calls and error handlers.

SSE changes how the response arrives. It does not remove hosting timeouts or guarantee that a proxy forwards each chunk immediately. The deployed path needs buffering and idle-timeout checks, and a stalled provider still needs a generation deadline.