Keeping Chat Context on the Server

A support chat kept its conversation history in the browser and sent the full transcript with every new message. The API was accepting earlier turns supplied by the client, including their role labels.

The client sent only the new message

I reduced the request to a session ID and the latest user message. The handler loaded recent turns, assembled the prompt, called the model, and passed the new exchange to session storage.

The handler uses the session ID to find history. An ownership check must happen before that load, though it isn't included here. Stored user messages remain user input; their location on the server doesn't turn them into trusted instructions.

session ID + message
→ load server history
→ assemble prompt
→ generate response
→ store exchange

Memory had limits

The memory design used a session key, a six-hour expiry setting, a maximum of ten turns, and an estimated 1,800-token history budget. The trimming helper keeps the latest ten turns, then drops older ones until the estimate fits.

Its estimate divides character length by four. That's a rough sizing check, so prompt assembly still needs to account for the current message and instructions. The stored conversation can contain more history than a particular request should send.

The storage sketch uses an in-process dictionary. Moving it to Redis would allow requests across processes to share history. The six-hour TTL is only a constant here; expiry still needs to be applied by the store.

Handle an empty session and a failed load separately

A missing key returns an empty history in the helper. A storage error raises a memory-load error, which the handler returns as HTTP 409 with a request to restart the conversation. It stops before calling the model.

Those are different states. A new conversation can start without prior turns. A conversation whose history failed to load shouldn't continue as though its earlier messages were available.