Large Context Windows Still Need Data Engineering

I tested a question against a large folder of documents. The model had enough room for all of them, yet the answer leaned on an older copy and repeated the same section from two files.

Giving the model more room didn't resolve which copy it should read. I needed to decide which sections belonged in the request before assembling the prompt.

Check access before ranking

The access helper checks three fields: the section's tenant must match the user's tenant, its status must be active, and its allowed roles must include the user's role. A relevant passage still has to pass those checks before it can become model context.

The ranking helper sorts by relevance first. When two sections have the same relevance score, it puts the more recently updated one first. That tie-break doesn't guarantee that an obsolete version disappears. Version selection needs its own rule and document metadata.

A short prefix is a rough duplicate check

The fingerprint helper normalizes whitespace, lowercases the text, and takes the first 500 characters. Two passages with that same prefix receive the same comparison key.

Sections can share an opening and differ later, so this key can merge passages that should stay separate. Exact duplicate detection needs the complete normalized text; version selection needs the document's version metadata.

Apply the budget while packing

The example sets an 8,000-token context budget and a 1,200-token limit per source. A packing loop has to count the selected text, enforce both limits, and leave room for the instructions, question, and response within the model's available window.

The helpers stop before packing. The packing loop needs to enforce those settings and retain the chosen document and section IDs with the request. That record connects a generated answer to the passages it received.