Testing Embedding Models for Legal Retrieval

A legal retrieval test returned text that looked relevant but came from the wrong clause type. Shared words such as party, agreement, vendor, and liability were hiding a bad match.

Write down what a match should contain

I wrote test cases with a query, expected document and section, acceptable chunk IDs, and known false positives. One example asked for the confidentiality clause governing third-party disclosure. Vendor-pricing text mentioned confidentiality but did not define that obligation.

Acceptable neighboring chunks allowed for a clause that crossed a chunk boundary. That made the expected result specific without requiring one arbitrary chunk ID for every passage.

Compare the retrieval results

I kept the cleaned corpus and queries fixed for the embedding comparison. The scorer checked whether the returned set contained the expected document, whether it included an acceptable chunk, and how many results came from rejected documents. It also recorded the first result.

A hit anywhere in the returned set counted as a hit. The test cases specified a top-five retrieval, but the scorer itself did not enforce that cutoff, measure ranking quality, compare the section field directly, or validate a generated answer. Reviewing the first result and the false positives kept those pass flags from hiding the failure that prompted the test.

Treat replacement as an index change

A query vector must use the same embedding model as the stored vectors it searches. Replacing the model requires rebuilding the corpus in a separate index and pointing queries at that model-index pair. Matching dimensions alone would not make two models' vectors interchangeable.

new index
→ re-embed cleaned chunks
→ pair query model with index
→ run legal eval
→ switch or roll back

This example defines the checks without reporting a winning model. Missed clauses and rejected matches are the results to inspect before deciding to switch indexes.