Keeping Training and Serving Features Consistent
Training and serving built the same features differently. A shared feature contract and explicit fallback kept incompatible cached vectors out of the prediction path.
A model behaved differently after I moved it from training code into an API. Before changing the model, I compared the feature vectors on both sides.
The same input produced different features
Training and serving had separate copies of the feature logic. One copy handled defaults differently, and the field order had started to drift.
One builder became the contract
I moved the transformations into one shared builder. Training and serving needed the same field order, types, default values, and handling of missing data. A feature version identified the contract used to build a vector.
The model also needed an expected feature version. Otherwise, a cache entry could be fresh and still contain a vector prepared for a different model input.
Reject incompatible cache entries
The recommendation handler reads the cached payload from Redis and checks for a matching version, a timestamp that passes its freshness check, and a non-empty vector. Only after those checks does it call the prediction function.
cached feature payload
→ check version, timestamp, and vector
→ pass: predict recommendations
→ fail: return fallback recommendationsA missing payload, version mismatch, stale timestamp, or empty vector returns a fallback with a separate reason code. This route doesn't rebuild features from the source record. Recomputing them belongs in a separate path; the request has a fallback while the cache cannot be used.
Check the builder as well as the version
The version check compares a label attached to the payload. To check the builder itself, pass a fixed source record through training and serving and compare the vectors, including missing fields and defaults. That fixture test belongs alongside the builder, outside the serving handler shown here.
Have a similar project?
Talk to Me