Separating I/O and PyTorch Inference in FastAPI

A Flask endpoint fetched metadata, prepared an input, ran PyTorch inference, and wrote the result before returning JSON. The synchronous worker stayed occupied through that whole sequence.

Separate waiting from computation

The database calls spent time waiting for another service. PyTorch spent time computing. Those steps needed different handling if the server was going to coordinate concurrent requests.

validate request
→ fetch metadata
→ run inference
→ write result
→ return response

Declaring the route async would not make synchronous inference yield to the event loop. The function needed an execution path outside it.

Await the database and offload inference

In the FastAPI route, metadata reads and result writes used awaitable database calls. While a call was waiting, the event loop could handle another request. This depended on the database helper doing asynchronous I/O; wrapping a blocking driver in an async function would still block.

The inference function remained synchronous and ran through run_in_threadpool. The route awaited its result while the function executed in a thread. The request still waited for inference to finish; this was not a background job that survived a lost connection.

Validate the outer request before doing either

A Pydantic model defined the required client ID, input dictionary, timestamp, and optional metadata. Requests that failed those rules stopped before the route reached the database or model.

The dictionary still accepted arbitrary values. Model-specific input keys, timestamp recency, and access to the supplied client ID needed application checks beyond those field types.

Follow accepted requests through the route

The handler created a request ID before the metadata lookup and included it in the stored result and response. That connected an accepted request to its output. Validation happened before the handler ran, so a rejected payload would not receive this handler-generated ID.

The thread pool frees the event loop while inference runs. Calls can still compete for the same model and device, so inference concurrency and GPU scheduling need their own limits.