Keeping Audio and Video Streams Responsive
Camera frames arrived faster than inference could process them. I kept a short frame buffer and grouped speech before preparing each model request.
The first video test treated the camera as a stream of image uploads. Inference ran slower than capture, so old frames kept piling up while the user had already moved on.
Keep a short view of the camera
I kept a rolling buffer with a maximum of twelve frames and a five-second age limit. The selector could build a new context at most once every two seconds, taking the latest three frames still inside the age limit. It returned nothing until three were available.
The selection interval does not space the three frames two seconds apart. If capture is frequent, they can be close together. The t_minus_4s and t_minus_2s labels in the example are display labels; the capture timestamps determine their age.
Group speech before transcription
The audio path used voice activity detection to decide when to collect speech chunks. An unvoiced chunk closed a segment once it had reached the minimum duration, then reset the collection for the next one.
That grouping reduces continuous audio submissions. It depends on the detector: background noise can still be classified as speech, and quiet speech can be missed. The excerpt shows the segment boundary, not the detector or the handling of an unfinished segment at the end of capture.
Keep the timestamps with the request
The context packet combined the selected frames, a transcript with its time window, and the previous description. Keeping frame and transcript timestamps made it possible to check whether the two described the same part of the session.
The frame buffer had a fixed bound. The packet alone did not enforce transcript length, expire the previous description, or limit requests already waiting for inference. Those needed separate checks at submission so a small capture buffer would not become a growing request queue.
Have a similar project?
Talk to Me