Running Website Analysis as an Async Job

A website-analysis endpoint had to fetch a page, clean its HTML, extract structure, and run several analysis steps. Returning all of that in one response kept the browser waiting without knowing which step had finished.

The endpoint returned a job ID

The Express controller normalized the submitted URL, created an analysis ID, and returned HTTP 202. A 250-millisecond timer then called the processing function. The browser had an ID to subscribe to while the analysis continued.

The response calls the job queued. In this implementation, that means scheduled in the same Node process. The timer releases the HTTP response; it does not create a separate worker or persist the task across a restart.

WebSocket events carried status

The browser subscribed with the analysis ID. The server acknowledged the subscription and replayed the buffered events for that ID. This covered a timing problem: an analysis step could finish before the browser had opened its WebSocket.

The loop replays the buffered events. A reconnect can receive events it has already seen, so the frontend has to apply status updates without appending duplicate results. The buffer lasts only as long as the process that owns it.

Cut page chrome before sending content for analysis

Cheerio removed scripts, styles, navigation, footers, hidden elements, cookie notices, and matching advert or banner elements. The scraper extracted the title, description, headings, links, and cleaned body text rather than passing raw HTML into analysis.

The extraction kept up to 50 absolute HTTP links and 12,000 characters of body text. Its word count came from that shortened text, not necessarily the full page. Those caps limited the payload, though a long page could lose content near the end.

Where this implementation stops

For restart recovery, the job and its results would need persistent storage and a worker queue outside this process.

The controller shows URL normalization, with no fetch-safety checks included. Accepting arbitrary URLs also requires checking private addresses and redirect destinations before making network requests.