Streaming, Queues, and the Death of the Request-Response Mindset

Web engineering spent twenty years assuming that a request finishes. Requests that take more than a few seconds were pathological, something to be optimised away. Then a category of features arrived where the work genuinely takes a minute or more, and the old shape started failing in familiar ways: gateway timeouts, duplicate work on retry, lost results when a user navigates away.

Separate accepting work from doing it

The correct shape is the one job systems have used forever. The request validates input, writes a task record, and returns an identifier immediately. A worker picks up the task and does the slow part. The client subscribes to progress. Nothing about this is novel; what is new is how many product features now need it.

Once you make this split, several problems solve themselves. Timeouts stop mattering. A user closing the tab no longer cancels the work. Retries become safe because the task row is the source of truth, and a second attempt updates the same record instead of starting a parallel one.

Choose your transport for the failure you can tolerate

• Polling is the least elegant and the most robust. It survives proxies, mobile networks, and sleeping devices. Start here.
• Server-sent events are ideal for one-directional token streams and degrade to reconnection cleanly.
• WebSockets earn their complexity only when you need genuinely bidirectional, low-latency traffic.

A pattern that works well is streaming for the live experience plus a durable task record as the fallback, so a dropped connection means the user reloads and sees the finished result rather than losing it.

Idempotency is not optional

Any long operation will be retried, by your infrastructure, by an impatient user, or by a mobile client that lost its connection at second eighty-nine. Give every submission a client-generated key, store it, and make a repeat submission return the original task instead of starting a new one. Without this, the visible symptom is duplicate charges, duplicate emails, and duplicate records, and the invisible symptom is a bill you cannot explain.

Show the pipeline, not a spinner

An indeterminate spinner for ninety seconds reads as broken. A list of steps where each completes in sequence reads as work being done, even when the total time is identical. Name the steps in the user's language: reading the document, finding relevant sections, drafting, checking. Perceived speed is mostly a legibility problem.

Users forgive slow far more readily than they forgive uncertain. Progress that can be read is progress that can be waited on.

What to instrument

Track queue depth, time to first byte of useful output, total task duration at the ninety-fifth percentile, and abandonment rate by step. That last one is the product metric hiding inside an infrastructure dashboard: it tells you exactly where the wait became intolerable.