If you are building anything on top of a large language model, there is a wall waiting for you at roughly sixty seconds. It does not matter how well your code is written. One day a user uploads a document, the request sits there while the model thinks, and then the browser shows the least helpful message in computing: “could not connect to server.” Nothing in your application logs looks wrong. The work was still running. Something in the middle gave up.
The failure that is not your fault
The first instinct is to blame the application, because that is the part you wrote. In this case the application was fine. The real culprit was a load balancer sitting in front of it, quietly enforcing a default idle timeout of sixty seconds.
Here is the sequence. A user submits a document. The request handler makes a synchronous call to a language model to analyse it. That analysis takes around eighty seconds. For the whole of that time the original connection is held open, waiting. The load balancer watches that connection, sees no bytes moving across it, decides it has gone stale, and cuts it at the sixty second mark. The model returns its answer twenty seconds later to a connection that no longer exists.
The reason this is so confusing to debug is that every individual piece is behaving correctly. The model is not slow, it is doing genuine work. The application is not crashing. The load balancer is doing exactly what an idle timeout is designed to do. The mistake is architectural: you asked one connection to stay alive longer than the network in front of it is willing to allow.
The mitigation that buys time but does not fix anything
The quickest response is to raise the load balancer idle timeout. Push it to a hundred and twenty seconds and the eighty second analysis fits comfortably inside the window, and the errors stop.
Be honest with yourself about what this is. It is headroom, not a fix. You are still holding a connection open for the entire duration of a slow operation, and you are still one unusually slow response away from the exact same failure. The next model that takes a hundred and thirty seconds, or the next request that queues behind three others, walks straight back into the wall. Raising the timeout is the right thing to do in the first hour of an incident. It is the wrong thing to still be relying on a month later.
The durable fix: submit, then poll
The pattern that actually removes the problem is to stop making the client wait on the slow work at all. Split the one long request into a fast submission and a series of quick status checks.
- The client posts the document. The server accepts it, creates a job with an identifier, and returns immediately with a 202 Accepted response and that job identifier. This happens in milliseconds.
- The heavy work, the call to the language model, runs in a background task rather than inside the request that the browser is waiting on.
- The client polls a status endpoint using the job identifier. The status moves from processing to complete or failed. When it reads complete, it fetches the result and renders it.
No individual connection is ever held open for more than a moment, so there is no idle connection for a load balancer to sever. The slow operation is still slow, but slowness is now expressed as “the job is still processing,” which is a state your interface can show honestly, rather than a dropped connection, which it cannot.
If your data model already tracks a status field on the item being processed, you may not need any schema change at all. You are already storing the thing that tells the client whether to keep polling.
The edge case worth closing before it bites
There is one honest gap in the pattern above, and it is better to name it than to discover it in production. A background task runs inside a process. If that process restarts while an analysis is in flight, because of a deploy, a crash, or the platform moving your container, the background work vanishes. The job identifier still exists, the status field still says processing, and it will say processing forever. The client polls a job that no longer has anyone working on it.
The fix is a stale reaper. Record when each job entered the processing state. Run a small periodic check that looks for jobs that have been processing longer than any real analysis could plausibly take, and flip them to failed. Now a lost background task surfaces as an honest failure the user can retry, instead of a spinner that never resolves. It is a few lines of code and it is the difference between a pattern that mostly works and one you can actually trust.
The takeaway
Language model calls are slow in a way that ordinary web requests are not, and the infrastructure between your user and your code was built with ordinary requests in mind. Any time you find yourself holding a connection open while you wait on a model, assume something in the middle has a timeout shorter than your patience. Submit the work, return a job identifier, poll for the result, and reap the jobs that get lost. Do that and the sixty second wall stops being your problem.
Leave a Reply