A gateway in front of the fleet, and why it has a circuit breaker

Once you’re running a handful of small services, a question arrives that you can’t dodge: how does anything actually reach them? The lazy answer is to expose each service directly and let callers keep a list. That works until the list is wrong, or one service is having a bad day and takes its callers down with it, or you need to add auth and realise you’d have to add it a dozen times.

So you put a gateway in front. A single front door that knows where everything lives, handles auth once, and, crucially, protects the fleet from itself. The last part is the one people skip, and it’s the one that earns its keep at 3am.

What the gateway is actually for

Strip away the buzzwords and a gateway does a few boring jobs:

• One address. Callers talk to the gateway, not to individual services. The

registry of what-lives-where is the gateway’s problem, not everyone else’s.

• Auth in one place. Authenticate at the door once, instead of

reimplementing it in every service and getting it subtly wrong in three of them.

• Rate limiting. A per-caller ceiling so one enthusiastic client can’t

starve everyone else.

• Health awareness. The gateway knows which services are healthy and stops

sending traffic to the ones that aren’t.

None of that is novel. All of it is the difference between a fleet that degrades gracefully and one that falls over in a cascade.

The circuit breaker is the part that matters

Here is the failure I’ve watched happen more than once. One downstream service gets slow, not dead, just slow. Requests to it start piling up. Each one holds a connection and a bit of memory while it waits. The waiting requests stack up faster than they drain. Now the gateway itself is out of capacity, so requests to the eleven healthy services also start failing. One slow service has taken down the whole fleet, and it did it through the shared front door you added for convenience.

A circuit breaker stops exactly this. The pattern is simple and old:

• Closed: traffic flows normally. The breaker counts failures.
• Open: too many failures in a window, so the breaker trips. Requests to

that service fail immediately instead of waiting. This is the key move: fast failure frees the resources that slow failure hoards.

• Half-open: after a cooldown, let a trickle of requests through. If they

succeed, close the breaker and resume. If they fail, open it again.

# the shape of it, per downstream service
if breaker.state == "open":
    if time.monotonic() < breaker.retry_at:
        raise ServiceUnavailable(service)   # fail fast, hold nothing
    breaker.state = "half_open"

try:
    resp = call(service, request, timeout=SHORT)
    breaker.record_success()
    return resp
except (Timeout, UpstreamError):
    breaker.record_failure()
    if breaker.failures >= THRESHOLD:
        breaker.trip(cooldown=COOLDOWN)   # open, set retry_at
    raise

The counterintuitive bit for people new to this: failing fast is the merciful option. A request that fails in a millisecond returns its resources instantly. A request that hangs for thirty seconds waiting on a sick service is holding the door open for the pile-up. When a downstream is in trouble, the kindest thing the gateway can do is stop sending it work and say so immediately.

Short timeouts are non-negotiable

A circuit breaker with a long timeout is a contradiction. If you let a call wait thirty seconds before you count it as a failure, the pile-up wins before the breaker ever trips. Every downstream call gets an aggressive timeout, sized to what a healthy response actually takes plus a margin, not to some comfortable round number. The breaker and the timeout are one control, not two.

Health checks feed the same system

The gateway polls each service’s health endpoint on a cheap interval and uses the result to route. A service that’s failing its health check gets taken out of rotation before real traffic finds out the hard way. Combined with the breaker, you get two layers: proactive (health checks route around known-sick services) and reactive (the breaker catches services that go bad between checks).

The boring conclusion

A gateway is not glamorous. It’s a front door with a bouncer and a fuse box. But the fuse box is why one slow service stays one slow service instead of becoming a fleet-wide outage. Put auth and rate limiting in one place, give every downstream call a short timeout, wrap each downstream in a circuit breaker that fails fast when things go wrong, and let health checks route around the obvious problems. The whole design exists to make failures small and local, which is the only kind of failure you can sleep through.