On subscription plans the meter that throttles you counts tokens you thought were free. Here’s the trap, and the only lever that actually works.
Prompt caching is one of the great wins of the modern LLM API: re-send the same prefix and the provider charges you a fraction, often a tenth, for the cached part. If your agent re-sends a big history every turn, caching sounds like the obvious fix for cost.
It is a fix for one kind of cost. If you’re on a subscription plan with usage quotas, it may do nothing for the cost that actually stops you working. We learned this the expensive way, chasing a meter that wouldn’t stop climbing.
Two different meters
Paid API usage is measured in dollars. Subscription plans are measured in quota, the rolling meters (a short-window one and a weekly one) that throttle you when you’ve used “too much.” These are not the same number, and they do not count cached tokens the same way.
- Billing (dollars): a cache-read is cheap, ~0.1x. Caching helps a lot.
- Quota (the meter): the short-window bucket discounts cache-reads, as documented, but the weekly “all-models” bucket, due to a confirmed open platform bug, counts cache-read tokens at full weight.
So on the meter that governs your weekly throttle, a cache hit and a cache miss cost roughly the same. The 10x discount you’re getting is entirely in dollars, and entirely absent from the quota that decides whether you can keep working today.
The symptom that exposed it
The tell was a weekly meter racing while the short-window meter stayed calm. The numbers were stark: a well-tuned agent running at 140k, 500k tokens of context moved the weekly meter about 1%. A naive agent, at a mere ~59k tokens, moved it 6% in about ten minutes.
We were smaller and burning the meter faster. Caching was working perfectly , we confirmed it with real two-turn probes; turn two read the cached tokens as expected. Caching was not the problem, and therefore caching was not the solution.
Where the burn actually came from
Three compounding causes, one dominant:
- We re-sent the full, unwindowed history every tool round. Each round’s huge cache-read got counted at full weight by the buggy weekly bucket. This was the dominant, fixable cause.
- A top-tier model default, which carries a heavier quota weight than a mid-tier one (roughly 1.7x input, up to 8x output).
- Thinking blocks re-sent in history, counted at the output rate (5 to 8x).
A well-tuned agent avoids all three: a mid-tier default, a bounded context window so it doesn’t re-send everything, and it strips thinking out of history.
The only lever that matters for quota
Once you accept that cache-reads count full on the meter that throttles you, the conclusion is forced and clarifying:
The only way to cut quota is to reduce the raw token count of every call: tokens sent (history + system + tools) and tokens received (output + thinking).
That reframes a lot of “optimizations”:
- Protecting the cache for quota’s sake is pointless. Busting the cache (say, with a leash or incognito toggle) is a quota non-issue, reads cost ~1x on the meter either way. Don’t spend effort there.
- A bounded context window is the real fix, and it’s a client-side one. There’s no server-side session to escape the resend, the messages API is stateless by design, so the answer is to stop sending the whole history: window it, evict old turns, strip re-sent thinking.
- Model choice is a quota lever, not just a quality one. A mid-tier default can be several times lighter on the meter than a top-tier one for the same work.
The takeaway
Caching and quota optimize on different axes, and conflating them sends you optimizing the wrong one. If you pay in dollars, cache aggressively. If you’re throttled by a subscription quota, caching is a comfort that doesn’t move the meter that matters, and the only thing that does is sending and receiving fewer raw tokens. Measure the meter you’re actually limited by, then optimize that one.
This is a field note from lean-coder, an open-source, dependency-free terminal coding agent that treats context as the scarce resource it is. Source and docs: github.com/codemonkeying/lean-coder. Licensed MIT.
Leave a Reply