“Just parallelize the agents” is a trap when they share a GPU. Measure before you believe it.
lean-coder can dispatch multiple workers, sub-agents that each take a task and run headless. The obvious next thought is: run them at the same time, get the work done faster. We tested that assumption. It was wrong in an instructive way.
The measurement
We dispatched two write-workers (qwen2.5-coder:14b) at the same task, both pointed at the same Ollama host, and compared concurrent vs sequential execution:
| Mode | Wall-clock time |
|---|---|
| Two workers concurrent | 63 seconds |
| Two workers sequential | 37 seconds |
Running them in parallel was 70% slower than running them one after the other.
Why parallel lost
Ollama serializes inference on a single backend by default. Send it two requests at once and they don’t run in parallel, they queue, and they contend for the same compute while they wait. You pay the coordination overhead of concurrency and get none of the benefit, because the underlying resource (one GPU’s compute) was never parallel to begin with.
Two agents “running at the same time” is an illusion when they’re both funneling through one inference engine on one device. They’re taking turns anyway, just less efficiently than if you’d told them to take turns on purpose.
The distinction that actually matters
The useful mental model is to separate compute-bound from I/O-bound work, and to separate one backend from many.
- Compute-bound work on one backend (like two models generating tokens on one GPU): parallelism gives you nothing. Sequential wins. The only thing you gained from “having room for two” was memory headroom, not throughput.
- I/O-bound work (a worker mostly waiting on network, disk, a slow external command): parallelism helps, because the waiting overlaps.
- Work spread across different backends (driver on one host, workers on separate machines or genuinely separate model instances): now fan-out pays, because there’s real independent compute to use.
So the guidance we landed on: fan out across different hosts or models, or for I/O-bound tasks. For compute-bound edits on a single box, run them sequentially and don’t kid yourself.
The general lesson
“Parallelize it” is a reflex borrowed from a world where adding workers adds workers. In local LLM inference, adding workers to one GPU doesn’t add compute, it adds contention. The concurrency knob (worker_max_concurrent and friends) is mostly about isolation, keeping sub-tasks in separate contexts so they don’t pollute each other, not about speed.
If you want real parallelism, give the work real parallel hardware: separate hosts, separate accelerators, or genuinely separate model servers. Otherwise, measure. The intuition that two is faster than one is exactly the kind of thing that’s obvious, appealing, and false.
This is a field note from lean-coder, an open-source, dependency-free terminal coding agent that treats context as the scarce resource it is. Source and docs: github.com/codemonkeying/lean-coder. Licensed MIT.
Leave a Reply