Putting a Gateway in Front of a Hundred ComfyUI Boxes: Two Days of Lessons
I recently built a video-generation gateway in front of a fleet of self-hosted ComfyUI machines. Twenty-odd boxes at launch, a hundred-plus two days later, and real traffic taught me a few things along the way. Here’s the story; some of the pits are fairly typical.
The first approach, and why it didn’t hold up
The original path was as straightforward as it gets: request comes in, pick a machine, POST /prompt, get a prompt_id, write it to the DB, and let a poller check for results every 15 seconds. ComfyUI has its own queue, so if several tasks arrive it just lines them up.
After running it for a while, a few things felt off:
- The queue lives on the machine, and the gateway can’t see it. When a machine dies, every task queued on it dies with it — while the gateway still thinks they’re running.
- Submit latency depends on the upstream. There’s a prompt-rewriting step in front of it that takes a few seconds, and the client has to wait through that.
- Two requests arriving at the same time both probe the same machine as “idle” and both land on it.
So the queue moved into the gateway. Submit now only validates, charges, and writes a QUEUED row, returning a task_id immediately; a background dispatcher periodically hands QUEUED tasks to idle machines; the poller is untouched and only deals with what happens after IN_PROGRESS.
How “idle” is decided: the gateway’s own DB shows no in-flight task on that machine, and a probe of its /queue comes back empty. The first condition covers what we dispatched ourselves; the second covers anything someone pushed in by hand.
What happens when the queue is full: the threshold is allowed queue depth per machine × machines currently online. If every machine is offline the threshold is zero and we return 429 right away instead of letting tasks rot.

The dispatcher doesn’t roll its own goroutine plus Redis lock; it hangs off the project’s existing distributed task framework, whose lease guarantees only one instance runs it. That decision saved a lot of trouble later.
Three problems I caught reviewing my own design
After writing the design doc and before touching code, I read it through once more and found three things:
Refunds could happen twice. Every step after the pre-charge can fail, and a failure has to refund. Once the task is in the DB, the background dispatcher also refunds on failure or timeout. With both paths live, any chance of “returning an error after the insert succeeded” means paying back twice. The fix is to make the DB insert the very last fallible step after the pre-charge — for example, moving the pure “project the response into the client format” function ahead of the insert. From there on every state transition goes through a CAS, and refunds only happen after winning it.

The dispatch interval was a lie. I’d given the dispatcher a 5-second interval setting, but the task framework’s scheduler itself only scans every 15 seconds, so that 5 was merely a “no sooner than”. The setting went away and the doc now says “dispatch latency ≤ 15s”.
Serial POSTs were too slow. With a few seconds per upstream call, dispatching to a dozen idle machines one after another took tens of seconds. It became serial pairing (maintaining a per-round “already taken” table) followed by concurrent POSTs.
Implementation: where the plan was wrong
Each task went to a separate agent to implement, then to another agent to review against the code. I wrote the plan, and several details written from memory were corrected during implementation:
- The error types I referenced lived in a different package.
- The dispatcher built an HTTP request with a
nilbody, while the downstream send function unconditionally callsBody.Close()— a guaranteed panic on every dispatch.http.NoBodyfixed it. - A set of status constants were untyped strings, so
assert.Equalagainst the typed field could never match.
Review caught two more serious ones, both logic errors in the plan itself:
- When the upstream had accepted a task but the local DB write failed, I recorded only the
prompt_id, not which machine took it. On the next round the task got re-paired, the machine actually running it probed as “busy” and got skipped, and the task was bound to a different idle machine — polling could never find the result. Now the machine id is recorded too and these tasks go straight back to it. - A task no machine could serve (say the model had been removed from config) went down the same branch as “all machines busy”, sat at the head of the queue, and everything behind it waited forever. The two cases are now separate; a task with nowhere to go fails and refunds immediately.
False greens in the tests
Two tests passed while testing nothing:
- The test process never ran environment init, a query-limit constant was 0, the SQL became
LIMIT 0, and the loop never executed. - A field had a default fallback (empty upstream id falls back to the public id), so the “empty upstream id” branch was unreachable.
The habit that stuck: delete the branch under test and run again to see whether the test goes red.
After launch
Busy machines flagged offline
Past a hundred machines with thirty or forty of them rendering, the dashboard kept showing two or three as “offline”. They were all producing video normally.
The probe was too crude: one shot at /queue, 2-second timeout, timeout means offline. ComfyUI’s main loop is busy while rendering, and /queue occasionally takes two or three seconds to answer. With few machines and no load before launch, this never showed.
It became a snapshot with history: a node is offline only after N consecutive failures or T seconds without a success, one success brings it back, and a failure keeps the last known queue numbers. Timeout relaxed to 5 seconds. The flapping disappeared.

Two probes stalling each other
The dashboard probed /queue and /system_stats (for VRAM) on each machine at the same time. ComfyUI’s HTTP server is single-threaded; /system_stats queries the GPU and gets stuck behind a CUDA sync while rendering — so /queue on the same machine queues up behind it, times out, and the machine is flagged offline.
The two probes became sequential, VRAM (which changes slowly) is refreshed every 30 seconds, and it moved off the request path into a background refresh.
Too many probes
At that point every dashboard refresh, every submit, and every dispatcher round probed every machine independently. Three sources, no authoritative state, and traffic scaling with the number of open dashboards and the submit rate.
It all collapsed into one background task maintaining a single node-state snapshot (in memory, mirrored to Redis for multi-instance). The dashboard, the 429 check, and dispatch candidate selection only read the snapshot. Before dispatching, the chosen machine gets one live confirmation probe. Total probe volume is fixed at “machines / interval”, and the dashboard went to milliseconds.

Along the way, a piece of information that had gone unused: for tasks the gateway dispatched itself, the DB already says exactly which machine they’re on. Those machines don’t need probing at all.
One wrong guess
Someone reported the query endpoint “seems to be rate-limited too”. Without looking at the logs, I decided from memory that “the client is using a different URL form, the route doesn’t match, and it’s falling into the static-asset rate limit” — and committed a fix. They sent over a log excerpt: every request in the normal form, all 200, two milliseconds. Reverted.
It turned out to be the gateway’s own per-IP rate limiting — one tier each for web assets, admin API, and login, with the login tier defaulting to 20 requests per 3 minutes. Refreshing the page repeatedly and then hammering the login button hit it. Nothing to do with the business logic.
The pipeline
With one docker build doing everything, the CI daemon is fresh every run, so frontend deps, Go modules, the build cache, and the apt layer all start from scratch. Split into three steps: frontend build and Go compile run as pipeline stages with dependency and build caches on persistent volumes; the image stage only assembles the runtime layer, with BuildKit layer cache pushed to the registry. On a warm node the frontend full build takes about a minute, Go incremental compile a few seconds, the image a few seconds.
What’s next
ComfyUI has a WebSocket that pushes queue status, execution progress, and completion events. Wire it up and “online” becomes “is the connection alive”, completion no longer waits for a poll, and you get a progress bar. The snapshot design leaves room for it: one more source writing into the same snapshot, consumers untouched. First, though, I need to confirm whether the upstream wrapper passes client_id through — otherwise the per-client events never arrive.
Looking back, most of these two days weren’t spent writing code. They were spent figuring out one thing: whether a machine was actually offline.