The Event Loop Is a Queue

The event loop is easy to mythologize because it sounds like a loophole in physics.

One thread. Many connections. Non-blocking I/O. Fast enough to make the old thread-per-connection diagrams look heavy and Victorian.

The myth is not that event loops work. They do. The myth is that async I/O turns all waiting into parallelism. It does not. It moves some waiting somewhere else: into the kernel, into a poller, into a worker pool, into a database, into another machine. When the operation is ready to resume, the callback comes back to a single place and asks for time.

That place is the queue.

Node’s own event-loop guide says the quiet part clearly: Node performs non-blocking I/O even though a single JavaScript thread is used by default, in part by offloading operations to the system kernel when possible.¹ When those operations complete, callbacks are queued for eventual execution. The same guide describes event-loop phases as FIFO queues of callbacks.¹ The libuv design document is even more direct: the I/O loop is tied to a single thread, network I/O is performed on non-blocking sockets, and callbacks are fired from that loop.²

So the useful mental model is not:

async = everything is parallel

It is:

async I/O = waiting can be multiplexed
callback CPU = ready work still serializes

This post is about that second line.

A Calm Service With One Bad Callback

Imagine a server whose usual request path is boring:

accept a request;
wait for network or database I/O;
run a small JavaScript callback;
send the response.

Most of the time, step 2 dominates elapsed time, but it does not occupy the event loop. Step 3 is tiny, but it does. That inversion is where tail latency gets unintuitive. A 35 ms I/O wait can be harmless to the loop. A 45 ms CPU callback can block everyone queued behind it.

The lab below simulates that distinction. Requests arrive, wait for synthetic I/O, and then become ready callbacks. Ordinary callbacks are short. A small fraction are slow CPU callbacks. You can queue everything, shed work when the ready-callback backlog crosses a cap, or move the slow CPU portion to a worker pool.

Arrivals: 150 / s Mean I/O wait: 35 ms Mean callback CPU: 2.4 ms Slow callbacks: 4% Slow CPU: 48 ms Backlog cap: 90 CPU workers: off Admission policy: Seed: 17

Backlog / normal callbacks Delay / latency Slow inline CPU Backlog cap Worker offload Shed work

Deterministic synthetic experiment. I/O wait becomes a ready callback later; it does not consume event-loop CPU. Callback CPU and worker-completion callbacks do consume the event loop. Shed mode drops new ready callbacks when the simulated event-loop backlog has reached the cap.

The default run is already enough to be uncomfortable. The offered CPU load is not absurd. Most callbacks are small. The mean I/O wait is much larger than the mean callback CPU. Still, rare slow callbacks create a visible ready-callback backlog, and the p99 event-loop delay climbs into human-noticeable territory.

Set “Slow callbacks” to zero and the queue calms down. Bring it back. Then set “CPU workers” to three. The p99 loop delay falls sharply because the slow CPU portion no longer monopolizes the loop. Now raise arrivals and switch “Admission policy” to “shed at cap.” You will see the essential trade: accepted requests get a bounded queue, but some requests are explicitly refused. The system did not become infinitely faster. It became honest about overload.

That is the entire thesis in miniature:

without backpressure, overload becomes waiting
with backpressure, overload becomes a product decision

Nonblocking Does Not Mean Non-CPU

The phrase “do not block the event loop” is often heard as a style rule. It is more severe than that. Node’s guide says that a server stays speedy when the work associated with each client at any given time is small, and that a blocked event-loop thread cannot handle requests from other clients.³

The part worth underlining is other clients. A callback is not only spending its own budget. It is spending the scheduling budget of every ready callback behind it.

This is why a function can be locally innocent and globally rude.

app.get("/search", async (req, res) => {
  const rows = await db.query(req.query);
  const page = render(rows);      // synchronous CPU
  res.send(page);
});

The database wait may be multiplexed. The rendering is not, unless render is itself broken into small pieces or moved away from the loop. JSON parsing, schema validation, compression, template rendering, regex backtracking, crypto, image transforms, giant array operations, and accidental quadratic loops all have the same shape. They look like normal code until one input makes them large.

The event loop does not care that the work began after an await. Once the callback resumes, the loop is occupied until that turn finishes.

The Old Lesson Was Explicit Queues

The staged event-driven architecture paper is from 2001, but its overload lesson still feels modern. SEDA structures applications as event-driven stages connected by explicit queues; that lets the service inspect load, tune resource use, batch events, and shed work when demand exceeds capacity.⁴

The important word is explicit. If the queue is visible, it can have policy. If it is invisible, it becomes an accident:

a growing in-memory array;
a socket buffer nobody is watching;
a promise chain that keeps allocating;
a database connection pool with no useful deadline;
an event loop whose delay is only noticed after users complain.

An unbounded queue is not kindness. It is a way to turn an overload incident into a latency incident, then into a memory incident, then into a restart.

Backpressure is the opposite bargain. It says:

there is a limit to how much unfinished work we will admit

That limit can be expressed as load shedding, 503s, Retry-After, pausing reads, bounded queues, per-tenant concurrency, deadlines, rate limits, or refusing to start optional work. The form depends on the product. The principle is the same: work that cannot be served within a useful budget should not silently become infinite waiting.

The C10K Ghost

The C10K problem was never just “can one machine keep ten thousand sockets open?” Dan Kegel’s old survey is full of concrete blocking hazards: if a nonthreaded server hits disk I/O that blocks, all clients wait; asynchronous I/O or worker threads can route around that specific bottleneck.⁵

That distinction matters because it keeps the event-loop story from becoming religious. Threads are not evil. Events are not magic. Lauer and Needham’s classic duality paper argued that message-oriented and procedure-oriented systems can be duals under comparable scheduling strategies, with the substrate often deciding which model is more natural.⁶ The real question is not “events or threads?” It is:

where are the queues, who schedules them, and what happens at saturation?

Worker threads are one answer for CPU-heavy work, but they do not delete the question. They move the queue. In the lab, worker offload usually improves event loop delay because the main loop only pays a small submit callback and a small completion callback. Raise the slow-callback fraction high enough, though, and the worker p95 wait starts to matter. The bottleneck has moved from the loop to the worker pool.

That can still be a win. It is just not a free lunch.

What To Measure

CPU utilization can be misleading for event-loop services. A process can have plenty of idle CPU and still have a sick loop if one synchronous operation is holding the turn. Node’s perf_hooks.monitorEventLoopDelay() exists precisely to sample event-loop delay; the docs note that timer delay reflects delay in the libuv event loop lifecycle.⁷

For a production service, I want at least these signals:

event-loop delay percentiles over time;
event-loop utilization, not only process CPU;
callback or handler duration histograms;
queue depth for every explicit pool, stream, and work queue;
admitted, completed, timed-out, canceled, retried, and shed counts;
response latency split into queue wait and service time when possible;
per-route and per-tenant versions of the above, because averages hide bullies.

The split matters. If p99 response latency rises, the service time might be slow. Or the service might be fine but queued behind a blocker. Or I/O might be slow. Or admission might be too generous. The remedy changes with the phase.

Small Turns Are A Fairness Policy

The most practical event-loop rule is also the least glamorous:

make each turn small

Small turns mean a callback does bounded work before yielding. They do not mean “never do CPU work.” They mean CPU work has a budget, a route, and a failure mode.

Some useful patterns:

bound input sizes before parsing, rendering, validating, or matching;
put deadlines around optional work, not only around network calls;
stream large responses instead of constructing one giant object;
move CPU-heavy or variable-cost work to workers or another service;
keep worker pools bounded and observable;
make load shedding cheaper than accepting work;
split long local loops with a scheduler yield only when that preserves correctness and memory bounds;
measure event-loop delay in the same dashboard as request latency.

None of this is Node-specific. Browser UIs, Python asyncio, Ruby reactors, single-threaded game loops, GUI toolkits, Redis-style servers, and any actor system with a hot mailbox all share the same warning. A single executor can hide enormous concurrency in waiting, but ready work still takes turns.

The event loop is a good servant when its turns are small. It is a terrible place to hide an unpriced queue.