A network link does not tell each sender what rate it deserves.

It drops a packet. It delays an acknowledgment. Maybe it marks a packet with ECN. Most of the time, the sender sees a few delayed, lossy shadows and has to infer the state of a shared bottleneck it cannot directly inspect.

That is the quiet beauty of TCP congestion control. The sender keeps a variable called the congestion window, cwnd, and uses it as a probe:

increase while the path seems healthy
decrease when the path says "too much"

This is not a bandwidth knob in the ordinary sense. It is a sensor and an actuator collapsed into one number.

RFC 5681 defines cwnd as a TCP sender-side state variable limiting how much data can be sent before acknowledgments return. It also standardizes the classic family of algorithms around slow start, congestion avoidance, fast retransmit, and fast recovery.1 The famous sawtooth is not decoration. It is the shape of a distributed system learning capacity through feedback.

The hard part is that feedback can arrive too late.

When Busy Became Broken

Van Jacobson and Michael Karels wrote “Congestion Avoidance and Control” after the early Internet had started to hurt itself.2 Their paper opens with a blunt observation: severe congestion had become common, and gateways could drop a large fraction of incoming packets because of local buffer overflow. The problem was not only bandwidth. It was control.

If senders inject packets faster than the bottleneck can drain them, queues fill. If queues overflow, packets are dropped. If dropped packets cause retransmissions that add still more load, the network can spend scarce capacity carrying work that will not help anyone. That is congestion collapse in the practical sense: the link is busy, but useful work falls.

The fix was not to assign a global traffic cop to every flow. The fix was to make endpoints cautious enough to discover capacity without destroying it.

Two ideas matter here.

First, slow start is not actually slow in the everyday sense. It grows the window quickly when the sender has little information, but it grows from a small initial condition so the sender does not blast a new path with arbitrary load.

Second, congestion avoidance uses additive increase and multiplicative decrease: grow roughly linearly while no congestion signal appears; cut the window sharply when one does. Chiu and Jain’s analysis of increase/decrease algorithms studied this style of control abstractly and found that additive increase with multiplicative decrease has the right convergence flavor for efficiency and fairness under distributed feedback.3

The folklore version is:

+1 packet per RTT
halve on loss

The actual TCP standards and implementations have many details. But that small cartoon contains the core asymmetry: be patient while claiming more capacity, be decisive when the path pushes back.

The mistake is to equate high utilization with health.

A bottleneck queue is not bad by existing. Packet networks need buffers to absorb short bursts. The bad state is a standing queue: a buffer that fills and stays filled. The link reports 100 percent utilization, but every packet now waits behind a pile of older packets. Throughput looks good. Interactive delay does not.

That is the bufferbloat lesson. Nichols and Jacobson describe the problem plainly in their CoDel article: unmanaged buffers have become more critical because buffers are larger, delay-sensitive applications are common, and large streaming downloads are normal.4 Buffers are necessary, but persistently full buffers convert excess capacity claims into delay.

The key timing issue is:

tail drop tells the sender after the buffer is full
delay-based marking tries to speak before the buffer becomes a parking lot

That is not a moral statement about one queue algorithm winning forever. It is about where the feedback lives. If the only signal is overflow, the queue may have already accumulated hundreds of milliseconds of delay before the control loop reacts.

Congestion Lab

The lab below is a deterministic toy, not a TCP stack. Each flow has a congestion window measured in packets. Each round is one base RTT. A shared bottleneck drains a fixed number of packets per round. If senders offer more than the bottleneck can drain, the queue grows.

The three feedback modes are intentionally simple:

  • Tail drop: signal congestion only when the buffer overflows; all flows cut together in this toy.
  • Delay mark: signal selected flows when queue delay exceeds a target.
  • No signal: windows keep increasing.

The toy is biased toward clarity. Real TCP observes ACK clocks, delayed ACKs, SACK, retransmission timers, pacing, receiver windows, application stalls, offload behavior, ECN, and many implementation details. The lab isolates the control problem: what happens when the congestion signal arrives early, late, or not at all?

I audited the simulator before trusting the story. Across 6,561 combinations of feedback mode, flow count, capacity, buffer, base RTT, delay target, decrease factor, and initial skew, the queue stayed bounded by the configured buffer, utilization stayed in [0, 1], fairness and loss shares stayed finite and bounded, and every summary value remained numeric.

The browser asset exports that audit:

const lab = require("./assets/js/congestion-window-lab.js");
const audit = lab.auditCongestionWindowLab();
if (!audit.passed) throw new Error("audit failed");
Flow windows Queue delay Delay target Tail drop Delay mark

Deterministic toy model, one round per base RTT. It is not packet-level TCP; it is a control-loop microscope for additive increase, multiplicative decrease, standing queues, and synchronized congestion signals.

Start with the default Tail drop setting. The utilization metric is excellent: 100%. That is the trap. The p95 queue delay is 238.9 ms, the average queue delay is 184.7 ms, and every congestion event is synchronized. All flows cut together, then climb together, then hit the wall together.

Switch to Delay mark. The queue-delay panel should fall sharply. Utilization usually remains high, though not always perfectly full in this stripped-down model. In the default comparison it runs at 91.8% utilization with p95 queue delay near 41.4 ms. The important difference is that the control loop receives information before the buffer is exhausted. The queue stops being the only way to discover overload.

Finally choose No signal. The link stays busy, but the sender side keeps inventing work the bottleneck cannot use. In the default run, discarded work rises to 77.9% of what senders tried to inject. That is the “busy is not healthy” lesson in its simplest form.

Why AIMD Has Teeth

The Chiu-Jain picture is useful because it separates two goals that are easy to blend together.3

The first goal is efficiency: use the available bottleneck capacity. The second is fairness: flows sharing the same bottleneck should converge toward reasonable shares. Additive increase lets flows climb toward unused capacity without one flow’s current size becoming the main driver of its future claim. Multiplicative decrease makes larger windows give back more absolute capacity when congestion appears.

This is not charity. It is geometry. In a two-flow diagram, additive increase moves the allocation roughly toward higher total load; multiplicative decrease moves it back along a ray through the origin. Repeating that motion nudges the system toward the fairness line while keeping the aggregate near capacity.

That is the elegant part.

The inelegant part is that the signal can be wrong, late, or synchronized. A wireless loss may not mean congestion. A huge queue may hide congestion until delay is already awful. Tail drop can make many flows halve together, creating a sawtooth for the whole link instead of independent small corrections.

The control law is simple. The measurement channel is not.

What the Queue Knows

Think of a bottleneck queue as state that endpoints can feel only indirectly.

If queue delay is small, the path has slack or at least is not persistently backlogged. If queue delay grows, offered load is exceeding drain rate. If the queue overflows, the path is screaming after it has already accumulated delay.

CoDel’s design tries to control persistent queueing delay rather than raw queue length.4 That distinction matters because packet counts are not delay by themselves. A queue of 100 packets means something different on a slow uplink than on a fast data-center link. Delay is the user-visible symptom.

This is also why “just add buffer” is an attractive trap.

More buffer can reduce drops for short bursts. It can also delay the congestion signal. A sender that waits for overflow will happily fill the new space. The network does not become less congested. It becomes more patient before admitting congestion.

That is a strange kind of politeness. It makes the dashboard look calmer and the call feel worse.

A Systems Checklist

For any congestion-like control loop, not only TCP, I would ask:

  • What state variable limits offered work?
  • What feedback tells the sender to increase or decrease?
  • Is the feedback loss, delay, explicit marking, queue depth, or application latency?
  • How delayed is the feedback relative to the action?
  • Do many senders receive the same signal at once?
  • Does the decrease rule give back capacity proportional to current demand?
  • Is the queue absorbing bursts, or hiding persistent overload?
  • Are we measuring useful throughput, offered work, loss, and delay separately?

That checklist applies to API rate limiters, worker pools, database admission control, retry budgets, model-serving concurrency, and human review queues. A shared resource rarely announces the correct rate. Clients infer it from pain.

The lesson from TCP is not “halve everything forever.” The lesson is that a good distributed control loop needs a state variable, a feedback signal, and humility about delay.

The congestion window is how a sender asks the network a question.

The queue is how the network answers when we wait too long to listen.

  1. Mark Allman, Vern Paxson, and Ethan Blanton, “RFC 5681: TCP Congestion Control”, IETF, 2009. ↩

  2. Van Jacobson and Michael J. Karels, “Congestion Avoidance and Control”, ACM SIGCOMM, 1988. ↩

  3. Dah-Ming Chiu and Raj Jain, “Analysis of the Increase and Decrease Algorithms for Congestion Avoidance in Computer Networks”, Computer Networks and ISDN Systems 17(1), 1989. DOI metadata: 10.1016/0169-7552(89)90019-6. ↩ ↩2

  4. Kathleen Nichols and Van Jacobson, “Controlling Queue Delay”, ACM Queue, 2012. See also the Communications of the ACM version: 10.1145/2209249.2209264. ↩ ↩2