Why does a busy server fall behind?

The trapThe counter is not even full yet — so why are customers already waiting a long time in line?

Move it yourself

0/84 stepSpeedutilization (%)

servingon the phoneat the vaultfree

0.0s · queue 0 · serving 0 · done 0 — nobody has arrived yet

Follow the sentences

1/47

Text mode advances to the next sentence every 3 seconds.

Picture it this way

There is one counter, and serving a customer takes six seconds. So it can take up to ten customers a minute.

Nine customers a minute actually turn up. One customer's worth of room is left over, so the counter is busy nine minutes out of ten — that is what 90% utilization means.

But customers do not arrive on a metronome. One minute brings twelve, the next brings six. The line that forms during a clump has only 10% of slack to drain into, so it lingers.

Move the utilization above from 50 to 90. Same counter, same six seconds — only the line gets longer.

Now the numbers

measured · computed just now

Run it long at 90% utilization and the average customer stands in line for 57.0 seconds — 9.5 times the 6 seconds the counter actually needs, spent doing nothing at all.

Change nothing but the utilization and the waiting climbs like this: 1.0x the service time at 50%, 3.8x at 80%, 9.5x at 90% and 21.6x at 95%. The last step let in about five percent more customers, and more than doubled the wait.

The line follows. At its longest, 50% utilization held 15 people and 90% held 81 (the picture above is a typical five minutes of that long run, so the line there is much shorter than these peaks). The counter never broke down, and never slowed.

This curve has a name: it is the waiting formula for a single-counter queue (M/M/1). Waiting = service time × utilization ÷ (1 − utilization) — at 50, 80, 90 and 95% that is 1x, 4x, 9x and 19x, and it holds when arrivals clump and service times vary from customer to customer. The measurements above track that line, which is why you do not squeeze a server toward 100% busy.

These numbers are not written into the sentence. They come from a simulation run while this page rendered.

Try it in the simulator

1.Open the 'three-tier basics' preset and run it for 10 seconds — at the default 1,000 RPS (requests per second) the Spring (Tomcat) card sits at 63% CPU, with room to spare.
2.Raise the client RPS to 1,500 and that card reads 91% CPU, still at a 100% success rate — the counter is busy, but the line is still being absorbed.
3.Now 1,700: CPU 99%, and p99 — what the slowest one customer in a hundred waits — is 291ms. Thirteen percent more requests bought a far larger jump in waiting: this chapter's curve, in the simulator.
4.At 2,000 the accept queue (the waiting room in front of the counter) overflows, queue-full drops begin, and the success rate falls to 89.2% — a line so long that people are being turned away.

As you raise the RPS, watch p99 move earlier and far more sharply than the CPU gauge — that is this chapter's curve, on screen.
There is a point where a small extra push collapses the success rate. That point is where the slack ran out, and it is the line you defend in production.

Open it in the simulator →

In one sentence

Lines grow not because the counter is slow, but because arrivals clump and the counter has no slack left.

Progress is stored in this browser only (no account, no sync between devices).

Quick check

0 / 3 correct

1.Why does waiting time climb so sharply at 90% utilization?
2.You raise utilization from 90% to 95%. What happens to the average wait?
3.When a queue starts building, what do you look at first?

New words here

utilization: The share of time the counter is busy. 90% means nine times out of ten it is already with a customer.
queue: The line waiting for a turn. Servers have one in front of them too, and there is a limit to how many can stand in it.
waiting time: Time spent standing in line. A different number from service time, which is how long the counter itself takes.