Discovering the wait — IO idles the CPU

A line that queries a database — how much CPU does it use while it waits for the answer? A customer who has gone to the vault still occupies the counter, but no work is happening at it.

See this first in FoundationsWhy is one fetch instant and another slow?

So far our code stayed inside the CPU. Real programs read files, receive network data, query databases — collectively, IO. The point of this chapter is simple: IO is astronomically slow by CPU standards, and while it happens your code is simply stopped.

① How slow, in numbers

An instruction takes about a nanosecond; a memory fetch about 100ns; an SSD read about 100µs; a network round trip milliseconds. Scale a CPU instruction to one second and the SSD read takes more than a day, the network reply a few months. That gap is the starting point of this whole series.

② What blocking means

Call db.query() and it does not return until the result arrives, so the next line cannot run. That is blocking. In the animation the PC sits pinned to one line while the CPU indicator switches to "idle".

③ Waiting still holds memory

Remember chapter 4: held references cannot be reclaimed. A waiting function keeps its stack frame, and its variables keep pointing at heap data. So waiting code uses no CPU but does use memory — a combination that causes real incidents later.

④ And the CPU?

This animation follows one program, so the CPU shows as idle. On a real machine it is not: the OS immediately puts other work on it. That is chapters 5 and 6.

⑤ How slow, in numbers

One CPU instruction takes about a nanosecond. A memory access is around 100 ns, an SSD read tens of microseconds, a network round trip inside one datacenter about 0.5 ms, and across regions tens of milliseconds. In CPU terms, one disk access is tens of thousands of instructions and one remote call is tens of millions. That gap is where this entire series starts.

⑥ So what should you optimise?

When waiting dominates, making the computation faster barely moves the total. Overlapping the waits does. Ten requests waited on one after another take ten times as long; waited on together they approach the cost of one. Thread pools, event loops and coroutines are all different answers to one question: how do we make that overlap cheap?

Step 1/6Speed

program

1handle(req) {
2  id = parse(req)PC ◀
3  user = db.query(id)   // ask the DB and wait
4  return toJson(user)
5}

When a request arrives the server calls handle (chapter 1's [register + jump]). Pull the id out — CPU working (0.3ms). PC = 2.

CPU0

staterunningPC · position→ line 2

Stack↑ newest frame on top

handle()return → caller

reqid = 42

Heap

∅

Orders of magnitude — if one nanosecond were one second

Every duration is scaled up one billion times, so one CPU instruction feels like a second.

read from L1 cache0.5 ns0.5 s — one heartbeat
read from main memory100 ns1 min 40 s — brushing your teeth
one SSD read150 µs1.7 days — a weekend
round trip inside one datacenter0.5 ms5.8 days — a short holiday
hard disk head seek10 ms16.5 weeks — a university term
intercontinental round trip150 ms4.8 years — a bachelor degree

While you wait for one external call, the CPU could have run hundreds of millions of instructions.

Source: Jeff Dean · Peter Norvig, “Latency Numbers Every Programmer Should Know” (human-scale conversion: hellerbarde). Bars are on a log scale.

Try it in the simulator

1.In the 'add cache' preset, the Redis node's cache hit rate (%) is 95 — run it 10 seconds and Redis answers 13,915 requests instantly while MySQL sees only 6,685.
2.Drop cache hit rate (%) to 50 in the inspector — misses jump 10x and MySQL's load rises to match. That knob is literally how often you hit the wait this chapter describes.

Try it in the simulator →

Terms in this chapter

IO: Exchanging data with devices outside the CPU: disk, network, keyboard.
blocking: Not proceeding to the next line until a result arrives.
latency: Time for one request to finish.
throughput: Work finished per unit time. Overlapping waits raises it even when latency stays.
latency hierarchy: Memory, disk and network differ by orders of magnitude.
overlapped waiting: Running many waits at once to cut total time — the goal of every model that follows.

Why it evolved this way

CPUs run at semiconductor speed while disks and networks are bound by physics. The gap is permanent, so most execution models in computing history are different answers to "how do we make the waiting useful?"

The bottleneck is the wait. Waiting code uses no CPU but still holds memory.

The remaining problem

Could the CPU do something else meanwhile? First there has to be something else →