Server method ① — the thread pool

What if every waiting request had a person of its own sitting on it? A dedicated clerk per customer is comfortable — but that clerk must stay in the seat even while the customer is off fetching papers.

See this first in FoundationsTwo hundred windows — why are only ten working?

The oldest and most intuitive way to serve requests: assign a thread per request, and let it run chapter 2's synchronous code top to bottom. The code is easy to read and easy to debug. Watch requests arrive and get assigned below.

① Why a "pool"

Creating and destroying a thread per request wastes real cost, so a fixed set is created up front and lent out. Tomcat's default of 200 is exactly this.

② Concurrent requests = thread count

One thread holds one request until its reply arrives, so four threads mean at most four requests in flight; the fifth waits in a queue. Try 1, 4, 16 and watch the completion rate change.

③ Look: most of them are asleep

At any moment most threads read "waiting on db". The CPU works only during the brief parse and response phases. A "busy" server usually means threads are all waiting, not that the CPU is loaded.

④ Why you cannot just add threads

Each thread reserves stack memory (often hundreds of KB to 1MB), so 10,000 threads is gigabytes. More threads also mean more context switches — time spent switching rather than working. The memory and switch numbers below are that bill.

⑤ How to size the pool

The common mistake is assuming more is better. Reuse the previous calculation: ten milliseconds of CPU and ninety of waiting gives about ten threads per core as a starting point. Then multiply by what is downstream. If the connection pool holds 20 and you run 200 threads, the extra 180 simply stand in line for a connection. Bottlenecks move; they do not vanish.

⑥ The queue policy is riskier than the size

Every pool has a wait queue. Leave it unbounded and overload grows memory while you belatedly serve requests the user already abandoned. Bound the queue and reject when it overflows. Failing fast beats failing slowly — the basic principle of overload design.

Handler (blocking style) — every thread runs this

1handle(req) {
2  id = parse(req)① CPU works
3  user = db.query(id)② the thread sleeps here
4  return toJson(user)③ CPU works
5}

SpeedThreads

Try it in the simulator

1.Same 'HikariCP pool exhaustion' preset — at default (pool 5 + HDD) the KPI panel's Success is 2.4% (14,280 drops, 12,856 of them queue-full). Raise HikariCP pool to 50 in the inspector and Success stays at 2.4%.
2.The real ceiling is the disk: HDD 200 IOPS / 5.7 IOPS per request is about 35 RPS. Switch MySQL's disk to SSD (5,000 IOPS) and Success jumps to 24.6% — but now Spring's CPU gauge hits 100%.
3.Widening the pool alone never fixes it. Bottlenecks reveal themselves one at a time.

Try it in the simulator →

Terms in this chapter

thread pool: Pre-created, reused threads that avoid per-request creation cost.
request queue: Where new requests wait when every thread is busy.
stack memory: Per-thread memory; grows linearly with thread count.
wait queue: Where requests pile up when every thread is busy; dangerous when unbounded.
downstream resource: Connection pools and external APIs behind the threads — often the real bottleneck.

Why it evolved this way

The real advantage here is simplicity, not speed: linear code, intact stack traces, ordinary debugging. That is why it was the default for decades and still runs most business servers.

Thread pool = overlap waiting with sleeping threads. Concurrency = thread count.

The remaining problem

But 10,000 connections means 10,000 stacks and constant switching. In 1999 this got a name: the C10K problem. What if a wait did not need a whole thread? →