Concurrency ≠ parallelism

Does adding more workers always finish the job sooner? In a shop with a single till, calling in extra staff does not move the queue any faster.

Many people use "handling many things at once" and "computing many things at once" interchangeably. They are different, and confusing them leads to piling on threads and wondering why nothing got faster. Flip the toggles below and see.

① Concurrency — making progress on several things

Even with one core several tasks can be in progress: while one waits on IO another uses the CPU. At any single instant only one is actually computing.

② Parallelism — computing at the same instant

With four cores, four computations physically happen together. Concurrency works on one core; parallelism requires more cores.

③ Workload decides

CPU-bound work (image processing, crypto, compression) gets no faster with more threads on one core — only more switching. IO-bound work (queries, API calls, file reads) overlaps with more threads even on one core, because waiting uses no CPU.

④ See it yourself

The panels run the same work with 1 and 4 threads. With CPU work on 1 core, four threads finish no sooner — sometimes later. Switch cores to 4 and it finally scales. Then switch the workload to IO and watch four threads win on a single core.

⑤ Cores are the ceiling on parallelism

No more work is computed at one instant than you have cores. With four cores, a hundred threads still compute at most four things at once. So for CPU-bound work, creating far more threads than cores gains nothing and costs context switches and memory. For IO-bound work the opposite holds: threads far outnumbering cores is fine, because most of them are asleep.

⑥ So how many should you run?

Start from arithmetic. If one request spends c on CPU and w waiting, keeping one core busy needs roughly (c+w)/c requests in flight. Ten milliseconds of CPU and ninety of waiting means about ten per core. CPU-bound: near the core count. IO-bound: scaled by that ratio. This is the first calculation behind both thread pool and connection pool sizing.

SpeedWorkloadCPUs

Try it in the simulator

1.The 'scaled to 10 apps, why?' preset runs App (Spring) with Instances at 10, in parallel. At 10,000 RPS over 10 seconds, App CPU sits at a lazy 32% while MySQL CPU runs hot at 88% (peak 93%).
2.Nudge App's Instances up and down in the inspector — however parallel the app gets, MySQL's card stays just as red. More concurrency upstream doesn't erase a shared bottleneck downstream.

Try it in the simulator →

Terms in this chapter

concurrency: Several tasks progressing by interleaving in time. Possible on one core.
parallelism: Several computations happening at the same instant. Needs several cores.
CPU-bound: Mostly computing. Needs more cores.
IO-bound: Mostly waiting. Benefits from overlapping.
parallelism ceiling: The maximum computed at once, set by the core count.
CPU-to-wait ratio: Compute time versus waiting time for one request; the basis for sizing concurrency.

Switching hides waiting; it cannot create computation.

The remaining problem

Which are server requests? A little parsing, a long database wait, a little response — textbook IO →