# Anatomy of a Context Switch: What the Kernel Does Between Two Threads

> A context switch looks free until you profile a thread-heavy workload and watch it vanish into the kernel. Here is everything the scheduler actually does to swap one thread for another.

URL: https://harshith.in/blog/context-switch-anatomy/
Author: Harshith Sunku (Staff Engineer) — https://harshith.in/about/
Tags: kernel, performance, linux

Schedulers feel like magic until you profile a thread-heavy workload and watch
real time disappear into `__schedule`. A context switch is not free, and the cost
is not where most people think. Let's trace exactly what happens between the
instant one thread stops running and the next one starts.

## What Counts as a Context Switch

Two very different things wear the same name:

- **Voluntary** — a thread blocks (I/O, a lock, `sleep`) and hands the CPU back.
- **Involuntary** — the scheduler preempts a thread that still wanted to run.

Both end up in the same kernel routine, but the trigger matters: involuntary
switches cluster under contention and are the ones that wreck tail latency.

## The Three Things the Kernel Saves

When `schedule()` decides thread B should run instead of thread A, it has to make
A resumable later and B runnable now. Three pieces of state move:

```c
// Simplified from kernel/sched/core.c
static void context_switch(struct rq *rq,
                           struct task_struct *prev,
                           struct task_struct *next)
{
    prepare_task_switch(rq, prev, next);
    switch_mm_irqs_off(prev->active_mm, next->mm, next); // address space
    switch_to(prev, next, prev);                         // registers + stack
    finish_task_switch(prev);
}
```

1. **CPU registers** — general-purpose registers, the stack pointer, and the
   instruction pointer get parked in thread A's kernel stack.
2. **The address space** — if B belongs to a different process, `switch_mm`
   installs B's page tables by writing `CR3` on x86.
3. **Bookkeeping** — runqueue accounting, scheduler statistics, and FPU state
   (saved lazily on most CPUs).

## Where the Cycles Actually Go

The register save/restore is cheap — tens of nanoseconds. The expensive part is
the address-space change:

```text
        thread A                 thread B
   ┌──────────────┐         ┌──────────────┐
   │ registers    │  fast   │ registers    │
   │ kernel stack │ ──────▶ │ kernel stack │
   └──────────────┘         └──────────────┘
          │  write CR3 (only if mm differs)
          ▼
   ┌─────────────────────────────────────────┐
   │  TLB entries for A may now be invalid    │  ← the real tax
   └─────────────────────────────────────────┘
```

Writing `CR3` can flush the TLB. The next thread then takes a burst of TLB
misses as it re-walks page tables for code and data it touches. That refill —
not the register copy — is what you see in a flame graph.

This is why **thread switches within the same process are cheaper than process
switches**: same `mm`, no `CR3` write, TLB stays warm. It's also why `PCID`
(process-context identifiers) exists — tagging TLB entries so a switch doesn't
have to flush everything.

## The Path, End to End

```mermaid
flowchart LR
    A[Thread A running] --> B{schedule called}
    B --> C[pick next: Thread B]
    C --> D[switch_mm: write CR3]
    D --> E[switch_to: swap registers + stack]
    E --> F[Thread B running]
    F --> G[TLB refills on first accesses]
```

## Measuring It Yourself

`perf` makes the cost visible without guesswork:

```bash
# Count context switches for a command
perf stat -e context-switches,cpu-migrations ./your_workload

# See who is switching and why
perf sched record ./your_workload
perf sched latency --sort max
```

If `context-switches` scales with load while useful work stalls, you're paying
the scheduler tax — usually from lock contention or oversubscribed threads.

## Takeaways

- A context switch saves registers, may swap the address space, and updates
  scheduler bookkeeping — in that order of cost.
- The register copy is cheap; the **TLB refill after a `CR3` write** is the real
  expense. Same-process thread switches avoid it.
- `PCID`/tagged TLBs exist specifically to soften that cost.
- Reach for `perf stat -e context-switches` and `perf sched latency` before
  blaming the scheduler — most "scheduler overhead" is contention in disguise.