# Buffer Overflows: From strcpy to Shell, and the Mitigations That Ended It

> A single unbounded copy can hand an attacker your instruction pointer. The classic stack smash, then the 25-year arms race — stack canaries, NX, ASLR — that turned a reliable exploit into a hard research problem.

URL: https://harshith.in/blog/how-buffer-overflows-work/
Author: Harshith Sunku (Staff Engineer) — https://harshith.in/about/
Tags: security, systems, memory

The most-studied bug in security history is also one of the simplest to
write: copy untrusted bytes into a fixed buffer without checking the
length. Understanding *exactly* how that becomes a compromise — and,
more importantly, how each mitigation layer breaks a specific link in
the chain — is core defensive knowledge. This is written for the
defender's side of the fence: to configure hardening correctly and read
a crash for exploitability, you have to know what the attacker needed.

## Why the Stack Is the Target

Local variables and the function's **return address** share the stack,
adjacent, growing toward each other. A buffer that overflows writes
*upward* into the saved return address:

```text
   [ char buf[64] ][ saved rbp ][ return address ]
        │ overflow ───────────────────▶ overwritten
   gets(buf); // 200 bytes of attacker input → return addr = attacker's
```

When the function returns, the CPU jumps to whatever sits in that slot.
Control-flow hijack from a copy loop. The historical exploit (Aleph
One's 1996 "Smashing the Stack", the intellectual starting point for
this whole field) placed shellcode in the buffer itself and pointed the
return address back at it. `strcpy`, `gets`, `sprintf`, unchecked
`memcpy` — the unbounded-copy family is the classic source, and
integer-overflow-into-undersized-`malloc` the modern one.

## The Mitigation Stack, Layer by Layer

No single defense "fixes" this; the durable win came from stacking
independent ones, each invalidating an assumption the exploit relied
on. Reading them in order *is* the mental model:

- **Stack canaries** (`-fstack-protector-strong`, on by default). A
  random value placed between buffers and the return address, checked
  on function exit; a linear overflow that reaches the return address
  must first overwrite the canary, and the mismatch aborts
  (`*** stack smashing detected ***`). Breaks *sequential stack
  overflow* — sidestepped by info-leaks (read the canary first) or
  writes that skip over it.
- **NX / DEP** (W^X — no page both writable and executable). The
  stack is data, marked non-executable, so shellcode-in-buffer simply
  faults. Breaks *code injection* — and attackers answered with
  **ROP** (return-oriented programming): don't inject code, chain
  together existing snippets ("gadgets") ending in `ret`. NX raised
  the bar from "write shellcode" to "build a ROP chain."
- **ASLR** — randomize the base addresses of stack, heap, libraries,
  (and PIE) the executable. Now the attacker doesn't *know* where the
  gadgets or their buffer are. Breaks the *hardcoded address*,
  including most ROP — its kryptonite is any **information leak** that
  discloses one runtime address (from which the rest are offsets), so
  leaks became the center of gravity of modern exploitation.
- **CFI / shadow stacks** (the current frontier: Clang CFI,
  Intel CET's hardware shadow stack + endbranch). Even with a leak,
  constrain indirect jumps/returns to legitimate targets — a shadow
  stack keeps an untamperable copy of return addresses and faults on
  mismatch, attacking ROP at its root.

The lesson generalizes past this one bug: **defense in depth as
assumption-invalidation.** Each layer is individually bypassable and
the stack is collectively strong, because chaining a bypass for all of
them (a leak *and* a suitable overflow *and* gadgets *and* CET evasion)
is the difference between a 1999 script and a modern research
write-up. It didn't kill the bug class — memory-safety bugs are still
~70% of the CVEs Microsoft and Chrome report yearly, which is the
actual argument for Rust and for `-fbounds-safety`-style work — but it
made each instance dramatically harder to weaponize.

## What Defenders Do With This

Concretely: compile with the modern defaults on (`-D_FORTIFY_SOURCE=3`
turns unbounded libc calls into checked ones when the size is known at
compile time; `-fstack-protector-strong`; `-fstack-clash-protection`;
PIE + full RELRO so the GOT and return path are hardened) and *verify*
they're present (`checksec`, `hardening-check`) rather than assuming
the toolchain did it. Find the bugs before shipping with ASan (shadow
memory catches the overflow at the write, with a stack trace) and
coverage-guided fuzzing on every parser that eats untrusted bytes —
that pairing is how the overflow gets caught in CI instead of in a
CVE. And when triaging a crash, "SIGSEGV with a controlled register"
or a tripped canary is the signal to treat it as security-relevant,
not a routine null-deref — the exploitability question is exactly the
mitigation-bypass question above.

## Takeaways

- Overflows win by overwriting the saved return address adjacent to
  stack buffers; unbounded copies and integer-overflow allocations are
  the sources.
- The mitigations each kill one link: canaries (sequential overwrite),
  NX (code injection), ASLR (known addresses), CFI/shadow stack
  (ROP/return hijack).
- Every layer is individually bypassable — via info leaks above all —
  and collectively raises weaponization from scripting to research.
- Turn the defaults on *and check them* (checksec, FORTIFY, PIE/RELRO,
  stack-clash); assume nothing about your build.
- Catch instances pre-ship with ASan + fuzzing; memory-safe languages
  remove the class, which is why the industry is migrating.