# ARP: The Protocol That Glues IP to Ethernet

> Every packet's last hop is decided not by routing but by a 1982 protocol with no authentication, a cache with quirks, and failure modes — gratuitous ARP, proxy ARP, spoofing — that explain a shocking fraction of 'the network is weird' tickets.

URL: https://harshith.in/blog/arp-and-the-last-hop/
Author: Harshith Sunku (Staff Engineer) — https://harshith.in/about/
Tags: networking, systems, security

IP routing gets your packet to the right subnet; it says nothing about
delivering it the last meter. Ethernet doesn't speak IP — the NIC delivers
frames to *MAC* addresses — so every single packet's final hop depends on a
translation: who, on this wire, is 192.168.1.7? The protocol answering that
(ARP, RFC 826, sixteen lines of logic from 1982) has no security, subtle
caching behavior, and a starring role in an outsized share of networking
mysteries. Layer 2½ deserves more respect than it gets.

## The Mechanism, Complete

Host A (192.168.1.5) wants to send to B (192.168.1.7) on the same subnet:

```text
  A ──▶ broadcast (ff:ff:ff:ff:ff:ff): "who has 192.168.1.7? tell 192.168.1.5"
  B ──▶ unicast to A:                  "192.168.1.7 is at 52:54:00:ab:cd:ef"
  A caches the mapping, sends the real frame
```

Off-subnet destinations never get ARPed — the routing decision picks a
next-hop gateway, and A ARPs for *the gateway's* IP instead. That one
sentence is the debugging fork: on-link problems are ARP problems;
everything else is routing.

The cache (`ip neigh` — the *neighbor table*, shared conceptually with
IPv6's ND) is what makes this survivable: entries go REACHABLE for a
configurable lifetime, decay to STALE, and get re-verified on next use
(DELAY → PROBE, a unicast re-ask) rather than re-broadcast. Reading those
states is diagnostic: a neighbor stuck FAILED means no reply on the wire
(wrong VLAN, dead host, L2 filtering); INCOMPLETE means the request itself
is vanishing. And the table is finite — hosts sitting on huge flat L2
networks (/16 container networks, big VM subnets) can thrash the neighbor
table's garbage-collection thresholds (`gc_thresh1/2/3`), producing the
classic "random timeouts on a big subnet" incident whose dmesg signature is
`neighbour: arp_cache: table overflow`.

## The Tricks Built on Top

The protocol's flexibility comes from an omission: *anyone can answer, and
answers are believed*. Three legitimate systems exploit it:

- **Gratuitous ARP** — announcing your own IP unasked. This is the beating
  heart of every IP-failover scheme: keepalived/VRRP takes over a virtual
  IP and broadcasts gratuitous ARPs so every switch and host on the segment
  repoints the MAC immediately, instead of after cache expiry. Failover
  that "takes 30 seconds for some clients" is a GARP not being sent — or
  being ignored.
- **Proxy ARP** — a router answering for addresses *behind* it, letting
  hosts believe a remote destination is on-link. Historically how dial-up
  and some VPN/DSL setups stitched segments; today mostly a surprise you
  discover enabled (`/proc/sys/net/ipv4/conf/*/proxy_arp`) while debugging
  why a misconfigured /8 netmask *works anyway*.
- **ARP announcement on address config** — the duplicate-address detection
  ping; two boxes claiming one IP produce alternating cache flaps visible
  as `ip neigh` MAC changes and intermittent 50% packet loss — one of
  networking's most recognizable symptom patterns once you've seen it.

The same omission is the attack: **ARP spoofing** — answer faster and more
often than the real host, become the man in the middle for a whole segment
(`arpspoof` is thirty years old and still works on flat unprotected LANs).
Which is why enterprise gear ships Dynamic ARP Inspection (validate replies
against DHCP-snooping state), why hypervisors filter guest ARP by policy,
and why "the segment is the trust boundary" is a rule that predates and
outlives most network security fashion.

Field kit for the last hop: `ip neigh` (states, and whether the MAC is the
one you expect), `tcpdump -e arp` (watch requests go unanswered — the `-e`
prints MACs and settles who's really talking), `arping -I eth0 <ip>`
(targeted ask, bypasses cache), and clearing a poisoned/stale entry with
`ip neigh flush dev eth0`. Ninety percent of "can ping the gateway but not
my neighbor" resolves inside those four commands.

## Takeaways

- ARP answers the only question Ethernet cares about: which MAC owns this
  on-link IP. Off-link traffic ARPs for the gateway — the fork in every
  L2-vs-L3 debug.
- The neighbor cache's states (REACHABLE/STALE/PROBE/FAILED) are readable
  diagnostics; big flat subnets overflow it — raise gc_thresh or shrink
  the L2 domain.
- Gratuitous ARP makes IP failover instant; its absence is why failovers
  half-work.
- Anyone can answer: spoofing is trivial on unprotected segments — DAI/
  snooping where it matters, and never treat a LAN as friendly by default.
- tcpdump -e arp + ip neigh + arping localize nearly every last-hop mystery.