# The Fleet: Six Machines, and What the Hypervisor Can't Do > The plan was to consolidate everything onto one Proxmox box. Then a perf tool needed real PMU counters, a firmware flash needed a real serial port, and testing on ARM meant a phone running a custom kernel. Meet the fleet — and what each machine taught me. URL: https://harshith.in/blog/the-fleet-why-bare-metal-earns-its-keep/ Author: Harshith Sunku (Staff Engineer) — https://harshith.in/about/ Tags: linux, systems, performance Series: [The Homelab](https://harshith.in/series/homelab/) (part 2 of 5) The lab was supposed to be one machine. [Part 1](https://harshith.in/blog/homelab-proxmox-zfs-lxc/) made the consolidation case — one Proxmox box, twenty containers, everything snapshotted and segmented. Then I built a performance-sampling tool, asked the obvious question — *does it behave the same on ARM?* — and discovered that the machine I had consolidated everything onto couldn't answer it. This post is about the fleet that grew around that lesson. Six machines, and none of them is here for fun: each one exists because I hit a problem the hypervisor could not solve, and each one taught me something I wouldn't have learned from a VM. That's the whole point of this lab — it's a classroom that happens to run services. *[Diagram: The fleet: six machines, one Tailscale mesh]* ## The center, and one cheap trick worth stealing The Proxmox host from part 1 is still the center of gravity: an Ultra 9 285K, 96 GB of DDR5-6600, and three 2 TB Gen4 NVMe drives carrying separate ZFS pools, plus an 8 TB mirrored pool on spinning disks for bulk data. The HDD pool taught me the first lesson. Spinning disks stream large files fine, but metadata is where they die — walking a directory tree of a few hundred thousand files means a seek per lookup, and `ls` starts to feel like an I/O benchmark. The fix was a **ZFS special vdev**: a mirrored pair of 256 GB SATA SSDs attached to the HDD pool that holds its metadata (and optionally small blocks). Directory walks hit flash; big sequential reads still stream off the platters. Two leftover SSDs turned out to be the cheapest upgrade a spinning pool can get. One caveat that matters: the special vdev *is* the pool — lose it and the pool is gone — which is why it's a mirror and not a single drive. ## The counters that wouldn't count The perf tool is where consolidation broke. Sampling profilers lean on the **PMU** — the CPU's performance monitoring unit, the hardware counters behind `perf stat`'s cycles and cache-misses. And a hypervisor is exactly the wrong place to learn how PMUs behave: virtualized counters are incomplete when they're exposed at all, the hypervisor multiplexes them behind your back, and the emulated ARM VM from part 1 has no real PMU whatsoever — QEMU proves your code *runs* on ARM, not how it *performs*. So the tool got tested where counters tell the truth — on bare metal, across three genuinely different microarchitectures: - **i5-3330** — Ivy Bridge, Intel's PMU, 2012 vintage. An old baseline is a feature: if the tool works here, it isn't accidentally depending on anything modern. - **A10-9600P** — AMD's PMU, which is not Intel's PMU. Same `perf` API, different counters, different quirks. You don't learn that from documentation; you learn it when the same code returns different shapes of data. - **Snapdragon 845** — the ARM PMU, big.LITTLE and all, in a phone (more on that below). That test matrix is the lab's thesis in miniature: the hypervisor virtualizes CPUs well and performance counters badly. Anything that touches hardware this directly needs real silicon. ## A phone is the cheapest ARM dev board The ARM machine in that matrix is a Poco F1 — Snapdragon 845, 8 GB of RAM, 256 GB of storage — running a NetHunter kernel. NetHunter is security tooling, but that's not why it's here: the point is a *rooted device with a replaceable kernel*. That combination turns a retired phone into an ARM development board with better specs than most actual dev boards — four Cortex-A75s and four A55s of real big.LITTLE scheduling, a real ARM PMU, and a battery-backed power supply built in. Every "works on my machine" assumption a career of x86 quietly builds up gets audited the first time your tool runs on this thing. ## The bench that has real ports Firmware work was the second thing the hypervisor lost. Flashing embedded boards — like the [ESP32 wake gateway](https://harshith.in/blog/waking-the-lab-esp32-mqtt-wol/) — means serial consoles and USB devices that reset and re-enumerate mid-operation. USB passthrough into a VM can be forced, but a device that drops and reattaches during a flash is a great way to end up with half-written firmware, and the failure is always ambiguous: the board, the cable, or the passthrough? The answer is a machine nobody would miss: an i5-3330 desktop with 16 GB and an assortment of leftover disks. Real USB controllers, a real serial port, no virtualization layer to suspect. When a flash fails there, the board is actually broken — and removing a layer of doubt is worth more than the electricity it burns. It doubles as the Ivy Bridge test target above. ## The machines that run the machines The rest of the fleet is about operating the lab rather than being experimented on. An i5-1240P laptop — 32 GB, a 2 TB and a 512 GB Gen4 NVMe, Windows 11 — is the daily driver and control plane: every console, dashboard, and SSH session in the lab happens from here. And a MacBook Air M4 with 24 GB is the portable half — on the Tailscale mesh from [part 1](https://harshith.in/blog/homelab-proxmox-zfs-lxc/), so "at the lab" is a network property, not a physical one. The A10 laptop rounds things out as the sacrificial box: risky OS installs and one-off servers land there first, where a bad day costs nothing. The deliberate redundancy: the control plane runs Windows, the portable runs macOS on ARM, the lab runs Linux. Three operating systems and two ISAs in daily rotation means cross-platform assumptions get caught by breakfast. ## Takeaways - Consolidation is right up until the work touches hardware. PMUs, serial ports, USB quirks — that's where bare metal earns its keep. - A hypervisor virtualizes CPUs well and performance counters badly. Test anything perf-shaped on real silicon, across more than one vendor's PMU. - A rooted phone with a replaceable kernel is the cheapest real-ARM test device you can own — better specs than a dev board, battery included. - A mirrored special vdev is the cheapest upgrade a ZFS spinning pool can get: metadata on flash, bulk data on platters. Mirror it — it's pool-critical. - Keep one machine nobody would miss. Risky experiments need a home where failure is free. - Next in this series: the strangest tenant on the always-on host — [the AI coworker in an LXC container](https://harshith.in/blog/an-ai-coworker-in-an-lxc/).