The Outage That Wasn’t My Server

Two incidents this summer had the same lesson: the obvious suspect was wrong. One was a reboot that took hours. The other was an outage that looked exactly like a failing network card. Here’s how each was traced to its real cause, and what I built so neither can catch me the same way again.

Incident 1: The Reboot That Took Hours

In August I applied routine updates to my primary Docker host and rebooted it. It never came back. The machine answered pings but sat at “stopping Docker” for hours, and I had to power-cycle it through out-of-band access (an IP-KVM and a smart plug I keep for exactly this).

Root cause

The host mounts a media share from the NAS over NFS. During shutdown, tearing down roughly thirty container filesystem mounts plus the NFS mount wedged in an uninterruptible wait that outlasted systemd’s stop timeout. The shutdown simply never finished.

 The second, quieter trap: 29 of the host’s 30 containers use Docker’s unless-stopped restart policy. If you stop containers by hand before a reboot, Docker deliberately does not start them again afterward. Nothing fails, nothing is logged. Everything just stays down.
 

The fix: a safe-reboot tool

Host-aware: on the primary host it stops containers gracefully (with a 60-second grace period so databases checkpoint), stops the media server, and cleanly detaches the NFS mount. On the secondary host it’s a plain reboot.
Preflight checks that block the reboot when something is wrong: someone actively streaming media (it caught a real one on its first live run), the NFS mount missing from the system’s mount configuration, the restore service missing, low disk space, or Docker already down.
Remembers what was running: before rebooting it records the set of running containers, and a one-shot service restarts exactly that set after boot, database first. It only fires after a planned reboot, so crash recoveries don’t trigger it.
Backstops: bounded stop timeouts at the system level, so even a bare reboot can’t hang indefinitely again.

Graceful shutdowns had a bonus effect: they ended a recurring cache-database corruption that had been caused by containers being killed mid-write.

 Result: stopping Docker went from hours to 0.04 seconds, the NFS unmount to 7.3 seconds, and both hosts are back in about 25 seconds, with every container that was running before the reboot running again after it.
 

Incident 2: The Outage That Wasn’t My Server

In September the same primary host logged its network link going down, the NAS share stopped responding, five containers crashed and restarted, and a wall of monitors went red at once.

The obvious suspect

This host’s network card driver had caused a genuine hard crash in July. It was the natural suspect, and the natural next step would have been changing kernels or drivers.

Checking upstream first

Before touching the server, I pulled device uptimes from the UniFi controller’s API. All five switches showed the same uptime, about 1.4 days. They had rebooted together, most likely a firmware auto-update. The gateway had also blipped for about 15 seconds two days earlier.

The timeline lined up: the switches rebooted, the server lost link for about 100 seconds, the NFS mount hung for roughly 14 minutes, and containers depending on it crashed and restarted. The server itself never went down.

 The tell: July’s incident was a hard hang with a crash record saved to persistent storage. September’s was a clean link-down and link-up while the host stayed running. Different failure, different cause, and no change needed on the server. Changing drivers would have “fixed” nothing.
 

It also surfaced a monitoring gap. My host-level ping monitor, set to retry twice before alerting, never alerted on a 100-second outage. That’s worth knowing before the next one.

Lessons

The first suspect is usually whatever broke last time. Check what sits upstream before changing anything.
Correlate timestamps across layers: host logs, network device uptimes, and monitoring history together tell the story.
Silent failures are the worst kind. Build verification into the process itself, like recording and restoring the running set.
Turn incidents into tooling and written rules, so the fix outlives the memory of the outage.

What This Means for Customers

Customers usually arrive at an escalation with a suspect already in mind, often the last thing that broke. The most valuable thing a TAM can do in that moment is establish what actually failed before anyone changes anything: correlate evidence across layers and vendors, build a clear timeline, and explain it in plain language. Then make the fix durable with preflight checks, runbooks, and monitoring that would catch it sooner next time. That’s the same approach I took with enterprise customers at Rackspace and with the platforms I owned at Comply365.

 Key Takeaway: Root cause beats first guess. The fastest path to a real fix is the extra ten minutes spent proving what broke.
 