During a routine check of a machine, I caught it mid-transfer: 72.5GB had gone out over the network in the thirty minutes since boot, at a measured 12.9KB/s. Chasing it down, the culprit was a backup tool quietly pushing a 118GB folder that had arrived the day before to a remote backup target. Nothing was wrong, exactly — it was doing its job — but nothing had told me it was happening, and without a running counter I would never have known.
Building the thing that would have caught it
The response was a small always-on monitor: one script samples network interfaces, reachability, and per-process traffic every 60 seconds and appends a line to a log; a second, more expensive script checks latency and loss on the wider network every 10 minutes. Splitting them mattered — the deep check takes about 9 seconds for a single loss measurement, which doesn't fit inside a 60-second cadence next to everything else.
A few small findings while building it: a per-process traffic breakdown that would normally need elevated privileges turned out to be readable without it, through one particular tool's flags. And a column-counting script that read network interface statistics by position broke depending on whether an interface had a hardware address, which shifted every column after it — fixed by counting from the end of the line instead of the start, since the trailing columns are stable regardless.
The reachability check that only worked in front of me
Once the monitor was live and running unattended, 98% of its samples for one particular reading came back null. The command behind it exited successfully but its output wasn't the JSON I expected — it was an error message saying the background helper had failed to start.
The mesh-networking client I was querying ships as an app-store build, where the command-line tool has to go through the background app to talk to the network, and an unattended background process launched outside any user session can't reach that app the way an interactive terminal can. It worked flawlessly every time I tested it by hand, because testing it by hand is an interactive session. The bug was invisible from the one vantage point I kept checking it from.
An early fix attempt — forcing a particular environment variable the client might have needed — seemed to work, and didn't. I'd tested it in a shell that was already warmed up from earlier interactive success, which papered over the real failure. Only a cold, unattended restart of the whole monitoring service showed the fix hadn't done anything.
What actually worked
Rather than depend on the client's own command-line tool at all, the monitor now pings each known node's tunnel address directly — plain ICMP, no CLI, no background app, and it runs in parallel across every node in about a second. The richer reachability detail (direct connection versus relayed, handshake state) is still gathered when the CLI happens to be reachable, and each reading is tagged with which method produced it, so a silent fallback doesn't get mistaken for the fuller answer. The one caveat left standing: a plain ping can read as unreachable on a mobile connection that's simply slow to answer, so an offline reading from that path alone isn't fully trusted yet.
The same day, watching the new dashboard render live, one number changed in front of me: every node that had shown as relayed since boot moved to a direct connection. That had been sitting there, unnoticed, until there was finally a screen to notice it on.