Rich Gibbs

One Loose Cable, Two Hidden Start Paths

homelab · networking · troubleshooting · dhcp · systemd

Notes from a two-day homelab session. A marginal ethernet cable produced five misleading symptoms that looked like name-resolution, routing, and hypervisor failures; reseating the cable at both ends fixed all of them, because carrier detected is not the same as the connection working. Separately, a gateway process showed active (running) while its messaging channel was deliberately suppressed by a restart-loop breaker, tripped by eleven unclean boots inside a five-minute window. The underlying cause was a duplicate user-scope service competing for the same port, and the unit was already disabled — the login-state property was what kept the second start path alive. The rule worth keeping: verify the function, not the status.

I want this one written down while the details are still sharp, because both faults in this session looked like something they were not.

The first was a cable. The hypervisor host could reach its own guest VM and nothing beyond it. Pinging the router by address failed, and a source fetch died with a name-resolution error. That error is what sent me looking at name resolution first, which was the wrong direction: every layer I could inspect in software was already correct. The physical interface showed link up with carrier detected, the bridge held the correct address, and the routing table had a valid default route. Everything looked perfect and nothing moved. The cause was an ethernet cable seated well enough to negotiate a link but not well enough to pass traffic reliably. Unplugging it and firmly reseating both ends fixed it.

That is the lesson I do not want to relearn: carrier detected is not the connection working. When the software stack is demonstrably correct and nothing passes, suspect the wire before the config.

The symptom chain pointed the wrong way the whole time, and that cost real time. The visible failures ran in order: the source fetch failing, then name resolution failing, then the router unreachable, then the hypervisor web interface unreachable, then an empty realm dropdown on its login page. Five separate-looking symptoms, one loose cable. I chased it top-down and should have chased it bottom-up: link, then routing, then name resolution, then application.

The clue that localizes it is the asymmetry. The host could reach the guest VM but not the router. Guest traffic stays inside the virtual bridge and never touches the physical NIC, so that working path proved the software stack was fine while saying nothing at all about the cable. The cheap general test that cuts the problem in half: move the suspect cable to a laptop. If that machine gets an address, the cable and the switch port are good and the server is at fault; if it does not, the cable or the port is.

Two flags I should stop reading as symptoms. A container bridge showing NO-CARRIER when nothing is attached to it is normal, because nothing is attached to it. And a device with a static address showing as offline in the router's client list means nothing — the router judges presence from lease activity, and a static device never talks to the router's DHCP server. I chased both of those on the way down.

The reservation part is short and worth keeping. Bind the NIC to a fixed address placed clearly outside the observed lease range, then remember the reservation does not take effect until the client renews — reboot and confirm rather than assuming. And record the pool's start and end in the network notes, because those bounds constrain every future static or reserved assignment on this LAN.

The second fault was the more interesting one. The gateway showed active (running), the host had working internet, and the messaging channel was silently dead. A restart-loop breaker had deliberately suppressed the channel layer after eleven unclean boots inside a five-minute window — the boot loop was the network fault above, the gateway starting with no network, crashing, and being restarted. Clearing it meant stopping the service, waiting out the window with zero restarts, then starting fresh. A plain restart may not reset a time-based counter; stop, wait, start does.

The root cause was a second start path. Two service scopes were competing for the same port: a stray user-scope unit had claimed it, and the intended system-scope unit kept failing with address-in-use. The tell was there earlier and I read it as success — address-in-use was the first sign of a duplicate start path, and I filed it under the migrated unit starting itself, all good.

The obvious fix was not the actual fix. Disabling a unit is not the same as stopping it — stop lasts until the next boot, disable is permanent — but the duplicate was already disabled and its door was still held open: the login-state property was keeping the user manager alive from boot with nobody logged in, which is exactly how a user-scope service ends up running on a headless server. It had almost certainly come across from the cloud host, where that property was legitimately needed. Checking only whether the unit was enabled would have missed the whole mechanism. Enumerate every start path — system units, and each account's user-scope units separately — before theorizing about which process owns a port.

Last piece worth remembering: authorization is not reachability. Creating an account and granting sudo is reversible and fine, but it does not create a route. A VM behind NAT on apartment WiFi has no inbound path no matter what credentials exist; that gap needs a mesh VPN, a reverse tunnel, or port forwarding, and port forwarding was not available on that network. Can it connect and is it allowed are different questions that fail differently and need different fixes.

On the mesh VPN trade-off, stated honestly: each node gets a stable address that works regardless of the physical network, with no port forwarding and no reverse tunnels. Traffic is end-to-end encrypted and usually peer-to-peer, but a third party operates the coordination plane — worth knowing, not hidden. Both machines have to join the same tailnet, and the free tier covers 100 devices.

What I am carrying forward. Verify the function, not the status: the repair was signed off with a real round-trip message, not a green unit state. Disabling the unit without disabling the mechanism that lets it start is half a fix. Access granted was also access withdrawn — remote-access SSH is disabled on the VM now, and the cloud host is logged out of the tailnet. The leftover repair backup on the host still needs cleaning up, the disk resize is still deferred behind a snapshot-first plan, and the reboot check — exactly one process owning that port — is still open. Backups still do not exist, and these notes call that the highest-value outstanding item, which is hard to argue with. And if the link drops again: replace the cable rather than reseating it.