Roughly once a day for about a week, the Proxmox host stopped passing traffic. No ping to its gateway, no internet, the attached VM unreachable, git pull failing with DNS errors, the web UI dead. Then I unplugged the ethernet cable, plugged it back in, and everything came back.
That is why it looked physical. Reseating a cable fixing a network problem is about as clear a signal as you get, and I read it the obvious way. Twice.
First time, I assumed a marginal connection, reseated it, and moved on. Second time I concluded the cable was failing, on the reasoning that a fault which recurs after a reseat is a failing cable rather than a loose one, and recommended replacing it. The cable was brand new. That should have re-opened the diagnosis on the spot. It did not, and the recommendation stayed "replace the cable."
The kernel had been saying it the whole time
When I finally read the kernel log instead of looking at the current interface state, the cause was sitting there in plain text, every two seconds, with timestamps, and had been for days: "Detected Hardware Unit Hang." That message comes from e1000e, Intel's driver for the network card, and it is a known bug in that driver. The NIC's transmit queue stalls and the hardware stops sending packets, while the physical link stays electrically up.
That last part is the bit worth keeping. Link state is negotiated at the physical layer. Moving packets is a separate job, done through a descriptor ring the driver fills and the NIC reads. A wedged ring does not affect negotiation, so the link light stays on, the interface still reports LOWER_UP, and nothing moves.
| What I saw | What I told myself | What was actually happening |
|---|---|---|
| Link up, nothing passing | Partially-seated connector | Link up, transmit queue stalled |
| Reseating fixed it | Contact restored | Forced link-down/up reset the adapter |
| It came back daily | Connector working loose | Hang recurs under sustained traffic |
| A new cable did not help | Bad new cable | The cable was never involved |
So the reseat was never fixing a connector. It was resetting the NIC. Unplugging the cable, an interface down/up bounce, and a reboot all clear the stalled queue, and from the outside they are indistinguishable.
Two things I should have noticed earlier. The attached VM went unreachable because its virtual NIC sits on the host's software bridge, and that bridge's only route to the outside world is the affected physical port, so when the port wedged everything behind the bridge lost external connectivity. But host-to-VM traffic kept working in earlier episodes, because that traffic never leaves the bridge and never touches the physical port. That asymmetry was a real clue and it was there the whole time. The other thing: each of these outages risked re-tripping the gateway service's crash-loop breaker again, since a gateway restarting with no network is exactly what produced the run of unclean boots that killed a messaging integration in an earlier session. That was a risk, not something I confirmed happened during this outage.
The fix
Clear the current stall with a link bounce: down, then up. Unloading and reloading the driver module is the heavier option and belongs on the console, because it drops the interface. A reboot always works.
Then disable the offloads: TSO, GSO and GRO. These are performance optimisations the kernel normally delegates to the NIC, and the segmentation logic around them is where the hang behaviour lives. Turning them off means the kernel does that work itself, which at this host's near-idle load costs nothing visible.
The batched form of that change failed with "could not change device features." Split into three separate commands, one feature each, and all three applied. -K, capital, sets features; -k shows them. A feature listed as [fixed] cannot be changed at all. General reminder worth keeping: a tool rejecting a combined command does not mean the individual operations are unsupported.
Persistence has the non-obvious part. ethtool settings live only in the running kernel and vanish on restart, so I added post-up hooks to the network interfaces configuration, under the host's software bridge stanza rather than the member port's stanza. The reason: the member port's stanza has no auto line, because that port is brought up as a side effect of the bridge claiming it, so whether its hooks fire reliably is uncertain. The bridge definitely runs its own hooks, and by the time they run the port exists. The hooks call ethtool by full path, because the boot environment's PATH is minimal. If the hook turns out not to fire, the next option is a systemd unit: more reliable, more setup.
What is still open
At the time of writing the fix was applied, not confirmed. Three things to check: no new hang entries over the next few days, the three offload settings still off after the next reboot, and whether the VM and the gateway service came back after this outage. If hangs continue anyway, interrupt moderation is the next lever, and past that it is genuinely firmware or hardware. Both are contingency, not something I have used.
The reminders
Read the kernel log before trusting current interface state. For an intermittent fault the log tells you what the machine has been complaining about; ip link only tells you what it looks like right now. The kernel had been logging the real cause for days and nobody looked.
"Reseating fixed it" narrows the diagnosis to "something the link reset cleared," which includes driver stalls, not just connectors. Do not buy hardware until that is excluded.
Carrier up means carrier detected, not that the connection works. That indicator has now produced two separate multi-hour debugging sessions in this lab, for two entirely different root causes. Distrust it faster the second time.
When someone tells you the part is new, re-open the diagnosis instead of replacing the part. "The new part is also broken" is a much weaker hypothesis than "the diagnosis is wrong."
Runtime network tuning does not survive a reboot, and persistence hooks belong on a stanza that actually runs.