Request a larger receive buffer for conntrack events

[hiroTamada]

Summary

The conntrack event socket ran on the kernel default receive buffer. On production hypeman hosts the kernel reports recv conntrack events: no buffer space available roughly once a minute each — 345 times across three hosts in a 4.5 hour window:

service.name = 'hypeman'
AND body = 'auto-standby conntrack subscription failed'
AND attribute.error CONTAINS 'no buffer space available'

Every one of those is a batch of connection events the auto-standby controller never sees. The tracked connection view is built from that stream, so a connection whose event is lost is not counted as inbound activity — and an instance still serving it looks idle. Since a suspended guest is only woken by a new connection, the already-open one then hangs until the client gives up.

Change

Request 40MiB for the socket, which Linux stores as 80MiB (it allocates twice what you ask for). net.core.rmem_default on these hosts is 8MiB, so this is roughly 10× the headroom.

SO_RCVBUFFORCE is tried first because it ignores net.core.rmem_max and so cannot be silently clamped; it needs CAP_NET_ADMIN, which hypeman has (runs as root with cap_net_admin in the unit's bounding set). Falls back to SO_RCVBUF where that capability isn't available. Both failing is tolerated — a socket with the default buffer still delivers events, so buffer sizing must not stop the subscription from opening.

What this does and does not do

It reduces how often events are dropped. That's the whole claim.

It does not make a drop recoverable. On ENOBUFS the reader still treats it as a stream failure and tears the subscription down, which loses everything else that arrives during the reconnect, and nothing re-reads the connection table on the way back up. And it does not make the idle decision verify itself — executeStandby still trusts the event-derived view without confirming it against the host table before suspending a guest.

Both of those are on hypeship/auto-standby-conntrack-resync-parked and hypeship/auto-standby-hardening-parked respectively, deliberately kept out of this PR.

There is also a reason the buffer overflows that more buffer cannot fix: handleConnectionEvent iterates every tracked instance under the global lock for every conntrack event on the host, so the consumer falls behind and the reader blocks. Indexing that lookup is the change that would attack the cause.

Testing

go test -race ./lib/autostandby/... passes and the package builds for linux and darwin. No new tests: this is a socket option whose effect is a kernel-side buffer size, and asserting it would require a privileged netlink socket that CI doesn't have.

The real verification is the drop rate in production. Run the query above before and after rollout, grouped by host.name. If the rate doesn't fall materially, the buffer was not the binding constraint and the consumer-side fix is the next thing to try.

Not included, and worth knowing: the socket is never read back with getsockopt, so the effective buffer size isn't logged anywhere. The drop rate is the only feedback signal.

🤖 Generated with Claude Code