rsp2k/headscale
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

state, servertest: property-test HA election + invariant catalogue

Browse source 
Expand TestPrimaryRoutesProperty (5 -> 9 ops). New ops mirror the
production shapes the failure cases hit: BatchProbeResults via
UpdateNodes, SimultaneousDisconnect via UpdateNodes, SetApprovedRoutes
that leaves announced RoutableIPs intact, OfflineExpiry that keeps
Unhealthy set. The model now tracks announced and approved separately
and recomputes the intersection.

Strengthen the per-op assertions to cover invariants the model alone
cannot prove: every primary must be online, every primary must
currently advertise its prefix, no flap onto an unhealthy candidate
when a healthy one was available, no flap off a previous primary that
remains a healthy candidate. The check now takes a pre-op snapshot so
the anti-flap rule has a stable reference.

Add TestHAProberProperty in servertest. It drives a real TestServer
with three HA-route-advertising clients through rapid-drawn sequences
of ClientDisconnect / ClientReconnect / ProberTick / WaitForSnapshot
ops and re-checks the same shape invariants after every step.

Document the system in hscontrol/state/HA_INVARIANTS.md: a state
machine over (Healthy+Online, Unhealthy+Online, Offline,
OfflineExpired), fifteen numbered invariants with predicates and
violation paths, and a coverage matrix mapping each invariant to its
unit, servertest, and integration tests. Three rows pin the recent
fixes to the invariants they enforce.
This commit is contained in:
Kristoffer Dalby 2026-05-17 20:32:53 +00:00
parent c7630b505b
commit e2f2f9211f
5 changed files with 968 additions and 43 deletions
Download patch file Download diff file Expand all files Collapse all files

23
hscontrol/state/primaries_test.go
View file

@ -218,12 +218,12 @@ func TestPrimaries_AllUnhealthyKeepsAPrimary(t *testing.T) {
}
func TestPrimaries_AllUnhealthyPreservesPrevious(t *testing.T) {
// Issue #3203: once a failover has moved primary to a higher-ID
// node, a subsequent all-unhealthy state must NOT churn primary
// back to the lowest-ID candidate. Under cable-pull semantics
// both nodes can linger as IsOnline=true (half-open TCP) and
// both go Unhealthy — naive `candidates[0]` would flap the
// primary to a node that is itself unreachable.
// Once a failover has moved primary to a higher-ID node, a
// subsequent all-unhealthy state must NOT churn primary back to
// the lowest-ID candidate. Under cable-pull semantics both nodes
// can linger as IsOnline=true (half-open TCP) and both go
// Unhealthy — naive `candidates[0]` would flap the primary to a
// node that is itself unreachable.
prefix := mp("10.0.0.0/24")
f := newPrimariesFixture(t, 1, 2)
f.advertise(1, prefix)
@ -247,12 +247,11 @@ func TestPrimaries_ExitRouteNotElected(t *testing.T) {
f.requireNoPrimary(exitV4)
}
func TestPrimaries_RegressionIssue3203_BothOfflineThenOneReturns(t *testing.T) {
// Issue #3203: with two HA advertisers, dropping both then
// bringing one back used to leave the prefix without any
// primary. After the refactor the snapshot recomputes primaries
// on every NodeStore write, so the returning advertiser must
// be elected.
func TestPrimaries_BothOfflineThenOneReturns(t *testing.T) {
// With two HA advertisers, dropping both then bringing one back
// used to leave the prefix without any primary. The snapshot
// recomputes primaries on every NodeStore write, so the
// returning advertiser must be elected.
prefix := mp("10.0.0.0/24")
f := newPrimariesFixture(t, 1, 2)
f.advertise(1, prefix)