state: batch HA probe results so dual-disconnect cannot flap primary

requirePrimaryStable in TestHASubnetRouterFailoverDockerDisconnect
Phase 5a (simultaneous cable-pull of both routers) intermittently
caught the primary flipping to the offline r1. Both probe goroutines
mark their target unhealthy back-to-back; SetNodeUnhealthy publishes a
fresh NodeStore snapshot each call, so the intermediate snapshot — r1
unhealthy, r2 still healthy — runs the election with one healthy
candidate left and picks it. The next snapshot then enters the
all-unhealthy preserve-prev path, which preserves the wrong choice.

Collect probe results from the cycle and apply them through a new
NodeStore.UpdateNodes batched op so the election only runs once, with
the cycle's final health state. PolicyChange dispatch moves outside
the wg.Go goroutines and fires once if the primary assignment
actually changed.
This commit is contained in:
Kristoffer Dalby 2026-05-14 17:04:12 +00:00
parent fb8eecae25
commit de6be71a86
6 changed files with 157 additions and 88 deletions

View file

@ -1222,35 +1222,64 @@ func (s *State) IsNodeHealthy(id types.NodeID) bool {
return s.nodeStore.IsNodeHealthy(id)
}
// SetNodeUnhealthy flips the runtime Unhealthy bit and reports whether
// the resulting primary route assignment changed, so the HA prober can
// decide whether to fan out a PolicyChange.
//
// A request to mark a node Unhealthy is dropped if the node is no
// longer an HA candidate (offline or no approved routes). The prober
// reads HANodes() at the start of a probe cycle and writes back after
// the timeout fires; in that window the node may have left the
// candidate set, and the bit would just be stale. The check happens
// SetNodeHealth flips the runtime health bit for one node and reports
// whether the resulting primary-route assignment changed, so the HA
// prober can decide whether to fan out a PolicyChange. true means
// healthy; false means unhealthy. An unhealthy mark is dropped when
// the node is no longer an HA candidate (offline or no approved
// routes) — between probe dispatch and result the node may have left
// candidacy, and the bit would just be stale. The check happens
// inside the writer goroutine so it serialises against the
// SetApprovedRoutes / Disconnect that removed candidacy.
func (s *State) SetNodeUnhealthy(id types.NodeID, unhealthy bool) bool {
// SetApprovedRoutes / Disconnect that removed candidacy. Single-node
// convenience wrapper around [State.BatchSetNodeHealth].
func (s *State) SetNodeHealth(id types.NodeID, healthy bool) bool {
return s.BatchSetNodeHealth(map[types.NodeID]bool{id: healthy})
}
// BatchSetNodeHealth applies a set of health updates atomically: the
// election runs once after every flag has been flipped, so observers
// never see an intermediate snapshot. Returns true when the
// primary-route assignment differs from before the batch so callers
// can gate a single PolicyChange dispatch. Map value true = healthy;
// false = unhealthy (gated as in [State.SetNodeHealth]).
//
// Per-call publication would let a writer applying two flips
// back-to-back elect a node that the next snapshot demotes,
// momentarily pointing peers at the wrong primary; the batched form
// closes that window.
func (s *State) BatchSetNodeHealth(updates map[types.NodeID]bool) bool {
if len(updates) == 0 {
return false
}
prevRoutes := s.nodeStore.PrimaryRoutes()
_, ok := s.nodeStore.UpdateNode(id, func(n *types.Node) {
if unhealthy {
fns := make(map[types.NodeID]UpdateNodeFunc, len(updates))
for id, healthy := range updates {
fns[id] = healthSetter(healthy)
}
s.nodeStore.UpdateNodes(fns)
return !maps.Equal(prevRoutes, s.nodeStore.PrimaryRoutes())
}
// healthSetter returns an UpdateNodeFunc that flips n.Unhealthy to
// the inverse of healthy, with the same gate as [State.SetNodeHealth]:
// an unhealthy mark only sticks when the node is still online and
// still advertises approved routes, so a node that left HA candidacy
// between probe dispatch and result does not carry a stale bit.
func healthSetter(healthy bool) UpdateNodeFunc {
return func(n *types.Node) {
if !healthy {
online := n.IsOnline != nil && *n.IsOnline
if !online || len(n.AllApprovedRoutes()) == 0 {
return
}
}
n.Unhealthy = unhealthy
})
if !ok {
return false
n.Unhealthy = !healthy
}
return !maps.Equal(prevRoutes, s.nodeStore.PrimaryRoutes())
}
// ValidateAPIKey checks if an API key is valid and active.