mapper/batcher: serialize per-node work to prevent out-of-order delivery

processBatchedChanges queued each pending change for a node as a
separate work item. Since multiple workers pull from the same channel,
two changes for the same node could be processed concurrently by
different workers. This caused two problems:

1. MapResponses delivered out of order — a later change could finish
   generating before an earlier one, so the client sees stale state.
2. updateSentPeers and computePeerDiff race against each other —
   updateSentPeers does Clear() + Store() which is not atomic relative
   to a concurrent Range() in computePeerDiff.

Bundle all pending changes for a node into a single work item so one
worker processes them sequentially. Add a per-node workMu that
serializes processing across consecutive batch ticks, preventing a
second worker from starting tick N+1 while tick N is still in progress.

Fixes #3140
This commit is contained in:
Kristoffer Dalby 2026-03-14 16:09:22 +00:00
parent 4d427cfe2a
commit 60317064fd
3 changed files with 172 additions and 40 deletions

View file

@ -42,6 +42,13 @@ type multiChannelNodeConn struct {
pendingMu sync.Mutex
pending []change.Change
// workMu serializes change processing for this node across batch ticks.
// Without this, two workers could process consecutive ticks' bundles
// concurrently, causing out-of-order MapResponse delivery and races
// on lastSentPeers (Clear+Store in updateSentPeers vs Range in
// computePeerDiff).
workMu sync.Mutex
closeOnce sync.Once
updateCount atomic.Int64