mapper/batcher: serialize per-node work to prevent out-of-order delivery
processBatchedChanges queued each pending change for a node as a separate work item. Since multiple workers pull from the same channel, two changes for the same node could be processed concurrently by different workers. This caused two problems: 1. MapResponses delivered out of order — a later change could finish generating before an earlier one, so the client sees stale state. 2. updateSentPeers and computePeerDiff race against each other — updateSentPeers does Clear() + Store() which is not atomic relative to a concurrent Range() in computePeerDiff. Bundle all pending changes for a node into a single work item so one worker processes them sequentially. Add a per-node workMu that serializes processing across consecutive batch ticks, preventing a second worker from starting tick N+1 while tick N is still in progress. Fixes #3140
This commit is contained in:
parent
4d427cfe2a
commit
60317064fd
3 changed files with 172 additions and 40 deletions
|
|
@ -42,6 +42,13 @@ type multiChannelNodeConn struct {
|
|||
pendingMu sync.Mutex
|
||||
pending []change.Change
|
||||
|
||||
// workMu serializes change processing for this node across batch ticks.
|
||||
// Without this, two workers could process consecutive ticks' bundles
|
||||
// concurrently, causing out-of-order MapResponse delivery and races
|
||||
// on lastSentPeers (Clear+Store in updateSentPeers vs Range in
|
||||
// computePeerDiff).
|
||||
workMu sync.Mutex
|
||||
|
||||
closeOnce sync.Once
|
||||
updateCount atomic.Int64
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue