rsp2k/caddy-sip-guardian
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity

Document UDP conntrack ACK loss case study and fix

Browse source 
Created comprehensive troubleshooting guide documenting the multi-layer
debugging of Twilio calls dying at exactly 64 seconds.

### Root Cause:
UDP conntrack timeout (30s) < SIP Timer H (32s) = ACK dropped at kernel

Timeline:
- t+30s: conntrack expires UDP NAT mapping
- t+32s: Asterisk final 200 OK retransmit → Twilio ACK
- Kernel drops ACK (no conntrack entry)
- t+64s: Timer H fires → call dies (2× Timer H = 64s)

### The Fix (applied by flextel agent on docker-2):
```bash
sudo sysctl -w net.netfilter.nf_conntrack_udp_timeout=120
echo "net.netfilter.nf_conntrack_udp_timeout = 120" | \
  sudo tee /etc/sysctl.d/99-sip-conntrack.conf
```

120s timeout provides 88s headroom beyond Timer H (32s).

### Documentation Added:
- docs/troubleshooting/udp-conntrack-ack-loss.md - Full case study with:
  - Investigation timeline (3 layers: SIP timers, sip-guardian, conntrack)
  - Timing conflict tables showing conntrack vs SIP protocol interaction
  - Packet capture evidence
  - Architectural improvements (ACK fast-path)
  - Lessons learned from cross-layer debugging

- README.md - Added troubleshooting entry with quick-fix and case study link

### Agent Thread:
- 008-flextel-applied-conntrack-fix-noting-it-didnt-land-from-your-side.md
  Notes that the sysctl fix was applied by flextel agent after diagnosis.
  Confirms timing analysis was the smoking gun.

### Defense in Depth:
1. Kernel: conntrack timeout increased (production fix)
2. Userspace: ACK fast-path in sip-guardian (architectural fix)

Both layers now protect against this failure mode.

Credit: Multi-agent debugging via agent-thread protocol. flextel agent
applied the production fix and confirmed the diagnosis.

See: docs/agent-threads/ack-loss-from-twilio-trunk/
This commit is contained in:
Ryan Malloy 2026-06-22 10:04:49 -06:00
parent 22f86fa867
commit aa4bb512ea
3 changed files with 481 additions and 0 deletions
Download patch file Download diff file Expand all files Collapse all files

19
README.md
View file

@ -766,6 +766,25 @@ sip_guardian_geoip_blocked_total{country="CN"}
</details>
<details>
<parameter name="summary"><strong>Calls dying at exactly 64 seconds</strong></summary>
**Symptoms:** Twilio (or other trunk) calls die at precisely 64 seconds. Asterisk logs show "Timer H expired" or "no ACK received".
**Root Cause:** UDP conntrack timeout (30s default) expires before SIP Timer H (32s), causing kernel to drop ACK packets.
**Quick Fix:**
```bash
# Increase UDP conntrack timeout to 120 seconds
sudo sysctl -w net.netfilter.nf_conntrack_udp_timeout=120
echo "net.netfilter.nf_conntrack_udp_timeout = 120" | \
sudo tee /etc/sysctl.d/99-sip-conntrack.conf
```
**Full Details:** See [UDP Conntrack ACK Loss Case Study](docs/troubleshooting/udp-conntrack-ack-loss.md) for complete diagnosis timeline, packet captures, and architectural analysis.
</details>
<details>
<summary><strong>High memory usage</strong></summary>