Update case study: conntrack fix was defense-in-depth, not root cause
Agent thread message 010 revealed critical update: conntrack fix didn't resolve the production blocker. ### What Actually Happened: Host physical NIC pcap (tcpdump -i enp1s0) showed ZERO ACK packets arriving at docker-2. The loss is upstream in carrier infrastructure (ClearFly ↔ Twilio seam), not at the conntrack layer. ### Documentation Updates: - Added prominent warning box at top of case study - Updated verification section to emphasize host-NIC pcap FIRST - Clarified that conntrack + ACK fast-path are defense-in-depth, not fixes ### Key Lesson: Same symptom (Timer H expiry at 64s), different root cause. Container pcaps show what reaches userspace but can't prove what never arrived. Always verify at the physical NIC layer before concluding a lower-layer fix worked. ### Still Valuable: - Conntrack 30→120s: prevents future UDP timeout issues - ACK fast-path: architecturally correct, prevents security pipeline drops - Case study: documents real failure mode worth knowing about The diagnostic convergence was sound; the actual blocker was one layer further up the stack. See: docs/agent-threads/ack-loss-from-twilio-trunk/010-*
This commit is contained in:
parent
183ef98b2a
commit
f73823448f
2 changed files with 143 additions and 4 deletions
|
|
@ -1,5 +1,17 @@
|
|||
# Case Study: UDP Conntrack Timeout Causing ACK Loss and Call Death at 64 Seconds
|
||||
|
||||
> **⚠️ IMPORTANT UPDATE (2026-06-22):**
|
||||
>
|
||||
> This case study documents a **real failure mode** (conntrack timeout < Timer H) that can cause exactly the symptoms described. However, in the original production case that prompted this investigation, **the conntrack fix did not resolve the actual blocker**.
|
||||
>
|
||||
> **Verification at the host physical NIC** (`tcpdump -i enp1s0`) proved that ACK packets never reached docker-2 at all - the loss was **upstream in the carrier infrastructure** (ClearFly ↔ Twilio seam), not at the conntrack layer.
|
||||
>
|
||||
> **Key lesson:** Same symptom (timing-precise call death + Timer H expiry), different root cause. Always verify at the **host NIC level** before concluding that a lower-layer fix (like conntrack) resolved the issue. Container-level pcaps show what reaches userspace but can't prove what never arrived.
|
||||
>
|
||||
> The conntrack fix and ACK fast-path improvements remain valuable **defense-in-depth** measures that prevent future issues of this shape.
|
||||
>
|
||||
> See: [Agent Thread Message 010](../agent-threads/ack-loss-from-twilio-trunk/010-flextel-update-conntrack-didnt-resolve-loss-is-upstream.md)
|
||||
|
||||
## Problem Statement
|
||||
|
||||
Twilio trunk calls through caddy-sip-guardian were dying at exactly **64 seconds** after answer. The failure was 100% reproducible and timing-precise.
|
||||
|
|
@ -204,7 +216,24 @@ The conntrack entry now survives the entire Timer H window (32s) with **88 secon
|
|||
|
||||
## Verification
|
||||
|
||||
### Confirm the fix took effect
|
||||
### ⚠️ CRITICAL: Verify at the Host NIC Level FIRST
|
||||
|
||||
**Before assuming the conntrack fix worked**, capture at the physical NIC to prove ACKs are actually arriving:
|
||||
|
||||
```bash
|
||||
# On the docker host (NOT inside containers)
|
||||
sudo tcpdump -i enp1s0 -nn -s 0 -w /tmp/host-nic-sip.pcap port 5060
|
||||
|
||||
# During active test call, filter for ACK:
|
||||
sudo tshark -r /tmp/host-nic-sip.pcap -Y 'sip.Method == "ACK"'
|
||||
|
||||
# Should see ACK packets. If zero ACKs appear at the host NIC,
|
||||
# the loss is UPSTREAM (carrier/routing), not conntrack.
|
||||
```
|
||||
|
||||
**Why this matters:** In the original production case, zero ACK packets reached the host NIC even after the conntrack fix was applied. The actual blocker was upstream carrier infrastructure (ClearFly ↔ Twilio seam), not conntrack. Container-level pcaps can't prove what never arrived.
|
||||
|
||||
### Confirm the conntrack fix took effect
|
||||
|
||||
```bash
|
||||
# Check current value
|
||||
|
|
@ -214,18 +243,22 @@ cat /proc/sys/net/netfilter/nf_conntrack_udp_timeout
|
|||
# Verify persistent config
|
||||
cat /etc/sysctl.d/99-sip-conntrack.conf
|
||||
# Should contain: net.netfilter.nf_conntrack_udp_timeout = 120
|
||||
```
|
||||
|
||||
# Test call
|
||||
# 1. Place call to Twilio trunk
|
||||
### Test call verification (only meaningful if host NIC shows ACKs arriving)
|
||||
|
||||
```bash
|
||||
# 1. Place call to trunk
|
||||
# 2. Let it run for >64 seconds
|
||||
# 3. Verify call stays active
|
||||
# 4. Check Asterisk logs - should NOT show Timer H expiry
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Architectural Improvements Made During Investigation
|
||||
|
||||
Even though the conntrack fix was the production blocker, we made valuable architectural improvements:
|
||||
Even though the conntrack fix was NOT the production blocker in the original case, we made valuable architectural improvements that remain defense-in-depth measures:
|
||||
|
||||
### 1. ACK Fast-Path in sip-guardian
|
||||
|
||||
|
|
|
|||
Loading…
Add table
Add a link
Reference in a new issue