Update case study: conntrack fix was defense-in-depth, not root cause

Agent thread message 010 revealed critical update: conntrack fix didn't
resolve the production blocker.

### What Actually Happened:
Host physical NIC pcap (tcpdump -i enp1s0) showed ZERO ACK packets
arriving at docker-2. The loss is upstream in carrier infrastructure
(ClearFly ↔ Twilio seam), not at the conntrack layer.

### Documentation Updates:
- Added prominent warning box at top of case study
- Updated verification section to emphasize host-NIC pcap FIRST
- Clarified that conntrack + ACK fast-path are defense-in-depth, not fixes

### Key Lesson:
Same symptom (Timer H expiry at 64s), different root cause. Container
pcaps show what reaches userspace but can't prove what never arrived.
Always verify at the physical NIC layer before concluding a lower-layer
fix worked.

### Still Valuable:
- Conntrack 30→120s: prevents future UDP timeout issues
- ACK fast-path: architecturally correct, prevents security pipeline drops
- Case study: documents real failure mode worth knowing about

The diagnostic convergence was sound; the actual blocker was one layer
further up the stack.

See: docs/agent-threads/ack-loss-from-twilio-trunk/010-*
This commit is contained in:
Ryan Malloy 2026-06-22 16:16:29 -06:00
parent 183ef98b2a
commit f73823448f
2 changed files with 143 additions and 4 deletions

View file

@ -1,5 +1,17 @@
# Case Study: UDP Conntrack Timeout Causing ACK Loss and Call Death at 64 Seconds
> **⚠️ IMPORTANT UPDATE (2026-06-22):**
>
> This case study documents a **real failure mode** (conntrack timeout < Timer H) that can cause exactly the symptoms described. However, in the original production case that prompted this investigation, **the conntrack fix did not resolve the actual blocker**.
>
> **Verification at the host physical NIC** (`tcpdump -i enp1s0`) proved that ACK packets never reached docker-2 at all - the loss was **upstream in the carrier infrastructure** (ClearFly ↔ Twilio seam), not at the conntrack layer.
>
> **Key lesson:** Same symptom (timing-precise call death + Timer H expiry), different root cause. Always verify at the **host NIC level** before concluding that a lower-layer fix (like conntrack) resolved the issue. Container-level pcaps show what reaches userspace but can't prove what never arrived.
>
> The conntrack fix and ACK fast-path improvements remain valuable **defense-in-depth** measures that prevent future issues of this shape.
>
> See: [Agent Thread Message 010](../agent-threads/ack-loss-from-twilio-trunk/010-flextel-update-conntrack-didnt-resolve-loss-is-upstream.md)
## Problem Statement
Twilio trunk calls through caddy-sip-guardian were dying at exactly **64 seconds** after answer. The failure was 100% reproducible and timing-precise.
@ -204,7 +216,24 @@ The conntrack entry now survives the entire Timer H window (32s) with **88 secon
## Verification
### Confirm the fix took effect
### ⚠️ CRITICAL: Verify at the Host NIC Level FIRST
**Before assuming the conntrack fix worked**, capture at the physical NIC to prove ACKs are actually arriving:
```bash
# On the docker host (NOT inside containers)
sudo tcpdump -i enp1s0 -nn -s 0 -w /tmp/host-nic-sip.pcap port 5060
# During active test call, filter for ACK:
sudo tshark -r /tmp/host-nic-sip.pcap -Y 'sip.Method == "ACK"'
# Should see ACK packets. If zero ACKs appear at the host NIC,
# the loss is UPSTREAM (carrier/routing), not conntrack.
```
**Why this matters:** In the original production case, zero ACK packets reached the host NIC even after the conntrack fix was applied. The actual blocker was upstream carrier infrastructure (ClearFly ↔ Twilio seam), not conntrack. Container-level pcaps can't prove what never arrived.
### Confirm the conntrack fix took effect
```bash
# Check current value
@ -214,18 +243,22 @@ cat /proc/sys/net/netfilter/nf_conntrack_udp_timeout
# Verify persistent config
cat /etc/sysctl.d/99-sip-conntrack.conf
# Should contain: net.netfilter.nf_conntrack_udp_timeout = 120
```
# Test call
# 1. Place call to Twilio trunk
### Test call verification (only meaningful if host NIC shows ACKs arriving)
```bash
# 1. Place call to trunk
# 2. Let it run for >64 seconds
# 3. Verify call stays active
# 4. Check Asterisk logs - should NOT show Timer H expiry
```
---
## Architectural Improvements Made During Investigation
Even though the conntrack fix was the production blocker, we made valuable architectural improvements:
Even though the conntrack fix was NOT the production blocker in the original case, we made valuable architectural improvements that remain defense-in-depth measures:
### 1. ACK Fast-Path in sip-guardian