> This case study documents a **real failure mode** (conntrack timeout < Timer H) that can cause exactly the symptoms described. However, in the original production case that prompted this investigation, **the conntrack fix did not resolve the actual blocker**.
>
> **Verification at the host physical NIC** (`tcpdump -i enp1s0`) proved that ACK packets never reached docker-2 at all - the loss was **upstream in the carrier infrastructure** (ClearFly ↔ Twilio seam), not at the conntrack layer.
>
> **Key lesson:** Same symptom (timing-precise call death + Timer H expiry), different root cause. Always verify at the **host NIC level** before concluding that a lower-layer fix (like conntrack) resolved the issue. Container-level pcaps show what reaches userspace but can't prove what never arrived.
>
> The conntrack fix and ACK fast-path improvements remain valuable **defense-in-depth** measures that prevent future issues of this shape.
- **caddy-sip-guardian**: Layer 4 SIP proxy with security features (rate limiting, enumeration detection, bans)
- **Asterisk**: PBX backend running in container
- **Linux conntrack**: Kernel connection tracking for NAT/stateful firewall
## Investigation Timeline
### Layer 1: SIP Protocol Timers (Red Herring)
**Hypothesis:** Asterisk's endpoint timers too aggressive?
```ini
; /etc/asterisk/pjsip.conf endpoints
timers_sess_expires = 1800 ; 30 minutes
timers_min_se = 90 ; 90 seconds
```
**Result:** Ruled out. Asterisk's session timers were set to 30 minutes, far beyond the 64-second failure.
**Key lesson:** Timer H (INVITE transaction timeout) is **32 seconds** per RFC 3261. The 64-second call death is exactly **2× Timer H**, suggesting Timer H fired once, sent BYE, waited another 32s for BYE response, then disconnected.
h.logger.Debug("ACK exempted from enumeration/rate checks (mid-dialog fast-path)",
zap.String("ip", host),
)
if h.guardian.metricsEnabled() {
RecordConnection("allowed")
}
return next.Handle(cx)
}
```
**Result:** Deployed to production... but **calls still died at 64 seconds**.
**Verification:** Packet capture **inside sip-guardian container** showed **zero ACK packets** arriving from Twilio. The ACKs were being dropped **upstream** of sip-guardian, before they reached the userspace layer.
**Key lesson:** The architectural fix was correct and valuable for other deployments, but it didn't solve this specific blocker. The loss was happening at the kernel layer.
---
### Layer 3: UDP Conntrack Timeout (Root Cause)
**Hypothesis:** Linux conntrack expiring UDP flows before SIP transaction completes?
**Investigation on docker-2.supportedsystems.com:**
# Should see ACK packets. If zero ACKs appear at the host NIC,
# the loss is UPSTREAM (carrier/routing), not conntrack.
```
**Why this matters:** In the original production case, zero ACK packets reached the host NIC even after the conntrack fix was applied. The actual blocker was upstream carrier infrastructure (ClearFly ↔ Twilio seam), not conntrack. Container-level pcaps can't prove what never arrived.
Even though the conntrack fix was NOT the production blocker in the original case, we made valuable architectural improvements that remain defense-in-depth measures:
ACKs now bypass ALL security checks (enumeration detection, rate limiting, validation, pattern matching). This prevents false-positive bans in deployments where ACKs DO reach the sip-guardian layer.
**Benefits:**
- Prevents enumeration false-positives from ACK retransmissions
- Prevents rate-limiting of legitimate mid-dialog ACKs
- Improves performance (one less layer of processing per ACK)
- More correct architecturally (ACKs are not a security threat)
- Stopped after fixing the ACK handling (incomplete)
- Never discovered the conntrack issue
### 2. Timing-Precise Failures Point to State Machine Conflicts
The **64-second precision** was the critical clue:
- Not random/intermittent (rules out packet loss, congestion)
- Not variable (rules out load-dependent issues)
- Exactly 2× Timer H (32s)
**Pattern:** When a failure occurs at a mathematically precise interval, look for:
- State machine timeouts
- Timer conflicts between independent subsystems
- Retry logic with exponential backoff hitting a cap
### 3. Packet Captures Don't Lie
The pcap **inside the sip-guardian container** was the smoking gun that proved the ACK never arrived at the userspace layer. Without it, we might have:
- Spent hours debugging sip-guardian code
- Blamed Twilio's implementation
- Never checked the kernel layer
**Lesson:** Capture at multiple layers:
- Wire-level (tcpdump on host interface)
- Container-level (tcpdump inside container)
- Application-level (SIP debug logs)
### 4. Defense in Depth: Fix All Layers
Even though the conntrack timeout was the blocker, the ACK fast-path fix is still valuable:
- Other deployments might not have conntrack issues but DO have enumeration detection
- Architectural correctness matters (ACKs shouldn't be treated as enumeration probes)
- Future-proofs against regressions
**Don't stop at the first fix that works** - fix the architectural issues you find along the way.
