Post-Mortem: How Karpenter v1.5.3 and a Missing PDB Caused a 10-Minute Outage! 🚨

---

Hey everyone! Today, I’m sharing a detailed post-mortem on a critical 10-minute service failure we experienced. 😱 It all started when our Kubernetes auto-scaler, Karpenter, decided to consolidate three nodes at once, taking down over 20 pods simultaneously. Here’s what happened and what we learned.

---

🔍 Root Cause Analysis (RCA)

This failure was a “perfect storm” 🌪️ caused by several factors aligning at the worst possible time:

1. Aggressive Karpenter Policy: We had recently changed our consolidationPolicy to the more aggressive WhenEmptyOrUnderutilized and upgraded to v1.5.3, which has a more assertive consolidation algorithm.
2. Missing PodDisruptionBudget (PDB): This was the killer. 🎯 We had no PDBs configured for any service, including our critical Config Server.
3. Cascading Failure (Dependency Hell): When the Config Server pod was terminated, all other services failed to start. Why? They couldn’t fetch their configurations, leading to a system-wide meltdown. domino:
4. Short Probe Timeouts: Our readinessProbe was set to only 60 seconds, which wasn’t nearly enough time for our complex Spring Boot apps to initialize properly. ⏳

---

🛠️ Solutions and Action Plan

We broke down our recovery into immediate, short-term, and mid-term actions.

• Immediate (P0) 🔥:
  • Apply PDBs (Mandatory!): We immediately added PodDisruptionBudget to all production services. This means minAvailable: 1 for critical services (like Config Server) and maxUnavailable (e.g., 1 or 33%) for others.
  • Revert Karpenter Policy: We switched the consolidationPolicy back to the safer WhenEmpty and are now implementing schedule-based consolidation (only during off-peak hours). 🌙
• Short-Term (P1) 🏃‍♂️:
  • Improve Probes: We increased the initialDelaySeconds to 120s for readinessProbe and added a startupProbe to give our apps plenty of time to start.
  • Add PreStop Hook: We implemented a preStop hook (e.g., sleep 15) to ensure graceful shutdowns and connection draining.
• Mid-Term (P2) 🏗️:
  • Decouple Dependencies: We’re configuring services with spring.cloud.config.fail-fast: false. Now, services can start even if the Config Server is temporarily down.
  • Implement Resiliency: We’re strengthening our systems with Circuit Breaker patterns (with fallbacks) and adding dependency checks (DB, Config Server) to our application health indicators.

---

📚 Lessons Learned

This incident was painful, but the lessons are invaluable:

• PDBs are NOT optional. They are mandatory for any production workload. Full stop. ✅
• Be conservative with infrastructure. Auto-scaling settings should be safe by default, not aggressive.
• Design for failure. Assume dependencies will fail and build resiliency (like circuit breakers and fallbacks) from day one.

Hope this breakdown helps someone else avoid the same “perfect storm”!

References : https://twodragon.tistory.com/m/695

---