Scenario: Cascading Failure from One Slow Downstream

Classic cascading failure. Slow downstream service blocks caller threads; caller's thread pool exhausts; caller appears down; its callers exhaust next; platform goes down. Walking through the response:

Minute 1 to 3: identify the actual slow service

• Open APM / tracing tool; look at service dependency map
• Tricky: upstream services waiting on the root will also look red
• Ask: which service has high processing time, not wait time?
• Distributed traces show the breakdown; the service with real work happening is the root
• Example: Pricing has 10 seconds of actual processing; Order, Checkout, Gateway are waiting on it

Minute 3 to 5: check what changed

• Recent deploys of the slow service or its DB?
• Traffic spike?
• DB connections maxed?
• Slow query log showing new query?
• Check pods, CPU, memory, GC pauses, thread dumps, DB connection pool

Minute 5 to 8: mitigate — stop the cascade

• Before fixing the root, stop the bleed
• Manually trip circuit breakers on callers' dependency on the slow service
• Callers return fallback (cached data, defaults); fast failures replace thread exhaustion
• Shed load at API Gateway: drop non-critical traffic; keep core flows
• Reduce calls to the slow service; cache aggressively during the incident

Minute 8 to 12: resolve the root cause

• Slow query: kill it, add index
• DB connections exhausted: scale pool, add RDS Proxy
• Recent deploy: roll back
• Traffic spike: scale horizontally
• Downstream of the service is broken: failover

Minute 12 to 15: verify recovery

• Circuit breakers auto-transition to HALF_OPEN; one test request; then CLOSED
• Upstream thread pools drain; latency normalizes
• Error rates drop to baseline

Why certain moves make it worse

• Restarting services: clears thread pool for one second; new requests flood in and block again; returns in seconds
• Scaling out: more caller pods equals more clients of the slow service; amplifies load on its DB
• Rolling back unrelated deploys: wastes time
• Suppressing alerts: tempting but dangerous

Prevention — baked-in resilience

Aggressive timeouts — the number one fix

• Default socket timeout is 30 seconds; audit every HTTP, JDBC, Redis client
• Set 1 second connect, 3 seconds read for internal calls
• 30-second default means 200 blocked calls equals 200 blocked threads for 30 seconds; Tomcat pool dies

Circuit breakers with slow-call detection

• Resilience4j trips on slow calls, not just errors
• Slow-call threshold 2 seconds; slow-call rate 50% trips the circuit
• When downstream gets slow, circuit opens, callers get fallback, thread pools preserved

Bulkheads per dependency

• Limit concurrent in-flight calls per downstream
• 20 max concurrent to Pricing; rest of service keeps serving
• Semaphore bulkhead lightweight; thread-pool bulkhead for critical isolation

Fallbacks

• Cached data, defaults, deferred queues
• Never call another service from fallback
• Slightly stale price beats a 503 error

Load shedding at the edge

• API Gateway drops low-priority traffic during overload
• Health checks, checkout, payment protected
• Analytics, recommendations shed first

Chaos testing

• Inject latency to downstream services regularly
• Verify circuits trip, fallbacks fire, core flows survive
• Netflix Chaos Monkey, Gremlin, AWS Fault Injection Simulator

Key lessons

• Mitigation first, resolution after
• Default timeouts are the silent killer; explicit 2 to 5 seconds
• Restart and scale amplify cascades
• Circuit breakers + bulkheads + tight timeouts make cascades near-impossible
• Observability debt means hours to find root cause; invest in APM
• Postmortem the failure mode, not the trigger; slow dependencies are inevitable, cascades are preventable

The strategic answer

• Incident response: identify root via traces, mitigate with circuits and shedding, resolve root cause, verify recovery
• Prevention: aggressive timeouts, circuit breakers with slow-call detection, bulkheads, fallbacks, chaos testing
• Don't restart or scale during a cascade; both make it worse
• Invest in resilience before the cascade; retrofitting during incident is hard