Circuit Breaker Pattern: Preventing Cascade Failures in Distributed Systems

The cascade failure it solves

Service A calls service B. B starts responding slowly. Without a breaker:

• A's calls to B time out at 30s instead of 50ms
• A's worker threads pile up waiting
• A's thread pool fills
• Incoming requests to A queue; latency explodes
• A's upstream sees A as slow; cascade continues

B's local issue takes down A and everyone above. Speed of failure beats slow degradation.

With a breaker: after N failures, the breaker on A trips OPEN. All subsequent calls fail instantly with a fallback. A's threads stay free. B can recover without fighting thousands of stuck retries.

Three states

• CLOSED: normal. Calls pass through; track success/failure on a sliding window.
• OPEN: dependency considered broken. Calls fail immediately with a fallback. After cooldown (10-60s typical) → HALF_OPEN.
• HALF_OPEN: cautious probing. Let 3-10 calls through. All succeed → CLOSED. Any fail → back to OPEN.

Half-open avoids hammering a recovering downstream with full traffic.

Trip thresholds — failure rate is better than count

Time-based sliding window: 50% failures in last 60 seconds. Volume-independent; works for both busy and quiet services. Modern default (Resilience4j).

What counts as a failure: exceptions (timeout, connect, 5xx), AND slow calls. A downstream returning 200 OK in 30 seconds is still broken from the caller's perspective — it eats threads. slowCallRateThreshold catches degradation before timeouts.

Resilience4j implementation

CircuitBreakerConfig cfg = CircuitBreakerConfig.custom()
    .failureRateThreshold(50)
    .slowCallRateThreshold(50)
    .slowCallDurationThreshold(Duration.ofSeconds(2))
    .waitDurationInOpenState(Duration.ofSeconds(30))
    .slidingWindowType(SlidingWindowType.TIME_BASED)
    .slidingWindowSize(60)
    .minimumNumberOfCalls(20)
    .build();

minimumNumberOfCalls=20 prevents tripping on first error in low-traffic service.

Pair with the rest of the resilience stack

Circuit breaker alone isn't enough. Order: request → bulkhead → breaker → retry → timeout → call.

• Timeout (always): without it, the breaker can't detect failure
• Retry with exponential backoff + jitter: 2-3 attempts max for transient blips; jitter prevents thundering herd
• Circuit breaker: wraps retry layer
• Bulkhead: each dependency gets its own bounded thread pool; slow downstream X eats only X-budget, others stay responsive
• Fallback: cached, default, degraded, or meaningful 5xx — never silent failure

Fallback choices

• Read paths: cached/stale value with a flag
• Recommendations down: empty list, UI hides section
• Critical writes (payment): rethrow with 503 + Retry-After; don't fake success
• Deferrable writes: queue + 202 Accepted; replay when downstream recovers
• Anti-pattern: silent swallow with no log

Per-service vs per-endpoint

Per-service by default. Per-endpoint when one route has different SLA or is known-fragile. Group endpoints by tier (critical/important/optional).

Breaker is per-process. With 50 instances, each has its own state; they converge during a real outage.

Common mistakes

• No timeout — breaker is meaningless
• Tripping on 4xx — those are caller errors, configure recordException to exclude
• Too-low minimum calls — false positives on first error
• No slow-call threshold — degradation cascades before classic failure trips
• Using breaker as load shedder — wrong tool; use rate limit / bulkhead
• Health check calls through breaker — tripped breaker makes you appear unhealthy; decouple deep from shallow health

Recommendation

Use Resilience4j with time-based sliding window, 50% failure threshold OR 50% slow-call rate threshold (slow = >2s), 30s wait in OPEN, 5 probe calls in HALF_OPEN, minimum 20 calls before any trip. Pair with explicit timeouts on every remote call, retry with exponential backoff + jitter (2-3 attempts), bulkhead per dependency, and a sensible fallback. Track state and trip count via Micrometer/Prometheus. Hystrix is deprecated — Resilience4j is the modern JVM pick.