Circuit Breaker Framework

Prevent cascading failures in distributed systems by detecting when a downstream service is unhealthy and failing fast.

Three States

1. Closed — Normal operation. All calls pass through. Failure metrics monitored.
2. Open — Failing fast. Calls rejected immediately; fallback returned.
3. Half-Open — Probe. Limited test calls to check if downstream recovered.

Python Implementation

class CircuitBreaker:
    def __init__(self, failure_threshold, recovery_timeout, half_open_max_calls):
        self.failure_threshold = failure_threshold
        self.recovery_timeout = recovery_timeout
        self.half_open_max_calls = half_open_max_calls
        self.failure_count = 0
        self.state = "closed"
        self.last_failure_time = None
    
    async def call(self, func, *args, **kwargs):
        if self.state == "open":
            if self._should_attempt_reset():
                self.state = "half-open"
            else:
                return self._fallback()
        
        try:
            result = await func(*args, **kwargs)
            self._on_success()
            return result
        except Exception as e:
            self._on_failure()
            raise
    
    def _on_success(self):
        self.failure_count = 0
        if self.state == "half-open":
            self.state = "closed"
    
    def _on_failure(self):
        self.failure_count += 1
        if self.failure_count >= self.failure_threshold:
            self.state = "open"
            self.last_failure_time = time.time()
    
    def _should_attempt_reset(self):
        return self.last_failure_time and \
               time.time() - self.last_failure_time >= self.recovery_timeout

When to Use

• Services making remote calls over network
• Downstream services have intermittent outages or latency spikes
• Failures could cascade and take down calling services
• Need graceful degradation instead of total failure

Cost Profile

| Category | Assessment | |----------|-----------| | Infrastructure | Low — circuit breaker logic in service code | | Operational | Low — self-regulating once configured | | Cognitive | Medium — understanding when/how to tune |

Real-World Examples

• Hystrix at Netflix: Open-sourced circuit breaker for microservices
• Polly in .NET: Retry and circuit breaker policy library

Alternatives

• Bulkhead Pattern — Isolate resources per dependency
• Timeout & Retry — Time-bound retries without state
• Dependency Isolation — Network-level isolation