Timeout and Retry Framework

Handle transient failures in distributed systems by implementing controlled timeouts and retry strategies that prevent cascading failures while respecting downstream service capacity.

Core Components

1. Timeout

Sets a maximum duration a call should wait for a response. If the timeout expires, the call is abandoned immediately. Prevents hanging threads and resource exhaustion. Typical timeouts:

• RPC calls: 50-200ms
• Database queries: 1-5s
• External APIs: 1-10s

2. Retry with Backoff

Retries failed calls after a waiting period. Backoff strategies prevent synchronized retry storms:

• Exponential backoff: 100ms, 200ms, 400ms, 800ms, ...
• Jitter: Add randomness (±20%) to avoid thundering herd
• Maximum retry attempts: 3 attempts maximum
• Total deadline: Stop retrying after 5-10s total

Recommended Strategy

| Parameter | Value | |-----------|-------| | Timeout | 50-200ms for RPC, 1-5s for DB | | Retry count | 3 attempts max | | Backoff | Exponential with jitter | | Circuit Breaker | Wrap retries for double protection |

When to Use

• Calling any remote service over the network
• Downstream services experience intermittent failures
• Operations are idempotent or can be made idempotent
• Circuit breaker is already in place

When Not to Use

• Non-idempotent operations (financial transactions)
• Permanently unavailable downstream
• Real-time systems where retry latency violates SLAs
• Without circuit breaker protection

Cost Profile

| Category | Assessment | |----------|-----------| | Infrastructure | Low — client-side timeout and retry logic | | Operational | Low — once configured, self-regulating | | Cognitive | Medium — choosing appropriate values |

Real-World Examples

• Hystrix retry configuration at Netflix: Open-sourced circuit breaker and retry logic
• Polly retry policies in AWS: Well-architected reliability lens guidelines

Alternatives

• Circuit Breaker — Fail fast when downstream is overwhelmed
• Bulkhead Pattern — Isolate resources per dependency
• Dependency Isolation — Network-level isolation

Implementation Example

from tenacity import retry, stop_after_attempt, wait_exponential, stop_after_delay

@retry(
    stop_after_attempt=3,
    wait=wait_exponential(multiplier=0.1, min=100, max=1000),
    stop_after_delay=5.0
)
def call_service():
    response = http_client.get(url, timeout=0.5)
    if response.status_code >= 500:
        raise Exception("Transient error")
    return response