Configure rate limiting and circuit breaking in API gateways

Imagine a misbehaving client sending 1,000 requests each second. The backend saturates. A downstream service becomes unresponsive. Queue buildup causes cascading failures. How do you protect an api gateway from traffic spikes and service failures? Two proven patterns help: rate limiting and circuit breaking. GitHub limits authenticated api users to 5,000 requests per hour. A circuit breaker stops requests to failing services. The server returns HTTP 429 when clients exceed limits. This guide shows how to configure rate limiting using token bucket algorithms and Redis counters. Implement a circuit breaker with closed, open, and half-open states. By the end, configure both policies on your gateway using YAML or code. Prevent overload and enable graceful recovery.

Fundamentals of rate limiting and circuit breaking

Rate limiting and circuit breaking solve different problems, and you need both for a resilient api gateway. Rate limiting controls how much traffic enters your system. A circuit breaker decides whether requests should reach a failing dependency at all. The table below shows the core differences.

When to use rate limiting

Apply rate limiting when one client can starve others. A public API with many consumers needs fair access, so you cap each client’s request count within a time window. The same logic protects a login endpoint from brute-force attempts and stops scrapers from draining your capacity.

You should also configure rate limiting to control costs. Sudden spikes in usage can burn through infrastructure budgets fast. Limits tied to API keys, OAuth tokens, or usage plans let you offer free tiers with low caps and paid tiers with higher throughput. This keeps queues stable and latency predictable for everyone.

When to use circuit breaking

Reach for the circuit breaker pattern when a downstream service starts failing. Calls to external ML APIs, payment processors, or inventory services can time out and tie up your threads. Without protection, every request waits the full timeout period, connection pools exhaust, and the failure spreads across your microservices.

The circuit breaker pattern watches for elevated failure rates, slow calls, and timeouts. Once failures cross your threshold, the breaker trips to Open and returns fallback responses immediately. After a cooldown, it moves to Half-Open and tests recovery with limited requests. This state machine isolates failures and supports automatic recovery. Use it in microservice architectures wherever a flaky dependency could trigger cascading failures.

Configure rate limiting on your API gateway

Real-world APIs show how limits look in practice. GitHub caps authenticated users at 5,000 requests per hour. These numbers reflect each platform’s capacity and fairness goals. You can configure rate limiting on your own api gateway with similar logic. The gateway counts incoming requests and rejects excess calls once a client crosses the threshold.

Redis often stores these request counters. A centralized store keeps counts consistent across multiple gateway instances. When a client exceeds the limit, the gateway returns HTTP 429 Too Many Requests. The response can include headers like X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset to tell the client when to retry. This approach works for both per-route limits and global limits across all routes.

Choosing token bucket vs leaky bucket

Two algorithms dominate rate limiting design. Each one shapes traffic differently. The table below compares them.

Token bucket parameters include the token fill rate and the bucket capacity. The fill rate measures tokens added per second. The bucket capacity sets the maximum tokens the bucket can hold. You choose these values based on your backend’s tolerance for bursts.

Pick token bucket when your clients need occasional bursts. Pick leaky bucket when your downstream services require a steady, predictable flow. If traffic is bursty, token bucket or a sliding window counter works well. If traffic is uniform or you need strict output rate enforcement, leaky bucket is the better choice.

Per-route and global limits with YAML

You can define rate limits per route or globally across all routes. Per-route limits apply to specific endpoints. Global limits enforce a ceiling across every route in your gateway. Redis acts as the shared counter store for distributed enforcement.

Here is a YAML configuration for a RequestRateLimiter filter that references Redis:

spring:
  cloud:
    gateway:
      routes:
      - id: auditflow_service_route
        uri: http://localhost:8080
        predicates:
        - Path=/api/v1/audit/**
        filters:
        - name: RequestRateLimiter
          args:
            key-resolver: "#{@ipAddressKeyResolver}"
            redis-rate-limiter.replenishRate: 10
            redis-rate-limiter.burstCapacity: 20
            redis-rate-limiter.requestedTokens: 1

The replenishRate sets how many tokens refill each second. The burstCapacity defines the maximum tokens available. The requestedTokens value specifies how many tokens each request consumes. The key-resolver determines which identifier the limiter uses, such as an IP address or user ID.

This configuration applies the limit to the /api/v1/audit/** path. You can add similar filters to other routes with different values. A global limit would use a shared key across all routes. When the limit is exceeded, the gateway returns HTTP 429 with a Retry-After header. This tells the client exactly when to try again.

The circuit breaker pattern complements rate limiting by handling service failures rather than request volume. Rate limiting controls how many requests enter your system. A circuit breaker stops requests from reaching a failing dependency. Together, they protect your gateway from both traffic spikes and cascading failures.

Implement circuit breaking on your API gateway

Now you implement the circuit breaker pattern. It stops requests to a failing downstream service. Your API gateway runs an inbound policy to check circuit state. It also runs an outbound policy to update that state. The state machine transitions between three states: closed, open, and half-open. This mechanism complements rate limiting. Rate limiting controls request volume at the entry point. Circuit breaking stops traffic in the call chain, but only when a service is failing.

Failure thresholds and cooldown periods

A circuit breaker relies on several key parameters. The failure threshold defines how many consecutive failures trigger the open state. For quick testing, you can set this value to 3. For production, a threshold of 5 consecutive failures provides a good balance. This setting avoids false positives from brief network hiccups while still detecting real outages promptly. The timeout, or cooldown period, determines how long the circuit stays open before attempting recovery. A production cooldown of 60 seconds works well — long enough for most provider incidents to resolve yet short enough to avoid blocking traffic after a transient issue. The retry time period resets the timer if recovery fails.

In circuit breaking, setting thresholds too low causes false positives. The circuit breaker trips on normal blips that would resolve on their own. Setting thresholds too high causes slow detection. The circuit breaker fails to protect until real damage happens. You must configure these values per project based on your service tolerance.

Half-open state for health recovery

After the timeout timer expires, the circuit breaker transitions to half-open. This state allows the circuit breaker to send a limited number of probe requests to the downstream service. The goal is to test whether the problem is fixed. If these probe requests succeed, the circuit breaker transitions back to closed and normal operations resume. The failure counter resets. If any probe request fails, the circuit breaker immediately returns to open and restarts the timeout timer. This approach prevents the thundering herd problem. A still-fragile service would otherwise receive a sudden flood of traffic.

The half-open timeout configuration deserves careful attention. You set the retry time period too short, and you hammer a service that has not recovered. You set it too long, and you block legitimate traffic for no reason. Testing best practices include starting conservative. Use a rolling window for failure counting, such as 5 failures in 60 seconds. Gradually adjust based on observed behavior.

The table below summarizes the state machine transitions.

A circuit breaker and rate limiting work together. Rate limiting controls input at the front. Circuit breaking cuts fault propagation in the middle of the call chain. Do not retry when the circuit is open. This prevents retry storms and cascading failures.

Combine both patterns for resilient APIs

Execution order and policy precedence

You now have two policies. The order you apply them matters. Rate limiting should run first at the gateway level. This throttles excessive clients before their requests consume any downstream capacity. Circuit breaking runs later in the chain. It stops requests to services already known to be failing.

The table below shows how each pattern contributes to resilience and why the ordering works.

A practical rollout follows a clear sequence. Add rate limiting first. It is the simplest pattern with the biggest impact. Set appropriate timeouts for every backend connection. Implement circuit breakers for critical or failure-prone backends. Add cached fallbacks for read-heavy endpoints. Monitor and iterate using gateway observability data.

Observability for gradual recovery

You cannot tune what you cannot see. Track retry attempts per dependency. Watch circuit breaker open events and their duration. Count the requests rejected or delayed by throttling. Measure average, P95, and P99 response times. Monitor error rate per endpoint and per build version. Compare requests per minute against 429 responses. Log circuit breaker open events per service. Flag retry count spikes for a specific dependency.

Start conservative. Set low limits and low failure thresholds. Observe behavior before adjusting. Establish a baseline by measuring peak connections, requests, and error rates. Set initial thresholds at twice the peak values with moderate outlier detection. Monitor overflows and ejections over one to two weeks. Tighten thresholds if no overflows occur during normal traffic. Loosen them if legitimate requests are rejected. Run load tests in staging to verify protection under simulated overload. Repeat the cycle when traffic patterns or service capacity change.

Rate limiting controls request volume. Circuit breaking handles service failures. Together, they create a resilient api gateway. You now hold both tools.

Set your thresholds from load tests and failure objectives, not guesswork. Internal limits define what your system can truly absorb.

Start with conservative values. Watch your observability data. Tune gradually as real traffic patterns emerge. A circuit breaker rewards patience, not aggression.

Review your gateway configuration today. Apply these patterns in your YAML or code. Your next traffic surge or outage will not wait. Prepare your requests handling before it arrives.

FAQ

Should rate limiting run before circuit breaking?

Yes. Apply rate limiting first at the gateway. It throttles heavy clients before their api calls consume downstream capacity. Circuit breaking then runs later in the chain. This order stops retries from eating your rate-limit quota and keeps failing services from taking more load.

How do I pick a failure threshold for my circuit breaker?

Start with 5 consecutive failures and a 60-second cooldown. Watch your error rate and queue depth. Lower the threshold if detection feels slow. Raise it if normal blips trip the breaker. Tune with load tests, not guesswork.

What does HTTP 429 mean for my clients?

HTTP 429 means the client sent too many requests. The gateway rejects the excess and returns a Retry-After header. Share that header in your docs. Clients then back off instead of hammering your endpoint again.

Do I really need both patterns?

Yes. Rate limiting caps inbound volume. Circuit breaking isolates a failing dependency. One controls how much traffic enters. The other decides whether traffic reaches a broken service. Together they prevent overload and cascading failures.