Rate Limiting Patterns

Algorithms

Algorithm	Accuracy	Burst Handling	Best For
Token Bucket	High	Allows controlled bursts	API rate limiting, traffic shaping
Leaky Bucket	High	Smooths bursts entirely	Steady-rate processing, queues
Fixed Window	Low	Allows edge bursts (2x)	Simple use cases, prototyping
Sliding Window Log	Very High	Precise control	Strict compliance, billing-critical
Sliding Window Counter	High	Good approximation	Production APIs — best tradeoff

Fixed window problem: A user sends the full limit at 11:59 and again at 12:01, doubling the effective rate. Sliding window fixes this.

Token Bucket

Bucket holds tokens up to capacity. Tokens refill at a fixed rate. Each request consumes one.

class TokenBucket:
    def __init__(self, capacity: int, refill_rate: float):
        self.capacity = capacity
        self.tokens = capacity
        self.refill_rate = refill_rate  # tokens per second
        self.last_refill = time.monotonic()

    def allow(self) -> bool:
        now = time.monotonic()
        elapsed = now - self.last_refill
        self.tokens = min(self.capacity, self.tokens + elapsed * self.refill_rate)
        self.last_refill = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

Sliding Window Counter

Hybrid of fixed window and sliding window log — weights the previous window's count by overlap percentage:

def sliding_window_allow(key: str, limit: int, window_sec: int) -> bool:
    now = time.time()
    current_window = int(now // window_sec)
    position_in_window = (now % window_sec) / window_sec

    prev_count = get_count(key, current_window - 1)
    curr_count = get_count(key, current_window)

    estimated = prev_count * (1 - position_in_window) + curr_count
    if estimated >= limit:
        return False
    increment_count(key, current_window)
    return True

---

Implementation Options

Approach	Scope	Best For
In-memory	Single server	Zero latency, no dependencies
Redis (INCR + EXPIRE)	Distributed	Multi-instance deployments
API Gateway	Edge	No code, built-in dashboards
Middleware	Per-service	Fine-grained per-user/endpoint control

Use gateway-level limiting as outer defense + application-level for fine-grained control.

---

HTTP Headers

Always return rate limit info, even on successful requests:

RateLimit-Limit: 1000
RateLimit-Remaining: 742
RateLimit-Reset: 1625097600
Retry-After: 30

Header	When to Include
RateLimit-Limit	Every response
RateLimit-Remaining	Every response
RateLimit-Reset	Every response
Retry-After	429 responses only

429 Response Body

{
  "error": {
    "code": "rate_limit_exceeded",
    "message": "Rate limit exceeded. Maximum 1000 requests per hour.",
    "retry_after": 30,
    "limit": 1000,
    "reset_at": "2025-07-01T12:00:00Z"
  }
}

Never return 500 or 503 for rate limiting — 429 is the correct status code.

---

Rate Limit Tiers

Apply limits at multiple granularities:

Scope	Key	Example Limit	Purpose
Per-IP	Client IP	100 req/min	Abuse prevention
Per-User	User ID	1000 req/hr	Fair usage
Per-API-Key	API key	5000 req/hr	Service-to-service
Per-Endpoint	Route + key	60 req/min on /search	Protect expensive ops

Tiered pricing:

Tier	Rate Limit	Burst	Cost
Free	100 req/hr	10	$0
Pro	5,000 req/hr	100	$49/mo
Enterprise	100,000 req/hr	2,000	Custom

Evaluate from most specific to least specific: per-endpoint > per-user > per-IP.

---

Distributed Rate Limiting

Redis-based pattern for consistent limiting across instances:

def redis_rate_limit(redis, key: str, limit: int, window: int) -> bool:
    pipe = redis.pipeline()
    now = time.time()
    window_key = f"rl:{key}:{int(now // window)}"
    pipe.incr(window_key)
    pipe.expire(window_key, window * 2)
    results = pipe.execute()
    return results[0] <= limit

Atomic Lua script (prevents race conditions):

local key = KEYS[1]
local limit = tonumber(ARGV[1])
local window = tonumber(ARGV[2])
local current = redis.call('INCR', key)
if current == 1 then
    redis.call('EXPIRE', key, window)
end
return current <= limit and 1 or 0

Never do separate GET then SET — the gap allows overcount.

---

API Gateway Configuration

NGINX:

http {
    limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
    server {
        location /api/ {
            limit_req zone=api burst=20 nodelay;
            limit_req_status 429;
        }
    }
}

Kong:

plugins:
  - name: rate-limiting
    config:
      minute: 60
      hour: 1000
      policy: redis
      redis_host: redis.internal

---

Client-Side Handling

Clients must handle 429 gracefully:

async function fetchWithRetry(url: string, maxRetries = 3): Promise<Response> {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    const res = await fetch(url);
    if (res.status !== 429) return res;

    const retryAfter = res.headers.get('Retry-After');
    const delay = retryAfter
      ? parseInt(retryAfter, 10) * 1000
      : Math.min(1000 * 2 ** attempt, 30000);
    await new Promise(r => setTimeout(r, delay));
  }
  throw new Error('Rate limit exceeded after retries');
}

• Always respect Retry-After when present
• Use exponential backoff with jitter when absent
• Implement request queuing for batch operations

---

Monitoring

Track these metrics:

• Rate limit hit rate — % of requests returning 429 (alert if >5% sustained)
• Near-limit warnings — requests where remaining < 10% of limit
• Top offenders — keys/IPs hitting limits most frequently
• Limit headroom — how close normal traffic is to the ceiling
• False positives — legitimate users being rate limited

---

Anti-Patterns

Anti-Pattern	Fix
Application-only limiting	Always combine with infrastructure-level limits
No retry guidance	Always include Retry-After header on 429
Inconsistent limits	Same endpoint, same limits across services
No burst allowance	Allow controlled bursts for legitimate traffic
Silent dropping	Always return 429 so clients can distinguish from errors
Global single counter	Per-endpoint counters to protect expensive operations
Hard-coded limits	Use configuration, not code constants

---

NEVER Do

1. NEVER rate limit health check endpoints — monitoring systems will false-alarm
2. NEVER use client-supplied identifiers as sole rate limit key — trivially spoofed
3. NEVER return 200 OK when rate limiting — clients must know they were throttled
4. NEVER set limits without measuring actual traffic first — you'll block legitimate users or set limits too high to matter
5. NEVER share counters across unrelated tenants — noisy neighbor problem
6. NEVER skip rate limiting on internal APIs — misbehaving internal services can take down shared infrastructure
7. NEVER implement rate limiting without logging — you need visibility to tune limits and detect abuse