Sign in Subscribe
3 min read

Building an API Gateway That Can Survive Heavy Traffic

Share

An API gateway is the single entry point for client requests into a microservice architecture. It handles routing, authentication, rate limiting, and load balancing. When traffic spikes, the gateway is the first component to feel the pressure. This article explores how to design an API gateway that remains stable under heavy load.

Key sources: "Building Microservices" by Sam Newman, Kong and Envoy documentation, engineering blogs from Netflix and Uber.

What Does an API Gateway Do?

In a microservice architecture, clients should not need to know about every individual service. The API gateway provides a unified interface:

  • Request routing: Routes incoming requests to the appropriate microservice
  • Authentication: Validates tokens, API keys, or certificates before requests reach internal services
  • Rate limiting: Prevents any single client from overwhelming the system
  • Load balancing: Distributes requests across multiple service instances
  • Response aggregation: Combines responses from multiple services into a single response
  • Protocol translation: Converts between protocols (HTTP, gRPC, WebSocket)

Without a gateway, each client must handle authentication, discover service locations, and manage multiple endpoints independently.

The Scalability Challenge

An API gateway is a bottleneck by design — all traffic flows through it. Under heavy traffic, several problems emerge:

  • Connection exhaustion: Each incoming connection consumes resources. At scale, the gateway runs out of TCP connections.
  • Latency amplification: If the gateway adds 10 ms of processing per request, and there are 10,000 requests per second, the aggregate latency impact is significant.
  • Memory pressure: Storing sessions, rate limit counters, and request contexts consumes memory.
  • Single point of failure: If the gateway crashes, no requests reach any service.

Strategies for Scaling an API Gateway

Horizontal Scaling with Load Balancers

Run multiple gateway instances behind a load balancer. Each instance handles a portion of traffic. If one fails, traffic is redirected.

Clients → DNS → Load Balancer → Gateway Instance 1
                                  Gateway Instance 2
                                  Gateway Instance N

The load balancer itself must be highly available. Use a pair of load balancers in active-passive or active-active configuration.

Stateless Design

Store session data externally (Redis, memcached) instead of in-memory. This allows any gateway instance to handle any request. Instances can be added or removed without affecting active sessions.

Stateful (problematic): - Session stored in local memory - Request must return to the same instance (sticky sessions) - Adding or removing instances disrupts sessions

Stateless (scalable): - Session stored in Redis - Any instance can handle any request - Instances are interchangeable

Rate Limiting

Rate limiting protects the gateway itself and downstream services. Implement it at multiple levels:

  • Global rate limit: Maximum total requests per second across all clients
  • Per-client rate limit: Maximum requests per second per API key or IP address
  • Per-endpoint rate limit: Different limits for different endpoints

Rate limiting algorithms:

| Algorithm | Behavior | Best For | |-----------|----------|----------| | Token bucket | Tokens refill at a fixed rate. Requests consume tokens. | Bursty traffic | | Leaky bucket | Requests are processed at a fixed rate. Excess is queued. | Smooth traffic shaping | | Sliding window | Counts requests in a rolling time window. | Accurate rate tracking | | Fixed window | Resets a counter at fixed intervals. | Simple implementation |

Caching at the Gateway

Cache responses for idempotent requests (GET, HEAD, OPTIONS). This reduces load on downstream services and improves response time.

Client → Gateway → Cache check → Cache hit? → Return cached response
                                  Cache miss? → Forward to service → Cache response → Return

Use CDN-level caching for static responses and gateway-level caching for dynamic but cacheable responses.

Circuit Breaking

If a downstream service becomes slow or unresponsive, the gateway should stop sending requests to it rather than waiting indefinitely.

The circuit breaker has three states:

  1. Closed: Requests pass through normally
  2. Open: Requests fail immediately without reaching the service
  3. Half-open: A probe request is sent periodically to check if the service has recovered

This prevents cascading failures and allows the gateway to remain responsive even when individual services fail.

Production Example

Netflix's API gateway handles billions of requests per day. Their approach includes:

  • Zuul: Their gateway is a JVM-based application running on multiple instances
  • Eureka: Service discovery so the gateway knows which service instances are healthy
  • Hystrix: Circuit breaking to isolate failures
  • Ribbon: Client-side load balancing to distribute requests across service instances

Traffic spikes (such as a new season release) are absorbed by auto-scaling gateway instances based on CPU utilization and request latency.

Key Takeaways

  1. An API gateway is a single entry point that handles routing, authentication, and rate limiting.
  2. Make the gateway stateless by storing session data externally.
  3. Rate limiting at multiple levels protects the gateway and downstream services.
  4. Circuit breaking prevents slow services from degrading the entire system.
  5. Horizontal scaling with auto-scaling is essential for handling traffic spikes.
  6. Netflix's Zuul, Kong, and Envoy are production-tested API gateway implementations.

Design principle: The API gateway should be the only component that knows about all services. Everything else should communicate through it.

Published by:

Iman Suparman