Rate limiting is a critical technique in System Design that controls the frequency of requests sent to the Server. It is essential for:

• Preventing System Overload

• Mitigating malicious attacks, such as DDoS

and mostly these can be accomplished using Static rate limiting.

But I was recently working with a client where we need to do Rate limiting dynamically based on the user’s purchased plan and Admin settings for that client.

Here is the simplified, working version of the Sliding Window protocol implemented using Redis — Github Link

Example Requests

1. User A had purchased credits to access the API “/orders” at the rate of 30 requests/min. If A sends more requests than this, then we will not process the request in that 1 min window.

2. User B had purchased credits to access the API “/trade-volume” at the rate of 100 requests / second

As you can see, these are the Dynamic Rate-limiting requirements,

1. What is the API that has to be restricted?

2. What are the rates at which the user has to access the API?

Options we considered

1. Cloudflare Workers to Intercept the requests

2. GCP Compute Engine with Cloud Armor for Rate Limiting

3. Implement custom logic in Nginx web server with Redis as the Data Store

Though the first 2 options are simple to set up & manage, either it is very costly or doesn’t fulfill our Dynamic Rate limiting requirement.

So we went with option 3 — Write a custom Lua script to do Rate limiting.

High-Level Approach with Pseudocode

1. Use OpenResty, which is Nginx loaded with a Lua JIT engine. This helps in running Lua scripts inside our Nginx Web Server.

server {
      listen       80;
      server_name  localhost;

      location /api/ {
          access_by_lua_file /usr/local/openresty/nginx/conf/rate_limit.lua;
          proxy_pass <backend api>;
          
          proxy_set_header Host $host;
          proxy_set_header X-Real-IP $remote_addr;
          proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      }
  }

2. Use Redis to store user configurations and maintain a sliding window for rate limiting. Module “resty.redis” is used to connect to Redis Server.

local redis = require "resty.redis"

local red = redis:new()
red:set_timeout(1000) -- 1 second

local ok, err = red:connect("<Redis Host>", 6379)
if not ok then
    ngx.log(ngx.ERR, "Failed to connect to Redis: ", err)
    return ngx.exit(500)
end

3. We will use Hash Data structure in Redis for storing the configuration details. For every API Key, we will store the Max Request allowed along with the Sliding Window duration.

rateLimit:api_key_1 { max_requests => 100, window_duration => 60 }

# Redis CLI command
HSET rateLimit:api_key_1 max_requests 100 window_duration 60

4. Implement rate limiting based on API keys instead of IP addresses. This allows users to access APIs from different systems using the same purchased plan.

local headers = ngx.req.get_headers()
local api_key = headers["api-key"]

if not api_key or api_key == ngx.null then
    ngx.log(ngx.ERR, "API key not provided: ", err)
    return ngx.exit(403)
end

5. In Redis, maintain a window of valid requests within a specific time period. Expire requests that fall outside the current window using “zremrangebyscore” redis function.

local trim_time = current_time - window
local res, err = red:zremrangebyscore(key, "-inf", trim_time)
if not res then
    ngx.log(ngx.ERR, "Error removing outdated entries: ", err)
    return ngx.exit(500)
end

-- Get the current request count
res, err = red:zcard(key)
if not res then
    ngx.log(ngx.ERR, "Error getting request count: ", err)
    return ngx.exit(500)
end

6. When a new request arrives, check the window size. If it’s below the limit, add the new request to the window and allow it. Otherwise, deny the request with a 429 status code.

if request_count < max_requests then
    local res, err = red:zadd(key, current_time, current_time)
    if not res then
        ngx.log(ngx.ERR, "Error adding new request: ", err)
        return ngx.exit(500)
    end

    local res, err = red:expire(key, window)
    if not res then
        ngx.log(ngx.ERR, "Error setting key expiry: ", err)
        return ngx.exit(500)
    end

    return 0  -- Indicate that the request is allowed
else
  return ngx.exit(429) -- Indicate that the request is blocked
end

You can find the working version of the Application with Readme on how to setup here.

My main motivation behind writing this blog.

This is the best, cost-effective solution we could come up with in a short span of time and there is always scope to improve.

How you have tackled the Dynamic Rate Limiting requirement in the past? Eager to know if there are good alternate solutions available.

EzyInfra.dev is a DevOps and Infrastructure consulting company helping clients in Setting up the Cloud Infrastructure (AWS, GCP), Cloud cost optimization, and manage Kubernetes-based infrastructure. If you have any requirements or want a free consultation for your Infrastructure or architecture, feel free to schedule a call here.