Today we build one from scratch. A DNS resolver with TTL caching and the Chain of Responsibility pattern.

The 3-Question Framework

What does the system DO? It translates domain names into IP addresses by walking the DNS hierarchy. This is the Chain of Responsibility pattern. A request passes through a chain of handlers (cache, hosts file, recursive resolver) until one of them can answer it.

What operations must be FAST? Most DNS lookups should return instantly from cache. A dict with TTL gives us O(1) cache hits. Cache misses trigger network calls, which are slower but infrequent for popular domains.

What is the core LOGIC? Recursive resolution. Start at the root DNS servers. They point you to the TLD servers (.com, .io). The TLD servers point you to the authoritative servers. The authoritative server gives you the final IP address. Cache every answer with its TTL so you only walk the tree once.

Pattern:    Chain of Responsibility (cache -> resolver -> root -> TLD -> auth)
Structure:  dict with TTL (DNS cache)
Algorithm:  Recursive resolution with caching at every level

How DNS Resolution Actually Works

When you type crackthat.io in your browser, here is what happens:

This is a chain. Each handler either answers the question or passes it to the next handler in the chain. That is the Chain of Responsibility pattern.

The key insight: after the first lookup, Step 1 returns the answer instantly for the next 300 seconds. Without caching, every HTTP request would add 50 to 100ms walking the DNS tree.

Step 1: The DNS Cache

The foundation. A simple dict with TTL expiration. Same lazy expiration pattern from Issue 3.

import time

class DNSCache:

    def __init__(self):
        self._cache = {}

    def get(self, domain, record_type="A"):
        key = (domain, record_type)
        entry = self._cache.get(key)
        if entry is None:
            return None
        if time.time() > entry["expiry"]:
            del self._cache[key]
            return None
        return entry["value"]

    def put(self, domain, value, ttl, record_type="A"):
        key = (domain, record_type)
        self._cache[key] = {
            "value": value,
            "expiry": time.time() + ttl,
        }

    def size(self):
        return len(self._cache)

Interview phrasing: “The DNS cache is a hash map keyed by (domain, record_type) with lazy TTL expiration. Same pattern as Redis. O(1) lookups. Expired entries are cleaned up on access.”

Step 2: The Resolver

The resolver sends a query to a DNS server and returns the IP address. We use Python’s socket library to make the actual network call.

import socket

class DNSResolver:

    def __init__(self):
        self.cache = DNSCache()
        self._queries = 0

    def resolve(self, domain):
        cached = self.cache.get(domain)
        if cached:
            return cached

        ip = self._do_lookup(domain)
        if ip:
            self.cache.put(domain, ip, 300)
        return ip

    def _do_lookup(self, domain):
        self._queries += 1
        try:
            result = socket.getaddrinfo(domain, None, socket.AF_INET)
            if result:
                return result[0][4][0]
        except socket.gaierror:
            return None
        return None

    def stats(self):
        return {
            "cache_size": self.cache.size(),
            "queries_made": self._queries,
        }

On the first call, the resolver makes a real DNS query using the operating system’s resolver. The result is cached for 300 seconds. Every subsequent call for the same domain returns instantly from cache.

In production DNS servers like CoreDNS or Unbound, the resolver builds raw UDP packets and walks the tree from root servers to authoritative servers. Our implementation delegates to the OS for simplicity, but the caching and chain of responsibility architecture is identical.

How production recursive resolution works:

1. Ask root server for crackthat.io

2. Root says “Try the .io TLD servers”

3. Ask .io TLD server for crackthat.io

4. TLD says “The authoritative server is ns1.vercel-dns.com”

5. Ask authoritative server for crackthat.io

6. Authoritative says “76.76.21.21” with TTL=300

7. Cache the answer and return the IP

Step 3: The Chain of Responsibility

The Chain of Responsibility pattern makes the resolver extensible. Each handler tries to answer the query. If it cannot, it passes to the next handler.

from abc import ABC, abstractmethod

class DNSHandler(ABC):

    def __init__(self):
        self._next = None

    def set_next(self, handler):
        self._next = handler
        return handler

    def pass_to_next(self, domain):
        if self._next:
            return self._next.handle(domain)
        return None

    @abstractmethod
    def handle(self, domain):
        pass


class CacheHandler(DNSHandler):

    def __init__(self, cache):
        super().__init__()
        self._cache = cache

    def handle(self, domain):
        result = self._cache.get(domain)
        if result:
            return result
        return self.pass_to_next(domain)

class HostsFileHandler(DNSHandler):
    """Check local host mappings before going to network."""

    def __init__(self, hosts_path=None):
        super().__init__()
        import os
        if hosts_path is None:
            hosts_path = os.path.join(os.sep, "etc", "hosts")
        self._hosts = self._load_hosts(hosts_path)

    def _load_hosts(self, path):
        hosts = {}
        try:
            with open(path) as f:
                for line in f:
                    line = line.strip()
                    if line and not line.startswith("#"):
                        parts = line.split()
                        if len(parts) >= 2:
                            ip = parts[0]
                            for hostname in parts[1:]:
                                hosts[hostname] = ip
        except FileNotFoundError:
            pass
        return hosts

    def handle(self, domain):
        result = self._hosts.get(domain)
        if result:
            return result
        return self.pass_to_next(domain)


class RecursiveHandler(DNSHandler):

    def __init__(self, resolver):
        super().__init__()
        self._resolver = resolver

    def handle(self, domain):
        return self._resolver.resolve(domain)

Now we wire the chain together:

def build_resolver_chain():
    cache = DNSCache()
    resolver = DNSResolver()
    resolver.cache = cache

    cache_handler = CacheHandler(cache)
    hosts_handler = HostsFileHandler()
    recursive_handler = RecursiveHandler(resolver)

    cache_handler.set_next(hosts_handler).set_next(recursive_handler)
    return cache_handler


chain = build_resolver_chain()
ip = chain.handle("crackthat.io")
print(f"crackthat.io resolved to {ip}")

Why the chain matters:

Most queries hit the cache handler and return in microseconds. Only cache misses reach the hosts file handler. Only misses from both reach the recursive handler which makes network calls. This is the same layered lookup your operating system uses:

1. Application DNS cache (Chrome, Firefox)

2. OS resolver cache (the local hosts file, systemd-resolved)

3. Local recursive resolver (CoreDNS, dnsmasq)

4. Upstream recursive resolver (8.8.8.8, 1.1.1.1)

5. Root servers to TLD to Authoritative

Comparison: DNS Resolution Strategies

The DevOps Connection

Scaling to Millions: DNS at Production Scale

Our resolver handles one machine. Production DNS serves billions of queries per day across the globe. Here is how it scales.

Anycast Routing

Every root DNS server has one IP address, but that IP is announced from dozens of locations worldwide. When you query 198.41.0.4, your packets go to the nearest physical server. This is called anycast. Cloudflare uses the same technique for their 1.1.1.1 resolver. One IP, 300+ locations. Your query never crosses the ocean.

GeoDNS

A single domain can return different IP addresses based on where the client is. A user in Dubai gets routed to a server in the Middle East. A user in London gets routed to a server in Europe. Route53, Cloudflare, and Akamai all support this. It is how CDNs deliver content from the closest edge node.

Caching Hierarchy

Production DNS uses multiple layers of caching to absorb traffic before it reaches authoritative servers.

Browser cache (Chrome, Firefox)
    -> OS cache (systemd-resolved)
        -> Local resolver (CoreDNS in your K8s cluster)
            -> ISP resolver (your provider's DNS)
                -> Authoritative server (only on full miss)

Each layer absorbs 80 to 90% of queries. By the time a request reaches the authoritative server, it represents thousands of identical queries that were served from cache.

Negative Caching

Production resolvers also cache “domain not found” responses (NXDOMAIN). Without this, a typo like “gogle.com” would hit the authoritative server on every single request. Negative caching with a short TTL (30 to 60 seconds) prevents this.

DNS Failover and Health Checks

Route53 and Cloudflare support health checked DNS records. If a server fails its health check, the DNS record is automatically updated to point to a healthy server. The TTL on these records is kept low (30 to 60 seconds) so clients pick up the change quickly.

Normal:     api.example.com -> 10.0.0.1 (healthy)
Failover:   api.example.com -> 10.0.0.2 (backup)
Recovery:   api.example.com -> 10.0.0.1 (restored)

This is the foundation of active/passive failover. Your Terraform configs probably already use this pattern.

Rate Limiting DNS Servers

At scale, DNS servers must protect themselves from abuse. Techniques include response rate limiting (RRL) to cap the number of identical responses per second, and query rate limiting per source IP. Without these, a DNS amplification attack can take down your resolver.

Interview upgrade: When the interviewer asks about DNS, mention anycast for global distribution, caching hierarchy for traffic absorption, and health checked records for failover. This shows you think at production scale, not just single machine implementations.

Interview Walkthrough Script

When the interviewer asks “Design a DNS resolver”:

“I would use the Chain of Responsibility pattern. A DNS query passes through a chain of handlers: first the local cache for O(1) lookups, then the hosts file, then a recursive resolver. The recursive resolver walks the DNS tree starting at root servers, following referrals through TLD servers to the authoritative server. Every response is cached with its TTL. Most production traffic hits the cache and returns in microseconds. For reliability I would add retry logic, multiple root servers, and negative caching for NXDOMAIN responses.”

Challenge: Try It Yourself

1. CNAME resolution. Add support for CNAME records. When you get a CNAME instead of an A record, resolve the CNAME target recursively.

2. Negative caching. Cache NXDOMAIN (domain not found) responses too. This prevents repeated lookups for nonexistent domains.

3. Benchmark cache hit rates. Resolve 1000 random domains, then resolve them again. Measure the hit rate and time difference.

Next week: Build a Task Queue in Python (Priority Queue)

Previous: Build a Load Balancer | Build a Rate Limiter | The GIL Explained

upstream backend {
    server 10.0.0.1 weight=3;
    server 10.0.0.2 weight=1;
    server 10.0.0.3 weight=1;
}

But have you ever built what happens inside that upstream block? Today, we build three load balancing algorithms from scratch and learn the design pattern that lets you swap them at runtime without changing a single line of caller code.

The 3-Question Framework

What does the system DO? It distributes incoming requests across multiple backend servers. This is the Strategy pattern — the load balancer owns a pluggable algorithm that decides which server gets the next request.

You can swap algorithms (round-robin, weighted, and least-connections) at runtime without changing the routing code.

What operations must be FAST? Every single request passes through the load balancer. Selecting the next server must be O(1).

We cannot scan all servers or sort them on every request.

A list + index counter gives us O(1) round-robin.

A pre-expanded list gives us O(1) weighted selection.

A min-heap or linear scan handles the least connections.

What is the core LOGIC? Each algorithm answers the same question differently: “Given N servers, which one should handle this request?”

Pattern:    Strategy (swap algorithms at runtime)
Structure:  list + counter (round-robin), expanded list (weighted), heap (least-conn)
Algorithm:  Stateless rotation, weighted expansion, connection-aware selection

The Strategy Pattern: Why It Matters Here

Before we write any load-balancing code, let us build the pattern that holds it all together. The Strategy pattern is one of the most useful patterns in system design interviews — and one you already use daily.

from abc import ABC, abstractmethod

class LoadBalancer(ABC):
    """Base class — all algorithms implement this interface."""

    @abstractmethod
    def select(self) -> str:
        """Return the address of the next server to handle a request."""
        pass

    @abstractmethod
    def add_server(self, address: str, weight: int = 1) -> None:
        pass

    @abstractmethod
    def remove_server(self, address: str) -> None:
        pass

Every algorithm we build will implement this interface. The caller never knows which algorithm is running, and it just calls select() and gets a server address. This is exactly how nginx, HAProxy, and Kubernetes Services work internally.

Interview phrasing: “I would use the Strategy pattern so we can swap load-balancing algorithms without changing the routing code. This is the same pattern behind Terraform providers — same interface, different implementation.”

Algorithm 1: Round-Robin

The simplest approach. Rotate through servers in order. Every server gets equal traffic.

import threading

class RoundRobinBalancer(LoadBalancer):
    """Distribute requests evenly across all servers in rotation."""

    def __init__(self):
        self._servers = []
        self._index = 0
        self._lock = threading.Lock()

    def add_server(self, address: str, weight: int = 1) -> None:
        with self._lock:
            if address not in self._servers:
                self._servers.append(address)

    def remove_server(self, address: str) -> None:
        with self._lock:
            if address in self._servers:
                self._servers.remove(address)
                if self._index >= len(self._servers):
                    self._index = 0

    def select(self) -> str:
        with self._lock:
            if not self._servers:
                raise RuntimeError("No servers available")
            server = self._servers[self._index]
            self._index = (self._index + 1) % len(self._servers)
            return server

Complexity: O(1) select. O(n) add/remove (list operations), but these are rare.

Request 1 → Server A
Request 2 → Server B
Request 3 → Server C
Request 4 → Server A  (wraps around)
Request 5 → Server B
...

When to use: All servers have equal capacity. Simple, predictable, stateless.

When it fails: If server A is a 32-core machine and server B is a 2-core machine, round-robin sends them equal traffic. Server B collapses.

Algorithm 2: Weighted Round-Robin

Assign weights to servers. Higher-weight servers get proportionally more traffic.

class WeightedRoundRobinBalancer(LoadBalancer):
    """Servers with higher weight receive more requests."""

    def __init__(self):
        self._servers = {}  # address -> weight
        self._expanded = []  # pre-expanded list for O(1) select
        self._index = 0
        self._lock = threading.Lock()

    def add_server(self, address: str, weight: int = 1) -> None:
        with self._lock:
            self._servers[address] = weight
            self._rebuild()

    def remove_server(self, address: str) -> None:
        with self._lock:
            self._servers.pop(address, None)
            self._rebuild()

    def _rebuild(self):
        """Expand weights into a flat list for O(1) selection."""
        self._expanded = []
        for address, weight in self._servers.items():
            self._expanded.extend([address] * weight)
        self._index = 0

    def select(self) -> str:
        with self._lock:
            if not self._expanded:
                raise RuntimeError("No servers available")
            server = self._expanded[self._index]
            self._index = (self._index + 1) % len(self._expanded)
            return server

Complexity: O(1) select. O(total_weight) rebuild on add/remove.

The trick: Pre-expanding weights into a flat list turns weighted selection into a simple array index. No math per request.

When to use: Servers have different capacities. This is the weight=3 in your nginx upstream config.

Algorithm 3: Least-Connections

Route to the server with the fewest active connections. The most adaptive approach.

class LeastConnectionsBalancer(LoadBalancer):
    """Route to the server with the fewest active connections."""

    def __init__(self):
        self._connections = {}  # address -> active connection count
        self._lock = threading.Lock()

    def add_server(self, address: str, weight: int = 1) -> None:
        with self._lock:
            if address not in self._connections:
                self._connections[address] = 0

    def remove_server(self, address: str) -> None:
        with self._lock:
            self._connections.pop(address, None)

    def select(self) -> str:
        with self._lock:
            if not self._connections:
                raise RuntimeError("No servers available")
            # Find server with minimum connections
            server = min(self._connections, key=self._connections.get)
            self._connections[server] += 1
            return server

    def release(self, address: str) -> None:
        """Call when a request completes."""
        with self._lock:
            if address in self._connections:
                self._connections[address] = max(0, self._connections[address] - 1)

Complexity: O(n) select (scans all servers). For small server counts (typically <100), this is negligible. For massive pools, use a min-heap.

Server A: 12 active connections
Server B: 3 active connections   ← selected
Server C: 8 active connections

New request → B (fewest connections)
B now has 4 active connections

When to use: Requests have variable processing time (some take 10ms, some take 10s). Round-robin would overload a server stuck on slow requests. Least-connections adapts in real time.

The release() pattern: The caller must notify the load balancer when a request completes. This is how HAProxy’s leastconn and Kubernetes’ endpoint tracking work.

Putting It Together: Swap Algorithms at Runtime

# Start with round-robin
balancer: LoadBalancer = RoundRobinBalancer()
balancer.add_server("10.0.0.1")
balancer.add_server("10.0.0.2")
balancer.add_server("10.0.0.3")

# Handle requests
for request in incoming_requests:
    server = balancer.select()
    forward(request, server)

# Hot-swap to weighted — no caller changes needed
balancer = WeightedRoundRobinBalancer()
balancer.add_server("10.0.0.1", weight=3)
balancer.add_server("10.0.0.2", weight=1)
balancer.add_server("10.0.0.3", weight=1)

# Same loop works — the interface is identical
for request in incoming_requests:
    server = balancer.select()
    forward(request, server)

This is the power of the Strategy pattern. The routing code never changes. Only the algorithm inside the select() call changes.

Terraform uses this exact pattern — terraform apply calls the same interface whether the provider is AWS, GCP, or Azure.

Adding Health Checks

A production load balancer needs to detect failed servers. Here is a simple health check wrapper:

import time
import threading
import urllib.request

class HealthCheckedBalancer:
    """Wraps any LoadBalancer with periodic health checks."""

    def __init__(self, balancer: LoadBalancer, check_interval: float = 5.0):
        self._balancer = balancer
        self._all_servers = {}  # address -> health path
        self._check_interval = check_interval
        self._start_health_checks()

    def add_server(self, address: str, health_path: str = "/health"):
        self._all_servers[address] = health_path
        self._balancer.add_server(address)

    def select(self) -> str:
        return self._balancer.select()

    def _check_health(self):
        for address, path in list(self._all_servers.items()):
            try:
                url = f"http://{address}{path}"
                urllib.request.urlopen(url, timeout=2)
                self._balancer.add_server(address)  # re-add if healthy
            except Exception:
                self._balancer.remove_server(address)  # remove if unhealthy

    def _start_health_checks(self):
        def run():
            while True:
                time.sleep(self._check_interval)
                self._check_health()
        threading.Thread(target=run, daemon=True).start()

Interview phrasing: “I would wrap the load balancer with a health check decorator. A background thread pings each server’s /health endpoint every 5 seconds.

Unhealthy servers are removed from the pool. When they recover, they are automatically re-added. This is how ELB health checks and Kubernetes readiness probes work.”

Comparison: When to Use What

The DevOps Connection

Interview Walkthrough Script

When the interviewer asks, “Design a load balancer”:

“I would use the Strategy pattern with a pluggable algorithm behind a common interface. The default would be weighted round-robin — pre-expand weights into a flat array for O(1) selection.

For variable-latency workloads, I would swap to least-connections, which routes to the server with the fewest active requests. I would wrap the balancer with health checks — a background thread pinging /health endpoints, automatically removing failed servers and re-adding recovered ones.

For session affinity, I would add an IP hash strategy. The key design decision is that all strategies implement the same interface, so the routing layer never changes when you swap algorithms.”

Challenge: Try It Yourself

1. IP Hash. Implement a strategy that hashes the client IP to consistently route the same client to the same server. Hint: hash(ip) % len(servers).

2. Weighted least-connections. Combine weights with connection counts: score = connections / weight. The server with the lowest score wins.

3. FastAPI integration. Build an HTTP reverse proxy that accepts requests and forwards them to backends using your load balancer. Use httpx for async forwarding.

Next week: Build a DNS Resolver in Python

Previous: Build a Rate Limiter | The GIL Explained | TTL & Key Expiry

Today we build three rate limiting algorithms from scratch in Python — and learn why “100 requests per minute” is a harder problem than it sounds.

The 3-Question Framework

What does the system DO? It decides whether to allow or reject a request based on the number of requests a client has made recently. This is the Strategy pattern — we can swap algorithms (fixed window, sliding window, token bucket) at runtime without changing the caller.

What operations must be FAST? Every single request hits the rate limiter. It must decide to allow/deny in O(1). We cannot afford to scan all past requests.

A deque gives us O(1) append and O(1) popleft — perfect for maintaining a sliding window of timestamps.

What is the core LOGIC? The sliding window log: store each request’s timestamp in a deque.

On every new request, evict timestamps outside the window, then count what remains. If count >= limit, reject.

Pattern:    Strategy (swap algorithms at runtime)
Structure:  deque (sliding window of timestamps)
Algorithm:  Sliding window log — evict old, count remaining

Why “100 Requests Per Minute” Is Harder Than It Sounds

The naive approach, fixed window and it has a fatal flaw:

Imagine a limit of 100 requests per minute, with windows at :00 and :01.

The boundary between windows creates a burst vulnerability. This is why every production rate limiter uses sliding windows or token buckets instead.

Strategy 1: Fixed Window Counter

The simplest approach. Fast, memory-efficient, but vulnerable to boundary bursts.

import time
from collections import defaultdict

class FixedWindowLimiter:
    """Count requests per fixed time window."""

    def __init__(self, max_requests: int, window_seconds: int):
        self.max_requests = max_requests
        self.window_seconds = window_seconds
        self._counters = defaultdict(lambda: {"count": 0, "window_start": 0})

    def allow(self, client_id: str) -> bool:
        now = time.time()
        entry = self._counters[client_id]
        window_start = now - (now % self.window_seconds)

        if entry["window_start"] != window_start:
            entry["count"] = 0
            entry["window_start"] = window_start

        if entry["count"] >= self.max_requests:
            return False

        entry["count"] += 1
        return True

Complexity: O(1) time, O(n) space where n = number of clients.

Interview phrasing: “Fixed window is the simplest rate limiter — one counter per window per client.

It fails at window boundaries where a burst can exceed the limit by nearly 2x. Good enough for rough protection, not for strict enforcement.”

Strategy 2: Sliding Window Log

The precise approach. Stores every request timestamp. No boundary problem.

import time
from collections import defaultdict, deque

class SlidingWindowLogLimiter:
    """Track exact timestamps in a deque. Precise but memory-heavy."""

    def __init__(self, max_requests: int, window_seconds: int):
        self.max_requests = max_requests
        self.window_seconds = window_seconds
        self._requests = defaultdict(deque)

    def allow(self, client_id: str) -> bool:
        now = time.time()
        window_start = now - self.window_seconds
        req_log = self._requests[client_id]

        # Evict timestamps outside the window
        while req_log and req_log[0] <= window_start:
            req_log.popleft()

        if len(req_log) >= self.max_requests:
            return False

        req_log.append(now)
        return True

Complexity: O(1) amortized (each timestamp is appended once, popped once). Space: O(max_requests * clients).

Interview phrasing: “The sliding window log stores every request timestamp in a deque. On each request, evict entries older than the window, then check the count. It is perfectly precise — no boundary bursts. The trade-off is memory: if you allow 10,000 requests per minute, you store 10,000 timestamps per client.”

Strategy 3: Token Bucket

The flexible approach. Allows controlled bursts. Used by AWS, Stripe, and most API gateways.

import time

class TokenBucketLimiter:
    """Refill tokens at a fixed rate. Each request costs one token."""

    def __init__(self, max_tokens: int, refill_rate: float):
        self.max_tokens = max_tokens
        self.refill_rate = refill_rate  # tokens per second
        self._buckets = {}

    def allow(self, client_id: str) -> bool:
        now = time.time()

        if client_id not in self._buckets:
            self._buckets[client_id] = {
                "tokens": self.max_tokens,
                "last_refill": now,
            }

        bucket = self._buckets[client_id]

        # Refill tokens based on elapsed time
        elapsed = now - bucket["last_refill"]
        bucket["tokens"] = min(
            self.max_tokens,
            bucket["tokens"] + elapsed * self.refill_rate,
        )
        bucket["last_refill"] = now

        if bucket["tokens"] < 1:
            return False

        bucket["tokens"] -= 1
        return True

Complexity: O(1) time, O(1) space per client (just two numbers: token count + last refill time).

Interview phrasing: “Token bucket refills at a constant rate and allows bursts up to the bucket size. It is the most memory-efficient — just two numbers per client.

AWS and Stripe use it because it naturally shapes traffic: steady traffic gets steady throughput, while short bursts are absorbed by the bucket.

The trade-off: less precise than sliding window, but much more memory-efficient at scale.”

Strategy Pattern: Swap Algorithms at Runtime

The real power: wrap all three behind one interface so you can switch algorithms without changing any caller code.

from abc import ABC, abstractmethod

class RateLimiter(ABC):
    @abstractmethod
    def allow(self, client_id: str) -> bool:
        pass

class FixedWindowLimiter(RateLimiter):
    # ... (implementation above)
    pass

class SlidingWindowLogLimiter(RateLimiter):
    # ... (implementation above)
    pass

class TokenBucketLimiter(RateLimiter):
    # ... (implementation above)
    pass

# Swap at runtime — no caller changes needed
limiter: RateLimiter = SlidingWindowLogLimiter(100, 60)
# Later: limiter = TokenBucketLimiter(100, 100/60)

This is the Strategy pattern — the same pattern behind Terraform’s provider system (swap AWS for GCP), nginx’s load balancing (swap round-robin for least-connections), and Kubernetes’ scheduler (swap scoring plugins).

Comparison: When to Use What

The DevOps Connection

Scaling to Distributed Systems

Our single-process implementation works for one server. In production with multiple servers, you need shared state:

# Distributed sliding window using Redis sorted sets
import redis
import time

class DistributedSlidingWindow:
    def __init__(self, redis_client, max_requests, window_seconds):
        self.r = redis_client
        self.max = max_requests
        self.window = window_seconds

    def allow(self, client_id: str) -> bool:
        key = f"ratelimit:{client_id}"
        now = time.time()
        window_start = now - self.window

        pipe = self.r.pipeline()
        pipe.zremrangebyscore(key, 0, window_start)  # evict old
        pipe.zadd(key, {str(now): now})               # add current
        pipe.zcard(key)                                # count
        pipe.expire(key, self.window + 1)              # auto-cleanup
        results = pipe.execute()

        return results[2] <= self.max

Interview phrasing: “For distributed rate limiting, I would use Redis sorted sets. Each request adds a member with score = timestamp. ZREMRANGEBYSCORE removes old entries, ZCARD counts remaining.

The pipeline makes it atomic. This is the standard pattern used by Kong, Envoy, and most API gateways.”

Interview Walkthrough Script

When the interviewer asks “Design a rate limiter”:

“I would use the Strategy pattern to support multiple algorithms behind one interface. For the default, I would pick sliding window log — store timestamps in a deque, evict entries outside the window on each request, reject if count exceeds the limit. It is O(1) amortized and perfectly precise. If memory is a concern at scale, I would switch to token bucket — just two numbers per client, allows controlled bursts, which is what AWS and Stripe use. For distributed systems, Redis sorted sets give us the sliding window pattern with atomic pipelines.”

Challenge: Try It Yourself

1. Add HTTP headers. Return X-RateLimit-Limit, X-RateLimit-Remaining, and X-RateLimit-Reset headers — these are the standard rate limit response headers.

2. Sliding window counter. Implement the hybrid approach — weight the previous window’s count by the overlap percentage. More precise than fixed window, less memory than sliding log.

3. Multi-tier limits. Implement rate limiting with multiple windows: 10 per second AND 100 per minute AND 1,000 per hour.

Next week: Build a Load Balancer in Python

Previous: The GIL Explained | TTL & Key Expiry | Redis Persistence | Build Your Own Redis

Our MiniRedis cannot serve 100 clients at full speed. Not because of our code. Because of Python itself.

Today, we crack open the GIL, Python’s Global Interpreter Lock, and understand why Redis chose C, why your Python services need Gunicorn workers, and how to give the senior-level answer when an interviewer asks about concurrency.

The 3-Question Framework

What does the system DO? It serves thousands of concurrent clients doing GET/SET operations. This is the Reactor pattern, a single event loop dispatches work without spawning threads per connection.

What operations must be FAST? Everything. Every GET and SET must complete in microseconds. The bottleneck is not the data structure (a dict is O(1)), but how we handle multiple clients at the same time.

What is the core LOGIC? Two competing models:

• Multi-threaded: One thread per client. Simple to write, but Python’s GIL serializes CPU work anyway.

• Event loop: One thread, non-blocking I/O. No GIL contention. This is what Redis actually does.

Pattern:    Reactor (single event loop dispatches I/O)
Structure:  Event queue + file descriptor table
Trade-off:  Threads (simple but GIL-bound) vs Event loop (fast but complex)

What Is the GIL, Actually?

The Global Interpreter Lock is a mutex inside CPython (the standard Python interpreter). It allows only one thread to execute Python bytecode at a time, even on a 64-core machine.

Here is the mental model:

This is not a bug. It is a design decision. The GIL makes CPython’s memory management (reference counting) thread-safe without fine-grained locking.

The trade-off: simpler internals, but no true CPU parallelism from threads.

Why This Kills Our MiniRedis

Remember our active expiration from Issue 3? We used a background thread:

# From Issue 3 - our background cleanup thread
def _start_cleanup(self):
    def run():
        while True:
            time.sleep(self._cleanup_interval)
            self._cleanup_expired()    # <-- needs the GIL
    t = threading.Thread(target=run, daemon=True)
    t.start()

When this cleanup thread runs, every client request is blocked. Not because of our lock, because of the GIL. The cleanup thread holds the GIL while iterating the heap, and no other thread can execute Python code until it releases it.

With 100 concurrent clients and a cleanup cycle running, you get:

This is why Redis uses C. In C, there is no GIL. Threads can truly run in parallel on multiple cores. But Redis goes further;

It does not even use multiple threads for the main workload. It uses something better.

How Redis Actually Works: The Event Loop

Redis uses a single-threaded event loop based on epoll/kqueue. One thread. No locks. No context switching. No GIL.

Why is this fast?

1. No lock contention. One thread means no locks needed. Zero overhead.

2. No context switching. The OS does not swap threads in and out. Pure CPU time.

3. CPU cache friendly. One thread keeps data in the L1/L2 cache. Threads thrash the cache.

4. I/O is the bottleneck, not CPU. A GET/SET in memory takes ~100ns. Network I/O takes ~100us. The CPU is idle 99.9% of the time, waiting for network packets. One thread is enough.

The Python Concurrency Landscape

So if threads do not work for CPU-bound Python, what are the options?

The interview-level insight: Python threads are only useful when you are waiting for I/O (network calls, disk reads).

The GIL is released during I/O waits, so threads can overlap their waiting time. But for CPU work like iterating a heap or computing a hash, threads give you zero speedup.

Building a Better MiniRedis: asyncio Version

Here is what our MiniRedis would look like using asyncio instead of threads — the same pattern Redis uses, just in Python:

import asyncio
import time

class AsyncMiniRedis:
    def __init__(self):
        self._store = {}

    async def handle_client(self, reader, writer):
        while True:
            data = await reader.readline()
            if not data:
                break
            command = data.decode().strip().split()
            response = self._execute(command)
            writer.write(f"{response}\n".encode())
            await writer.drain()
        writer.close()

    def _execute(self, command):
        cmd = command[0].upper()
        if cmd == "GET":
            entry = self._store.get(command[1])
            if entry is None:
                return "(nil)"
            if entry["expiry"] and time.time() > entry["expiry"]:
                del self._store[command[1]]
                return "(nil)"
            return entry["value"]
        elif cmd == "SET":
            ttl = None
            if len(command) > 3 and command[3].upper() == "EX":
                ttl = time.time() + int(command[4])
            self._store[command[1]] = {"value": command[2], "expiry": ttl}
            return "OK"
        return "ERR unknown command"

    async def start(self, host="0.0.0.0", port=6380):
        server = await asyncio.start_server(
            self.handle_client, host, port
        )
        # Schedule background expiry
        asyncio.create_task(self._expire_cycle())
        async with server:
            await server.serve_forever()

    async def _expire_cycle(self):
        """Non-blocking expiry cycle - never blocks the event loop."""
        while True:
            await asyncio.sleep(0.1)
            now = time.time()
            expired = [
                k for k, v in list(self._store.items())
                if v["expiry"] and v["expiry"] <= now
            ]
            for k in expired:
                self._store.pop(k, None)

No threads. No locks. No GIL contention. The await points are where the event loop can switch to another client. The _expire_cycle cooperatively yields control so client requests are never blocked.

The DevOps Connection

Interview Walkthrough Script

When the interviewer asks, “Why doesn’t Redis use Python?” or “Explain the GIL”:

“The GIL is a mutex in CPython that allows only one thread to execute Python bytecode at a time.

This means Python threads cannot achieve true CPU parallelism; they are only useful for overlapping I/O waits. Redis needs microsecond latency for every operation, so it uses C with a single-threaded event loop based on epoll.

One thread, no locks, no context switching. The CPU work per command is tiny; it is I/O bound, so a single thread handling thousands of file descriptors via epoll is more efficient than thousands of threads fighting for the GIL.

In Python, the equivalent architecture is asyncio.”

Challenge: Try It Yourself

1. Benchmark threads vs asyncio. Spin up 100 concurrent clients hitting your MiniRedis. Compare latency with the threaded version (Issue 3) vs the asyncio version above.

2. Add pipelining. Real Redis clients send multiple commands in one TCP write. Modify handle_client to parse and execute batched commands.

3. Implement pub/sub. Add SUBSCRIBE and PUBLISH commands using asyncio — this is where event loops truly shine.

Next week: Build a Rate Limiter From Scratch in Python

Previous: TTL & Key Expiry | Redis Persistence | Build Your Own Redis

When a key expires, it remains in memory. Our lazy expiration only removes it when someone asks for it. What about the millions of keys nobody ever reads again? They are just leaking memory. Forever.

Today, we fix that by building two expiration strategies from scratch.

The 3-Question Framework

What does the system DO? It monitors all keys with a TTL and removes them when they expire. This is the Observer pattern.

What operations must be FAST? Find the next key to expire instantly. Min-heap (heapq) -- earliest expiry always on top.

What is the core LOGIC? Two complementary algorithms:

• Lazy expiration: Check on read. If expired, delete it. O(1) per read.

• Active expiration: Background process using a min-heap to proactively remove expired keys.

Share

Strategy 1: Lazy Expiration (The Baseline)

Check TTL on every GET. If expired, delete and return None. Zero write overhead, but dead keys stay in memory if never read.

Interview phrasing: Lazy expiration is ideal when most keys get read at least once after their TTL. It falls apart with write-heavy workloads where keys are never read again.

Strategy 2: Active Expiration with a Min-Heap

A min-heap keeps the next-to-expire key at position 0. Background thread polls the heap and removes expired keys.

• heappush: O(log n) -- add new expiry

• heappop: O(log n) -- remove the minimum

• heap[0]: O(1) -- peek at next expiry

Watch out for stale heap entries: when a key is overwritten with a new TTL, the old entry stays in the heap. Always verify the store current expiry matches before deleting.

Strategy 3: Redis-Style Probabilistic Sampling

Redis runs this algorithm 10 times per second:

1. Pick 20 random keys that have a TTL

2. Delete the ones that are expired

3. If more than 25 percent were expired, go back to step 1

4. Otherwise, stop until next cycle

This is adaptive -- few expired keys means minimal CPU, many expired keys means aggressive cleanup. Self-balancing.

Interview phrasing: Redis avoids maintaining a sorted structure for expiry because it would add O(log n) overhead to every write. Instead, it uses probabilistic sampling that is statistically guaranteed to converge.

Comparison: When to Use What

The DevOps Connection

Interview Walkthrough Script

I would use two complementary strategies: lazy and active expiration. Lazy checks on every read.

But lazy alone leaks memory for unread keys. So I would add active expiration, either a min-heap for deterministic cleanup, or Redis-style probabilistic sampling for scale. For millions of keys, go with the Redis approach.

Challenge: Try It Yourself

1. Implement EXPIRE and PERSIST commands

2. Implement heap compaction to remove stale entries

3. Benchmark all three strategies with 100K keys

Next week: Why Redis Uses C, Not Python: The GIL Explained

Previous: Redis Persistence: WAL and Snapshots | Build Your Own Redis: Key-Value Store From Scratch

Why Go for This?

In the Python version, we built a key-value store using a dict and FastAPI. It works, but it has a fundamental problem we explored in Issue 4: the GIL. Python threads cannot run in parallel on CPU work. Our background expiration thread blocks client requests.

Go has none of these problems:

• No GIL. Goroutines run on real OS threads across all CPU cores.

• Built-in concurrency. sync.RWMutex gives us concurrent reads with exclusive writes — exactly what a KV store needs.

• Static typing. The compiler catches type errors before runtime. No None surprises.

• Single binary. go build produces one binary. No virtualenv, no pip, no dependency hell.

This is why tools like Redis (if built today), etcd, CockroachDB, and Kubernetes are all written in Go.

Step 1: The Core Store

Python used a dict. Go uses a map — same O(1) hash map, but with explicit types.

package store

import (
	"sync"
	"time"
)

type entry struct {
	Value  string
	Expiry *time.Time // nil = no expiry
}

type KeyValueStore struct {
	mu   sync.RWMutex
	data map[string]entry
}

func New() *KeyValueStore {
	return &KeyValueStore{
		data: make(map[string]entry),
	}
}

What changed from Python:

The RWMutex is the big win. In Python, the GIL means only one thread runs at a time, regardless. In Go, RLock() allows unlimited concurrent readers, while Lock() (write) blocks everyone. This is exactly how production databases handle concurrent access.

Read more

That’s fine for a cache. It’s not fine for a database. So how do real databases like Redis, PostgreSQL, and Kafka survive crashes without losing data?

They all use the same two techniques: write-ahead logs and snapshots. Today, you’ll implement both from scratch.

The 3-Question Framework

What does the system DO? Our persistence layer records every mutation to disk so state can be rebuilt after a crash. This is the Observer pattern — the store publishes events (writes), and the persistence layer observes and records them.

What operations must be FAST? Writing to disk on every PUT must not kill performance. Sequential append to a file is the fastest disk operation — O(1) per write. We need an append-only list (a file).

What’s the core LOGIC? Two algorithms working together:

• Write-Ahead Log (WAL): Append every operation to a file before confirming it. On crash, replay the file.

• Snapshotting: Periodically dump the entire state to disk. On restart, load the snapshot first, then replay only the WAL entries after it.

Pattern:    Observer (store publishes writes → persistence layer records them)
Structure:  Append-only file (sequential writes — fastest disk I/O)
Algorithms: WAL replay + periodic snapshotting + log compaction

Why Append-Only? The Disk I/O Insight

This is the kind of insight that impresses in interviews:

Random writes to disk:    ~0.1 MB/s  (seek time dominates)
Sequential appends:       ~500 MB/s  (5000x faster!)

Writing the entire file on every PUT → random I/O → slow
Appending one line on every PUT → sequential I/O → fast

This is why databases don’t just overwrite a JSON file on every change. They append to a log. It’s the same reason Kafka is fast — it’s fundamentally an append-only log.

As a DevOps engineer, you’ve seen this tradeoff in redis.conf:

# Redis AOF fsync policies — same concept we're building
appendfsync always      # fsync every write (safest, slowest)
appendfsync everysec    # fsync once per second (default — good balance)  
appendfsync no          # let OS decide (fastest, riskiest)

Step 1: Write-Ahead Log (WAL)

The WAL is simple: before confirming any write to the client, append the operation to a file. If the process crashes, replay the file on startup.

import json
import os
import time

class WriteAheadLog:
    def __init__(self, path="appendonly.aof"):
        self.path = path
        self._file = open(path, "a")

    def append(self, operation: dict):
        """Append one operation to the log — O(1) sequential write"""
        self._file.write(json.dumps(operation) + "\n")
        self._file.flush()       # push to OS buffer
        # os.fsync(self._file.fileno())  # uncomment for full durability

    def replay(self):
        """Read all operations from disk to rebuild state"""
        if not os.path.exists(self.path):
            return []
        with open(self.path, "r") as f:
            operations = []
            for line in f:
                if line.strip():
                    operations.append(json.loads(line))
            return operations

    def truncate(self):
        """Clear the log (called after a snapshot)"""
        self._file.close()
        self._file = open(self.path, "w")  # overwrite with empty file

    def close(self):
        self._file.close()

What flush() vs fsync() Actually Means

This distinction comes up in interviews, and you already understand it from managing databases:

your code          →  OS buffer (RAM)  →  disk (permanent)
                   ↑                   ↑
              flush()              fsync()

flush(): "I'm done writing, push my data to the OS"
         Data is in the OS page cache — survives your process crashing
         Does NOT survive a power failure

fsync(): "Force the OS to write to physical disk RIGHT NOW"
         Survives power failure
         But costs ~1-10ms per call (huge at high throughput)

Redis appendfsync everysec calls fsync once per second — losing at most 1 second of data on power failure. That’s the sweet spot.

Step 2: Snapshots (RDB-Style)

Snapshots dump the entire in-memory state to a file periodically. On restart, loading a snapshot is instant — no need to replay thousands of log entries.

import json
import os
import time
import threading

class SnapshotManager:
    def __init__(self, path="dump.rdb.json"):
        self.path = path

    def save(self, store_data: dict):
        """Atomic snapshot: write to temp file, then rename"""
        snapshot = {
            "timestamp": time.time(),
            "num_keys": len(store_data),
            "data": store_data
        }

        # Write to temp file first
        tmp_path = self.path + ".tmp"
        with open(tmp_path, "w") as f:
            json.dump(snapshot, f)

        # Atomic rename — this is the trick
        # If we crash during json.dump, the old snapshot is still intact
        # os.replace is atomic on most filesystems
        os.replace(tmp_path, self.path)

    def load(self) -> dict:
        """Load snapshot from disk"""
        if not os.path.exists(self.path):
            return {}
        with open(self.path, "r") as f:
            snapshot = json.load(f)
        print(f"Snapshot loaded: {snapshot['num_keys']} keys "
              f"(saved at {time.ctime(snapshot['timestamp'])})")
        return snapshot["data"]

Why the Temp File + Rename?

This is an important systems concept. If you write directly to dump.rdb.json and crash halfway through, you’ve corrupted both the old data AND the new data.

By writing to a temp file and atomically renaming, either the old file or the new file exists — never a half-written one.

Redis does exactly this with its BGSAVE command. PostgreSQL does this with WAL checkpoints. You’ve probably seen .tmp files in production systems — now you know why they exist.

Step 3: Putting It All Together — MiniRedis

Now we integrate both persistence strategies with our key-value store:

import json
import os
import time
import threading

class MiniRedis:
    def __init__(self, wal_path="appendonly.aof", snapshot_path="dump.rdb.json",
                 snapshot_interval=300):
        self._store = {}
        self.wal = WriteAheadLog(wal_path)
        self.snapshots = SnapshotManager(snapshot_path)
        self.snapshot_interval = snapshot_interval
        self._write_count = 0

        # Recovery: snapshot first, then replay WAL
        self._recover()

        # Background snapshots
        self._start_auto_snapshot()

    def _recover(self):
        """
        Recovery order (same as Redis):
        1. Load snapshot  → gets bulk of data instantly
        2. Replay WAL     → applies writes since last snapshot
        """
        # Step 1: Load snapshot
        snapshot_data = self.snapshots.load()
        self._store = snapshot_data

        # Step 2: Replay WAL on top of snapshot
        operations = self.wal.replay()
        for op in operations:
            if op["op"] == "PUT":
                self._store[op["key"]] = {
                    "value": op["value"],
                    "expiry": op.get("expiry")
                }
            elif op["op"] == "DELETE":
                self._store.pop(op["key"], None)

        wal_count = len(operations)
        total = len(self._store)
        print(f"Recovery complete: {total} keys "
              f"({total - wal_count} from snapshot, {wal_count} from WAL)")

    # ── Core Operations ──

    def put(self, key, value, ttl_seconds=None):
        expiry = time.time() + ttl_seconds if ttl_seconds else None
        entry = {"value": value, "expiry": expiry}

        # WAL first, then memory (write-AHEAD log)
        self.wal.append({
            "op": "PUT", "key": key,
            "value": value, "expiry": expiry
        })
        self._store[key] = entry
        self._write_count += 1

    def get(self, key):
        entry = self._store.get(key)
        if entry is None:
            return None
        if entry["expiry"] and time.time() > entry["expiry"]:
            self.delete(key)
            return None
        return entry["value"]

    def delete(self, key):
        if key in self._store:
            self.wal.append({"op": "DELETE", "key": key})
            del self._store[key]
            return True
        return False

    def keys(self):
        now = time.time()
        return [
            k for k, v in self._store.items()
            if v["expiry"] is None or now <= v["expiry"]
        ]

    def dbsize(self):
        return len(self._store)

    # ── Snapshot Management ──

    def save(self):
        """Manual snapshot — like Redis BGSAVE"""
        self.snapshots.save(self._store.copy())
        self.wal.truncate()     # WAL is now redundant — snapshot has everything
        self._write_count = 0
        print(f"Snapshot saved: {len(self._store)} keys. WAL truncated.")

    def _start_auto_snapshot(self):
        """Background thread — like Redis save 300 10"""
        def auto_save():
            while True:
                time.sleep(self.snapshot_interval)
                if self._write_count > 0:
                    self.save()
        t = threading.Thread(target=auto_save, daemon=True)
        t.start()

Step 4: Testing the Crash-Recovery Cycle

Let’s prove it works:

# === Session 1: Write data ===
store = MiniRedis(snapshot_interval=10)

store.put("user:1", "sharon")
store.put("user:2", "alex")
store.put("city", "dubai", ttl_seconds=3600)
store.put("temp_key", "delete_me")
store.delete("temp_key")

print(f"Keys: {store.keys()}")
# Keys: ['user:1', 'user:2', 'city']

# Force a snapshot
store.save()
# Snapshot saved: 3 keys. WAL truncated.

# Write more data AFTER the snapshot
store.put("user:3", "jordan")
store.put("user:4", "casey")

# Now imagine the process crashes here...
# The WAL has user:3 and user:4
# The snapshot has user:1, user:2, city

# === Session 2: Restart after "crash" ===
store = MiniRedis(snapshot_interval=10)
# Output:
# Snapshot loaded: 3 keys (saved at Thu Feb 27 20:30:00 2026)
# Recovery complete: 5 keys (3 from snapshot, 2 from WAL)

print(store.get("user:1"))    # "sharon"  — from snapshot
print(store.get("user:4"))    # "casey"   — from WAL replay
print(store.get("temp_key"))  # None      — correctly deleted

Nothing was lost. The snapshot provided most of the data instantly, and the WAL filled in the gaps.

The Recovery Timeline (What Interviewers Want to Hear)

Timeline of writes:
─────────────────────────────────────────────────────►

 write  write  write  SNAPSHOT  write  write  CRASH
  1      2      3     (saves     4      5
                       1,2,3)         ↑ these are
                                       in the WAL

On restart:
1. Load snapshot    → restores keys 1, 2, 3       (instant)
2. Replay WAL       → restores keys 4, 5          (fast)
3. Ready to serve   → lost: 0 keys

The fsync Tradeoff Table

How This Maps to Your DevOps World

You’ve been configuring these settings in production for years. Now you can implement them and explain why they exist.

Interview Walkthrough Script

When the interviewer extends “Design a key-value store” with “How would you make it persistent?”:

“I’d combine two persistence strategies — write-ahead logging and periodic snapshots.

For every write, I’d append the operation to an append-only file before updating memory. This is sequential I/O — about 5000x faster than random writes. On crash, I replay the log to rebuild state.

The problem is the log grows forever and replay gets slow. So I’d run periodic snapshots — dump the entire state to a file using an atomic write (temp file + rename). After each snapshot, I truncate the log.

On recovery: load the snapshot first for speed, then replay only the WAL entries since that snapshot. You get fast recovery plus minimal data loss.

The durability knob is fsync frequency — fsync every write for zero data loss, or every second for a good balance. That’s a product decision, not an engineering one.”

Challenge: Try It Yourself

Extend MiniRedis with these features (solutions next week):

1. fsync policies: Add a fsync_policy parameter that supports "always", "everysec", and "no". Implement the "everysec" mode using a background thread.

2. WAL compaction without snapshot: Instead of truncating the WAL after a snapshot, rewrite it with only the current state. This is what Redis BGREWRITEAOF does.

3. Binary protocol: Right now the WAL uses JSON (human-readable but large). Implement a binary format using struct.pack for the WAL entries. Compare file sizes.

Next week in Crack That Weekly: Implement TTL & Key Expiry in Python — we’ll build both lazy and active expiration, exactly how Redis handles millions of expiring keys without scanning them all.

Previous issue: Build Your Own Redis: Key-Value Store From Scratch

About Crack That Weekly: A weekly newsletter that bridges the gap between real-world DevOps experience and coding interview prep. Every issue builds a real system from scratch in Python, connecting design patterns, data structures, and algorithms to tools you already use.

Written by Sharon Sahadevan, a DevOps engineer who got tired of interview prep that ignores operational experience.

But if an interviewer asks you to build a key-value store from scratch in Python, can you do it?

Most DevOps engineers freeze here — not because they don’t understand the concepts, but because they’ve never translated their operational knowledge into code. This issue fixes that.

By the end, you’ll have a working key-value store in Python that supports get, put, delete, and TTL-based expiry. More importantly, you’ll understand exactly which design pattern and data structure powers it — and how to explain your choices in an interview.

The 3-Question Framework: How to Approach Any System Design Problem

Before writing a single line of code, train yourself to ask three questions. This framework works for every system design problem you’ll face in an interview:

Question 1: What does the system DO? → This reveals the design pattern.

A key-value store sits between a client and data storage. It intercepts requests, stores data, and returns it on demand. That’s the Proxy pattern — the same pattern behind Nginx reverse proxying, CDN caching, and API gateways. You already use proxies daily; now you’re building one.

Question 2: What operations must be FAST? → This reveals the data structure.

A key-value store needs O(1) lookup, O(1) insert, and O(1) delete. There’s exactly one data structure that gives you all three: a hash map. In Python, that’s a dict.

Question 3: What’s the core LOGIC? → This reveals the algorithm.

For basic get/put, it’s straightforward hash map operations. For TTL (time-to-live), we need lazy expiration — check if a key has expired only when someone tries to access it. This is exactly how Redis handles expiry in practice.

System: Key-Value Store
Pattern: Proxy (sits between client and data)
Data Structure: dict (hash map) — O(1) for get, put, delete
Algorithm: Lazy expiration for TTL

Memorize this framework. It works for every system design question.

Step 1: The Simplest Key-Value Store (10 Lines)

Let’s start with the absolute minimum. This is what you’d write in the first 2 minutes of an interview:

class KeyValueStore:
    def __init__(self):
        self._store = {}

    def put(self, key, value):
        self._store[key] = value

    def get(self, key):
        return self._store.get(key)

    def delete(self, key):
        return self._store.pop(key, None) is not None

    def keys(self):
        return list(self._store.keys())

That’s it. A working key-value store. The _store prefix signals it’s a private implementation detail — a Pythonic convention interviewers notice.

But an interviewer will immediately ask: “What about expiry? What happens when keys should expire after a certain time?”

Step 2: Adding TTL (Time-To-Live) — Lazy Expiration

This is where it gets interesting. Redis supports SETEX and EXPIRE commands that make keys auto-delete after a timeout. Let’s implement the same thing.

The trick is lazy expiration: don’t actively scan for expired keys. Instead, check the expiry timestamp when someone accesses the key. If it’s expired, delete it and return None.

import time

class KeyValueStore:
    def __init__(self):
        self._store = {}    # key → {"value": ..., "expiry": timestamp | None}

    def put(self, key, value, ttl_seconds=None):
        expiry = time.time() + ttl_seconds if ttl_seconds else None
        self._store[key] = {"value": value, "expiry": expiry}

    def get(self, key):
        if key not in self._store:
            return None

        entry = self._store[key]

        # Lazy expiration: check on access
        if entry["expiry"] and time.time() > entry["expiry"]:
            del self._store[key]
            return None

        return entry["value"]

    def delete(self, key):
        return self._store.pop(key, None) is not None

    def keys(self):
        now = time.time()
        return [
            k for k, v in self._store.items()
            if v["expiry"] is None or now <= v["expiry"]
        ]

Why lazy expiration?

Redis actually uses both lazy and active expiration:

• Lazy: Check TTL on every GET. If expired, delete and return nil. This is what we implemented above.

• Active: A background job randomly samples 20 keys, deletes expired ones, and repeats if more than 25% were expired.

In an interview, implementing lazy expiration is enough. But mentioning that Redis combines both approaches shows depth — and that’s what separates a “pass” from a “strong hire.”

Step 3: Making It a Real Server With FastAPI

Your Python class is the engine. But how does a browser, CLI tool, or another service actually use it? Through an API layer. This is the piece most tutorials skip, and it’s the piece that clicks everything together:

from fastapi import FastAPI, HTTPException
from pydantic import BaseModel

app = FastAPI()
store = KeyValueStore()     # lives in memory

class PutRequest(BaseModel):
    value: str
    ttl_seconds: int | None = None

@app.put("/store/{key}")
def put_key(key: str, req: PutRequest):
    store.put(key, req.value, req.ttl_seconds)
    return {"status": "ok", "key": key}

@app.get("/store/{key}")
def get_key(key: str):
    value = store.get(key)
    if value is None:
        raise HTTPException(404, "Key not found or expired")
    return {"key": key, "value": value}

@app.delete("/store/{key}")
def delete_key(key: str):
    if not store.delete(key):
        raise HTTPException(404, "Key not found")
    return {"status": "deleted"}

@app.get("/store")
def list_keys():
    return {"keys": store.keys()}

Run it with uvicorn api:app --reload, and now you can test with curl:

curl -X PUT localhost:8000/store/name \
  -d '{"value":"sharon","ttl_seconds":60}' \
  -H "Content-Type: application/json"

curl localhost:8000/store/name
# {"key":"name","value":"sharon"}

In system design interviews, being able to describe this layered architecture — class → API → client — demonstrates that you think in terms of real systems, not just algorithms.

How to Explain This in an Interview (The 5-Minute Script)

Here’s exactly how to walk through this when an interviewer says “Design a key-value store”:

“I’d start with an in-memory hash map for O(1) reads and writes — in Python, that’s a dict.

For TTL support, I’d store each value alongside an expiry timestamp. On every GET, I’d check the timestamp and lazily evict expired keys. This avoids background threads for the basic case.

For the API layer, I’d use FastAPI with PUT, GET, and DELETE endpoints mapping directly to the store’s methods. Pydantic handles request validation.

The tradeoff right now is durability — everything is in memory. If the process restarts, all data is lost. To fix that, I’d add a write-ahead log: append every PUT and DELETE to a file. On restart, replay the log to rebuild state. This is exactly what Redis does with AOF persistence.

For production, I’d also consider: snapshots for faster recovery, background compaction to keep the log from growing unbounded, and replication for high availability.”

This answer covers data structure choice, algorithm design, API architecture, persistence strategy, and production concerns — all in under 2 minutes. That’s a strong hire signal.

Your DevOps Advantage: What You Already Know

If you’re a DevOps engineer reading this, you’ve been working with these concepts every day without realizing they’re “interview material”:

You’re not learning new concepts. You’re learning to name them and code them in Python.

Challenge: Try It Yourself

Extend the key-value store with these features (solutions in next week’s issue):

1. Active expiration: Add a background thread that periodically scans and removes expired keys. Hint: use threading.Thread with daemon=True.

2. Max capacity: Add a max_keys parameter. When the store is full, reject new keys with an error. (Bonus: instead of rejecting, evict the oldest key — that’s an LRU cache, which we’ll build in Issue 3!)

3. Key patterns: Implement a search(pattern) method that returns all keys matching a glob pattern like session:*. Hint: fnmatch.fnmatch().

Next week in Crack That Weekly: Redis Persistence in Python — Write-Ahead Logs & Snapshots. We’ll make our key-value store survive restarts using the exact same techniques Redis uses under the hood.

If this helped you see the connection between your DevOps work and coding interviews, share it with a fellow engineer who’s preparing for interviews.

About Crack That Weekly: A weekly newsletter that bridges the gap between real-world DevOps experience and coding interview prep. Every issue builds a real system from scratch in Python, connecting design patterns, data structures, and algorithms to tools you already use.

Written by Sharon Sahadevan, a DevOps engineer who got tired of interview prep that ignores operational experience.