URL Shortener · System Design Deep Dive

Build a service like bit.ly: take a long URL, return a short alias, redirect on click. The classic system design interview problem.

We use real production components: nginx, Node.js, Redis, PostgreSQL, Kafka, and ClickHouse.

Where This Is Actually Used

URL shorteners are everywhere. Most major platforms run one in-house, both as a product and as internal plumbing for analytics, sharing, and tracking.

Real services running this exact design

Where short URLs actually show up

Architecture · Trace a Real Request

Three flows below: create a short URL, redirect on cache hit (~2 ms), redirect on cache miss (~11 ms). Hover any component for details. Use the fullscreen button to expand the diagram.

Requirements

Functional

• Short URL generation: generate a unique short alias for any long URL.
• Redirection: visiting the short URL redirects to the original.
• Custom short links: users can request their own alias (e.g. sho.rt/my-blog).
• Deletion: authorized users can delete a short URL.
• Update: authorized users can change the long URL behind an existing short link.
• Expiry: a default TTL applies, but users can set a custom expiration. Expired URLs are deleted after 5 years even if unused so the datastore index does not grow unbounded.
• Per-URL click analytics (count, geo, referrer).

Non-Functional

• Availability: 99.99% (4.3 min downtime / month). The domain is part of every generated URL, so downtime breaks every link.
• Scalability: horizontal scaling at every layer (LB, API, cache, DB).
• Readability: short links must be easy to type. No visually ambiguous characters.
• Latency: p99 redirect < 100ms.
• Unpredictability: short URLs must not be guessable. Sequential IDs leak link existence (just decrement to find your neighbor's link).
• Read-heavy: 1:100 write-to-read ratio. Cache must absorb most traffic.

Back-of-Envelope Estimation

Size first, design second. These numbers drive every later decision.

Assumptions

• 200 million new URLs per month.
• Write-to-read ratio 1:100 (heavily read-dominated).
• Each row uses 500 bytes on disk.
• Each URL retained for 5 years unless explicitly deleted.
• 100 million daily active users (DAU).

System APIs

Three REST endpoints expose the service. Every endpoint requires an api_dev_key for rate-limit accounting and abuse tracking.

POST /shorten · create a short URL

POST /shorten
{
  "api_dev_key":  "...",         // user identifier (required)
  "original_url": "https://...", // long URL to shorten (required)
  "custom_alias": "my-blog",     // optional user-chosen code
  "expiry_date":  "2031-05-14"   // optional, default 5 years
}

→ 201 Created
{ "short_url": "https://sho.rt/2JjVS" }

GET /:url_key · redirect

GET /:url_key?api_dev_key=...

→ 302 Found
Location: https://example.com/article/long-path

DELETE /:url_key · remove a short URL

DELETE /:url_key
{ "api_dev_key": "..." }   // must match owner

→ 200 OK
{ "message": "URL removed" }

nginx · Edge Gateway

nginx is the entry point. Every request hits it first. Three jobs:

1. TLS termination: decrypts HTTPS once at the edge.
2. Rate limiting: blocks abuse via token bucket.
3. Load balancing: spreads traffic across stateless API replicas using least_conn.

Three swim lanes below, one per client IP. Each IP has its own independent token bucket. Click Burst 9 on one row to flood that IP. Only that bucket drains. The other two stay full. That is the per-IP isolation nginx gives you.

Production config

http {
    # Token bucket: 5 req/s per IP, burst up to 10
    limit_req_zone $binary_remote_addr zone=urlapi:10m rate=5r/s;

    upstream api {
        least_conn;
        server api-1.svc:3000 max_fails=2 fail_timeout=10s;
        server api-2.svc:3000 max_fails=2 fail_timeout=10s;
        server api-3.svc:3000 max_fails=2 fail_timeout=10s;
        keepalive 64;
    }

    server {
        listen 443 ssl http2;
        ssl_certificate     /etc/ssl/cert.pem;
        ssl_certificate_key /etc/ssl/key.pem;

        location / {
            limit_req zone=urlapi burst=10 nodelay;
            limit_req_status 429;
            proxy_pass http://api;
        }
    }
}

Node.js API · Base58 Encoding

The API is stateless. Any replica can serve any request. The interesting bit is how it generates short codes. Three approaches:

id   = redis.INCR("url:counter")    // atomic, globally unique
code = base58_encode(id)             // collision-free

Base58 uses 58 URL-safe characters: 1-9, A-Z without I/O, a-z without l. A 64-bit ID fits in at most 11 Base58 characters. 58⁷ ≈ 2.2 trillion 7-char codes, far more than we need. Step through the encoding below:

ID Management · Custom URLs · Scope

Scope of the short URL generator

A 64-bit sequencer can produce numbers from 1 to 2⁶⁴ − 1. We need to constrain that range so every code is between 6 and 11 characters:

• Minimum length 6: start the sequencer at 1,000,000,000 (1 billion). Anything smaller would encode to fewer than 6 Base58 chars.
• Maximum length 11: a 64-bit value uses up to log₂(2⁶⁴) / log₂(58) ≈ 10.9 Base58 digits. Round up to 11. The longest possible short URL is 11 chars.

Custom short links

A user can ask for sho.rt/my-blog instead of an auto-generated alias. The flow:

1. Validate format: only Base58 chars, length 6 to 11.
2. Decode to base-10: compute the numeric ID the alias would correspond to.
3. Check availability: look up that ID in the database.
4. Reserve the ID: if free, insert the row and mark the ID as used so the auto-sequencer never re-issues it.

Used / Unused ID tracking

The sequencer generates IDs into an "unused" pool. Once an ID is allocated (either by the sequencer or by a custom alias claim), it moves to the "used" pool. This guarantees a one-to-one mapping between numeric IDs and short URLs and prevents collisions.

// On custom-alias request:
const id     = base58_decode("my-blog");          // e.g. 9,181,722,813
const exists = await pg.query(
  "SELECT 1 FROM urls WHERE id = $1 OR short_code = $2",
  [id, "my-blog"]
);
if (exists) return { error: "alias unavailable" };

await pg.query(
  "INSERT INTO urls (id, short_code, long_url, user_id) VALUES ($1, $2, $3, $4)",
  [id, "my-blog", longUrl, userId]
);
// id is now in the "used" pool. Sequencer skips it on next INCR.

Base58 → base-10 (decoding)

Decoding the alias back into a numeric ID is straightforward positional notation: each char's index × 58^position, summed.

// "2JjVS" → 14,776,337
//
//   2  → idx  1  × 58⁴ = 11,316,496
//   J  → idx 17  × 58³ =  3,316,904
//   j  → idx 42  × 58² =    141,288
//   V  → idx 28  × 58¹ =      1,624
//   S  → idx 25  × 58⁰ =         25
//   ─────────────────────────────────
//                    sum = 14,776,337  ✓

Deletion + expiry

Deletion is a soft remove first (so click analytics aren't orphaned), hard delete after a grace window. Expired short URLs are purged after 5 years even if never accessed: the datastore index would otherwise grow without bound, blowing up query latency.

-- Daily cleanup job
DELETE FROM urls
WHERE expires_at IS NOT NULL
  AND expires_at < now() - INTERVAL '5 years';

Redis · In-Memory Cache

Redis sits between the API and PostgreSQL. For our 1:100 write-to-read ratio, ~90% of redirects should hit Redis. That is the difference between a healthy system and a database on fire.

What Redis actually is

Redis is a fast in-memory data structure server, not just a KV cache. It supports strings, hashes, lists, sets, sorted sets, and pub/sub. For us it is mostly a string store: SET url:00000Q "https://...". Sub-millisecond reads because everything lives in RAM.

Sharding via hash slots

A single Redis node maxes out around 100K ops/sec. Not enough for us. Redis Cluster splits the keyspace into 16,384 hash slots. Each shard owns a contiguous range. The slot for any key is CRC16(key) mod 16384. Resharding moves slots without downtime.

Persistence: RDB vs AOF

Redis can survive restarts via two persistence modes (you can combine them):

Failover via Sentinel

Each primary has 1 to 2 replicas. Redis Sentinel monitors primaries. If a quorum agrees a primary is dead, Sentinel promotes a replica and updates clients. Failover happens in seconds.

Cache code (cache-aside pattern)

async function getLongUrl(code: string): Promise<string | null> {
  const cached = await redis.get(`url:${code}`);
  if (cached) return cached;                             // HIT: 90% of traffic

  const row = await pg.query(
    "SELECT long_url FROM urls WHERE short_code = $1",
    [code]
  );
  if (!row) return null;

  await redis.setex(`url:${code}`, 3600, row.long_url);   // populate for 1h
  return row.long_url;
}

PostgreSQL · Source of Truth

Redis is fast but volatile. PostgreSQL is durable. Every URL mapping lives here.

Schema

CREATE TABLE urls (
  id          BIGSERIAL    PRIMARY KEY,
  short_code  VARCHAR(8)   NOT NULL UNIQUE,
  long_url    TEXT         NOT NULL,
  user_id     BIGINT,
  created_at  TIMESTAMPTZ  NOT NULL DEFAULT now(),
  expires_at  TIMESTAMPTZ,
  click_count BIGINT       NOT NULL DEFAULT 0
);

CREATE INDEX idx_short_code ON urls USING btree (short_code);

B-tree index lookup

The UNIQUE constraint creates a B-tree index. Lookups take O(log n) page reads. For our 100M-row table, that is ~3 page reads, ~9ms on cold cache.

Read replicas via streaming replication

One primary cannot absorb 7,600 reads/sec at sustained load. PostgreSQL streams every WAL record to replicas in near-real-time (~10ms lag). The API routes SELECT queries to replicas and INSERT to the primary.

Kafka · Async Click Analytics

The redirect path must be fast. Doing UPDATE click_count++ on every redirect would create a hot row, lock contention, and tank latency. Solution: the API publishes a click event to Kafka and returns the 302 immediately. A worker pool consumes events and writes batched rollups to ClickHouse.

Partitioned log

The clicks topic has 3 partitions (production: dozens). Same key always lands in the same partition. This preserves per-URL ordering. Each partition is an immutable append-only log with its own consumer offset.

Why not Redis pub/sub or a DB queue?

• Redis pub/sub: at-most-once delivery. Worker restart loses events.
• DB-as-queue: fine at low scale. Caps out around 5K msg/s.
• Kafka: durable, replayable, partitioned. Millions of msg/sec.

ClickHouse · Analytics OLAP

PostgreSQL is a row store. ClickHouse is columnar. Row stores are great for SELECT * on one row, terrible at SUM(clicks) GROUP BY day over billions. Columnar reads only the columns it needs and compresses them 10 to 100 times.

The worker batches click events (e.g. 1000 per insert) from Kafka into ClickHouse. Dashboards query ClickHouse, never PostgreSQL. This is the standard split: OLTP for the hot path, OLAP for analytics.

Reliability · What Goes Wrong at Scale

At scale, failure is not the exception. It is a daily event. With 10,000 disks across the fleet and MTTF of 10 years per disk, expect ~3 disk failures every single day. Reliable systems do not try to eliminate failure. They survive it.

Hardware faults vs systematic faults

Three real failure modes

Reliability principles

• Replicate everything. PostgreSQL streaming, Kafka RF=3, Redis Sentinel.
• GSLB (Global Server Load Balancing). Route traffic to the nearest healthy data center. On regional failure, GSLB drains the bad region automatically. Eventual consistency between regions is fine here because a brand-new short URL is typically not accessed for a few seconds, leaving time for replication.
• Daily backups to durable object storage (e.g. S3). Worst-case recovery loses URLs created since the last snapshot. RPO measured in hours, not days.
• Idempotent writes. A worker may consume the same Kafka message twice. Inserts must tolerate that.
• Async over sync. Click tracking is async. A Kafka outage does not break redirects.
• Health checks. nginx removes unhealthy API replicas via max_fails.
• Rate limit per api_dev_key. Fixed-window counter is enough at this scale. Protects against DoS and accidental loops.
• Observability. Trace every request, alert on p99, dashboard error rates per endpoint.
• Blast-radius limits. One bad replica should not take down the cluster.

Requirements compliance summary

Interview Follow-ups

Q1: How do you prevent collisions in Base58 codes?

By construction. We use a counter (INCR), not a hash. Atomic INCR is globally unique. Hash/random needs retry-on-collision, which does not scale.

Q2: 301 or 302 for redirects?

302. A 301 is cacheable forever by browsers and CDNs. They never hit our service again, so we lose analytics. 302 is temporary, not cached, every click reaches us.

Q3: How do you handle 10× more traffic?

• nginx: add LB instances behind a TCP load balancer.
• API: already stateless. Just add replicas.
• Redis: add shards (move hash slots).
• PostgreSQL: add read replicas, then shard by hash(short_code).
• Kafka: increase partition count, add consumers.

Q4: What if Kafka is down?

Redirects still work.Kafka is off the hot path. The producer buffers events in memory. If Kafka stays down past the buffer limit, we drop events. Click counts go stale, the core service stays up. That is async's whole point.

Q5: A user generates 100K URLs in a minute. What happens?

nginx limit_req caps them at 5 req/s per IP. They get 429s. For distributed attackers, add per-user-account limits at the API level and CAPTCHA on signup.

Q6: Support custom aliases like sho.rt/my-blog?

Same table, just INSERT the user-supplied short_code. The UNIQUE constraint protects against collisions. Reserve a namespace prefix (e.g. generated codes start with a digit) so user aliases cannot collide with the counter sequence.

Q7: What is the trickiest failure mode?

Silent data corruption. If you write to Redis first and PostgreSQL fails, the user sees a working short URL that was never persisted. Always write to PostgreSQL first (source of truth), then populate cache. Never the other way around.