The click day that can't oversell
A time-boxed "click day": thousands of users competing for fixed-capacity slots, traffic spiking on the same seats in the same second, one hard invariant — never exceed capacity.
The challenge
In a competitive enrollment event, the losing failure modes are both fatal: oversell a seat and you break a promise to a real person; go down under load and the event itself fails publicly. The system had to absorb the spike, stay fair, and be provably correct.
Architecture
The hot path never waits on SQL; SQL never trusts Redis blindly.
What made it work
- Redis-first reservation: a Lua token bucket as the primary path — atomic, fast, contention-friendly.
- Honest degradation: a circuit breaker trips to a transactional SQL row-lock path when Redis misbehaves, and an expired-block sweeper cleans up abandoned reservations.
- Self-healing state: a leader-elected 60-second reconciliation realigns Redis counters against the database, acting only on drift confirmed twice in a row.
- Contract-grade verification: a client-signed test plan, a k6 suite with unit-tested helpers, monitoring dashboards and an operational runbook.
- Failure rehearsed, not feared: a Redis failover deliberately induced mid-test caused no measurable degradation.
- Bottleneck found scientifically: a multi-VM load fleet isolated the breaking point at 6–7k concurrent users; a single-variable A/B proved the database tier — not the app — was what to scale.
Outcome
The event ran on a fully Terraformed Cloud Run + Cloud SQL + Memorystore stack with one-flag pre-event scaling. In the target load run — 2,000 concurrent users for a full hour, 700k+ complete journeys — contested lines filled exactly to capacity, verified by direct database query. Zero oversold, zero errors.