Cache Miss

I have made countless mistakes in my career. This one stayed with me.

It was a large project. Steady-state traffic exceeded 100 requests per second. At peak, 300 to 500. The architecture was essentially a jQuery-based SPA calling REST APIs on the backend.

That day we were upgrading jQuery. A frontend-only change. No load testing. If anything goes wrong, we roll back. Easy.

The entire system went down immediately after deploy.

The new jQuery version appended an extra GET parameter to every REST query.

In a system at that scale, read paths depend on carefully designed caching layers. On the web, CDN cache keys are URLs. One additional parameter means a different URL. Every cache entry becomes a miss.

The system was designed around cache hits. Every request now went straight through to the database. The DB that had assumed a 100% cache hit rate. It was hit with full production load. It gave up the ghost.

A frontend change killed the backend. Two places that seemed unrelated were connected by a transparent thread called cache. Invisible dependencies are dangerous precisely because they are invisible.