Stay Connected

When we introduced WebSocket to the chat service, the notion of scale changed.

HTTP is stateless. A request arrives, a response goes back, the connection drops. The server forgets the client exists. Line up servers behind a load balancer and any one of them can answer. That is the premise of scaling out.

WebSocket is the opposite. The connection stays open. A persistent pipe between server and client. The server remembers who is on the other end. Stateful. Add more servers behind the load balancer and it does not matter — if User A is connected to Server 1, messages for User A must travel through Server 1. Any server will do no longer applies.

Before WebSocket, we faked it with Ajax polling loops. Hit the server every few seconds. It looked real-time, but every request carried the full HTTP overhead. As users grew, the load became absurd. When WebSocket arrived, we thought we finally had proper bidirectional communication.

The infrastructure was not ready. Load balancers of the era did not anticipate long-lived WebSocket connections. Idle timeouts killed them silently. An LB restart wiped out every connection at once. Connection counts climbed and ports ran dry. CDNs were built around HTTP caching — always-on connections clashed with the model entirely.

Today Cloudflare and AWS ALB both support WebSocket natively. Cloudflare Durable Objects let you run stateful logic at the edge. But limits on concurrent connections and cost remain. Easier, yes. Solved, no. Server-Sent Events entered the picture too, eliminating the need for WebSocket in certain use cases.

Still, scaling stateful connections is fundamentally hard. To deliver a message to the server holding the connection, you need a relay — PubSub, Redis, something in between. When a server goes down, the client must find a new home. Every time I touch WebSocket, I am reminded of the decisiveness of HTTP's "designed to disconnect" premise.