Design the reconnect first: sequence numbers, replay and snapshots

Real-time systems rarely fail while the connection is healthy. They fail in the few seconds after it comes back: messages that were in flight are gone, some arrive twice, and the client quietly continues from a state the server never had. The fix is not a better socket library. It is a resume protocol designed before the first feature: every message carries a sequence number, the client reconnects with the last one it applied, and the server either replays the gap or sends a snapshot.

#Why do real-time bugs cluster around reconnects?

TCP guarantees order and delivery inside one connection, and "delivery" means the peer's kernel buffer, not your application. Nothing more. The moment a connection drops, every guarantee resets, and real connections drop all the time: mobile handovers, load balancer idle timeouts, a laptop going to sleep, your own deploy draining a node.

At the moment of the drop, three things are usually true at once:

• The server has written messages to the socket that the client never read.
• The client has state that reflects some prefix of the stream, and nobody recorded which prefix.
• The server's idea of "sent" says nothing about what the client applied.

Without a protocol for that moment, the reconnect handler does what most do: subscribe again and carry on. The gap is silent. The UI looks fine until a value is wrong, and by then nobody can reproduce it.

#What should every message carry?

Four fields do most of the work:

type Envelope<T> = {
  stream: string; // the unit of ordering, e.g. "vehicle:42" or "order:9a1f"
  epoch: string;  // changes whenever the sequence space resets
  seq: number;    // monotonic within (stream, epoch)
  data: T;
};

Sequence per stream, not global. A single global counter turns every write in the system into one coordination point. Correctness rarely needs a total order; it needs order within the thing being changed. Pick the stream boundary to match the entity whose state the client rebuilds.

Assign the sequence where the order is decided. That is the database transaction or the log append that commits the change, not the WebSocket gateway. Two gateway nodes will happily disagree about order; the commit cannot. In PostgreSQL, a counter row per stream, updated in the same transaction as the state change, gives you a gap-free sequence, and the row lock serializes writers to the same stream until commit, so sequence order equals commit order:

-- inside the transaction that applies the change
UPDATE streams SET head = head + 1 WHERE id = $1 RETURNING head; -- lock held until commit
INSERT INTO stream_events (stream_id, seq, type, data)
VALUES ($1, $2, $3, $4); -- $2 is the head returned above

The price: one very hot stream becomes a single-writer queue, so keep those transactions short. Publish only after the commit; a message sent and then rolled back would have its sequence number reused by different content.

A plain SEQUENCE is the wrong tool here. nextval is not transactional, so rolled-back transactions leave gaps, and a client cannot tell a harmless gap from a lost message. Worse, values are handed out in call order, not commit order: transaction B can commit 6 while A, holding 5, is still open, and a reader that has seen 6 resumes past 5 for good.

An epoch for the day the counter resets. Restores, migrations and in-memory brokers restarting all reset sequence spaces. If the client compares sequence numbers across two epochs, it will drop valid messages as "duplicates". An epoch that changes on reset lets the client detect it and resynchronize instead.

#What does the client do with each message?

The client keeps, per stream, the epoch and the last sequence it applied, and applies messages strictly in order:

type Applied<S> = { epoch: string; seq: number; state: S };

function receive<S, T>(
  current: Applied<S>,
  msg: Envelope<T>,
  apply: (state: S, data: T) => S,
  resync: () => void,
): Applied<S> {
  if (msg.epoch !== current.epoch) {
    resync(); // sequence space was reset: start from a snapshot
    return current;
  }
  if (msg.seq <= current.seq) return current; // duplicate from replay: ignore
  if (msg.seq > current.seq + 1) {
    resync(); // gap: never apply out of order
    return current;
  }
  return { epoch: current.epoch, seq: msg.seq, state: apply(current.state, msg.data) };
}

Three branches, and each one is a decision. Duplicates are dropped silently because replay produces them by design. Gaps are never patched over, because applying message 4815 on top of 4812 produces a state that never existed on the server. resync must be idempotent (one request in flight per stream), because every message that arrives before the snapshot lands in the same branch.

The snapshot is the one message that replaces state instead of applying to it, and it carries its own epoch and sequence:

function receiveSnapshot<S>(
  current: Applied<S>,
  snap: { epoch: string; seq: number; data: S },
): Applied<S> {
  if (snap.epoch === current.epoch && snap.seq < current.seq) return current; // stale response
  return { epoch: snap.epoch, seq: snap.seq, state: snap.data };
}

#What does the client send on reconnect?

One pair per stream: the epoch and the last sequence it applied. The server answers in a fixed order, and the order is what prevents the classic race.

async function resume(conn: Connection, stream: string, epoch: string, lastSeq: number) {
  const live = conn.subscribe(stream, { buffer: true }); // 1. capture live messages first
  const head = await log.head(stream);                    // { epoch, seq }
  const oldest = await log.oldestSeq(stream);
  const replayable = epoch === head.epoch && lastSeq <= head.seq && lastSeq + 1 >= oldest;

  if (replayable) {
    for (const m of await log.range(stream, lastSeq + 1, head.seq)) conn.send(m); // 2a. replay the gap
  } else {
    const snap = await snapshots.read(stream);            // 2b. state + the epoch and seq it reflects
    conn.send({ kind: "snapshot", stream, epoch: snap.epoch, seq: snap.seq, data: snap.state });
  }
  live.flush(); // 3. drain; the client drops anything at or below what it already applied
}

The tempting order is the opposite: read the snapshot, then subscribe. Every message committed between those two steps is lost, and if the stream then goes quiet nothing reveals it, because the client checks continuity only when the next message arrives. The window is small enough to pass every test and large enough to show up in production under load. Subscribing first means the worst case is a duplicate, and duplicates are already handled.

The snapshot must be consistent with its sequence number. Read the state and the stream's head in one REPEATABLE READ transaction (or a single statement); under the default READ COMMITTED, two statements in the same transaction can see different commits. Then the snapshot says exactly "this is the state after message N".

#Should every stream replay its history?

No. Streams differ in what a missed message means, and the protocol should say so explicitly:

Telemetry and control on the same connection are the typical mix: replaying ten seconds of stale positions is wasted work, while a single skipped control command is a correctness bug.

#How big should the replay buffer be?

Bounded, and sized on purpose. Keep it by count and by age, whichever ends first, and do the arithmetic before production does it for you. With illustrative figures, 10,000 active streams × 256 buffered messages × 300 bytes is about 770 MB of raw payload, before the per-object overhead of your runtime. That number decides whether the buffer lives in process memory, in Redis, or simply in the stream_events table with a retention job.

Size for the worst reconnect, not the average one: a deploy or a load balancer restart brings every client back at once. Use exponential backoff with jitter on the client and cap concurrent snapshot reads on the server, or the recovery path becomes the outage.

If you already run a log with offsets, use it instead of building one. Kafka offsets, Redis Streams IDs and NATS JetStream sequences all give you an ordered position to resume from, but not a contiguous one per logical stream: Kafka offsets belong to a partition and JetStream sequences to a JetStream stream, so both interleave many of your streams; transaction markers, compaction and deletes leave holes; Redis Streams IDs are ms-seq strings, not counters. Replace the client's seq + 1 test with a prev field, the position of the previous message on the same stream, and check msg.prev === current.seq.

#How do you make retries from the client safe too?

The same reasoning applies in the other direction. Commands the client sends will be retried after a reconnect, because the client cannot know whether the server processed the last attempt. Give every command a client-generated idempotency key. Record it in the same transaction as the effect, under a unique constraint so two concurrent retries cannot both win, and keep it longer than the client's longest retry window. A repeated key returns the original result; a repeated key with a different payload is an error. At-least-once delivery plus idempotent handling is the honest version of "exactly once".

Prefer messages that set state over messages that nudge it. Applying setSpeed(12) twice is harmless; applying increaseSpeed(1) twice is not.

#How do you test it?

Reconnect is a feature, so it gets tests like one:

• Put a fault-injecting proxy (Toxiproxy or your own) between client and server, and cut connections at random byte offsets, including mid-frame.
• Duplicate and delay frames across the reconnect boundary.
• Restart the sequence source to exercise the epoch path.
• Assert one property after every scenario: once traffic settles, the client's state equals the server's state for every stream, and no command produced its side effect twice.

Property-based tests fit well: generate random interleavings of writes and disconnects and check convergence. They find races that hand-written cases miss.

#When is this overkill?

If every stream is "latest value wins" and the client can always fetch the current state cheaply, a snapshot on reconnect is enough and sequence numbers add little. The protocol pays off as soon as one stream carries messages whose loss or duplication changes state, and many real-time products reach that point sooner than planned.

#In short

• One sequence number per stream, assigned at commit, plus an epoch for resets.
• The client applies in order, drops duplicates, and resynchronizes on gaps.
• On reconnect: subscribe first, then replay or snapshot, then drain.
• Declare per stream whether history matters.
• Size the replay buffer with arithmetic, back off with jitter, or reuse a log you already run (with a prev check).
• Idempotency keys, recorded with the effect, for everything the client sends.
• Kill connections on purpose in CI and assert convergence.