How Reliable detects broken WebSockets without collecting a single payload

For its first month in our codebase, Reliable's WebSocket anomaly engine flagged nothing at all. Not because every app it watched was healthy. Every session summary the SDK sent was being rejected by the database, and nobody noticed, because telemetry that fails quietly looks exactly like a system with nothing to report.

This post is about that engine. What it is for, how it turns traffic it is not allowed to read into numbers it can reason about, the math behind those numbers, and the calls we got wrong on the way. Most of it applies to any system that has to watch something without looking inside it.

#A pipe with no status codes

HTTP is easy to monitor because every request describes itself. It has a method, a URL, a status code and a duration. Error rates and latency percentiles per route fall straight out of the logs.

A WebSocket gets one HTTP request, the upgrade, which returns 101 Switching Protocols. After that it is an open, two-way stream of frames that can live for hours. There is no status code per message and no route. The protocol does not even have a notion of a request and its reply. Whatever structure exists lives inside the payloads, and is different for every app.

Reliable already tracked the loud failures. We call it L1: one row per connection with open time, close code, reconnect storms, and message and byte counts. It catches sockets that will not open, that die with close code 1006, or that reconnect in a loop.

The failures L1 cannot see are the quiet ones:

• Slow replies. A trading app sends order.create and normally gets order.ack back in about 40 ms. After a backend deploy, acks take 900 ms. The connection is healthy. There are zero errors and zero bad close codes. Users are furious.
• Missing or wrong replies. A client sends subscribe and a snapshot normally follows. Today nothing follows, or something else does. No exception is thrown anywhere.

We called detection for these two L2 (timing) and L3 (pattern).

#The constraint that decided everything

Before any design, one rule: message payloads never leave the browser.

WebSocket traffic is often the most sensitive data an app has: direct messages, orders, live documents. The Reliable SDK runs inside other teams' products, and we did not want to become a place where that data piles up. The numbers made the same point. A busy client can push 100 messages a second. At 1 KB each, that is about 370 MB per user per hour. Even if privacy allowed it, storing it would not be affordable.

Two more constraints came from where the code runs. The SDK hooks into the app's own message listener and send() call, so everything we do per message has to be synchronous and cheap, and it has to add close to nothing to the bundle we ship into other people's pages.

So the question was: how do you learn what normal traffic looks like, and notice when it stops being normal, without ever looking at what the traffic says?

#The idea: learn normal per message shape

Real protocols are low-cardinality. A chat app has maybe ten kinds of message and a trading desk maybe twenty. The text inside changes on every message. The shape barely changes at all.

If every message can be mapped to a stable shape ID, we can keep, for each shape: how many appeared in a session, how big they were, and how far apart they were. For each pair of "this was sent, that came back", we can keep how long the reply took. Aggregate those per project and you have a baseline. Compare each new session with the baseline and you have anomaly detection that never saw a payload.

%% caption: Figure 1. The whole pipeline. Only hashes, counts and sketches ever leave the browser.
flowchart TD
    subgraph Browser["In the browser (Reliable SDK)"]
        M["Message sent or received"] --> F["Fingerprint: 64-bit hash of its shape"]
        F --> S["Connection summary: top shapes, sketches, request/reply pairs"]
    end
    S -->|"once, when the socket closes (4 to 8 KB)"| I["Reliable backend: ingest"]
    I --> R["Fold into project baselines"]
    I --> Z["Score the session against the baselines"]
    Z -->|"joint score 3.0 or more"| A["Anomaly on the dashboard"]

The rest of this post walks through each box.

#Step one: turning a message into a fingerprint

extractFingerprint() runs on every message in both directions and picks a strategy by type:

• JSON (text that starts with { or [): the sorted list of top-level keys, plus a discriminator. That is the value of the first field named type, event, kind, action, op or method that holds a plain value, capped at 32 printable characters. For arrays, the shape of the first element.
• Binary: the length plus the first 16 bytes. Most binary protocols put a type header up front. The length is included so a 4-byte message cannot collide with a 100-byte one that starts the same way.
• Blob: a Blob cannot be read synchronously (Blob.text() returns a Promise), so it is bucketed by floor(log2(size)).
• Plain text: the first 256 characters, with UUIDs, long hex strings and runs of digits replaced by placeholders.

Here is real output from the SDK:

j:0d99781206c72c3c   {"type":"chat.send","to":42,"text":"hi"}
j:0d99781206c72c3c   {"type":"chat.send","to":7,"text":"see you at 5"}
j:7cfc7e39806a7a0a   {"type":"ping"}
j:49eaa83963bec488   {"type":"pong"}
t:07afecba1f9711ac   user_123 joined
t:07afecba1f9711ac   user_999 joined
t:96ec7cd6f86f51c1   password=hunter2

The two chat messages hash the same input, text,to,type|type=chat.send, so they share a fingerprint even though their contents differ. The discriminator is why ping and pong do not. Their keys are identical, and without the discriminator a server that stopped answering pings would look perfectly normal. And password=hunter2 becomes 16 hex characters that cannot be turned back into anything.

#Why FNV-1a, and why our first design said BLAKE3

The first draft of the architecture doc said the fingerprint would be "BLAKE3 or SHA-1". That was the wrong tool. Both are cryptographic hashes, and we have no security boundary here: we need similar shapes to land in the same bucket, that is all. They were also impractical. The browser's crypto.subtle.digest is asynchronous, which would force a Promise onto every single message. The smallest JavaScript BLAKE3 port we found added 5.4 KB minified.

We used FNV-1a, a non-cryptographic hash that fits on one screen:

hash = 0xcbf29ce484222325                    // 64-bit offset basis
for each character c:
    hash = hash XOR c
    hash = (hash * 0x100000001b3) mod 2^64   // 64-bit FNV prime

There is one wrinkle. JavaScript numbers are doubles, which hold integers exactly only up to 2^53, so a 64-bit multiply is not available directly. The SDK carries the hash as two 32-bit halves and multiplies them with Math.imul. The prime is 2^40 + 0x1b3, which splits neatly into a high half of 0x100 and a low half of 0x1b3. We checked the SDK's output against a reference implementation, and the example hashes above match.

#Why 64 bits and not 32

The chance that at least two of n different shapes share a b-bit hash is the birthday problem:

P(collision) ~= 1 - exp( -n(n-1) / 2^(b+1) )

Inside one connection, with at most 16 shapes, even 32 bits is fine: about 3 in 100 million. But baselines are kept per project, across every page and every version of the app, and those shapes pile up over time.

• 10,000 shapes with 32 bits: 1.2% chance of a collision somewhere.
• 65,536 shapes with 32 bits: 39%.
• 10,000 shapes with 64 bits: about 3 in a trillion.

A collision means two different shapes share one baseline, so one shape's normal can hide the other's anomaly. Eight more hex characters per fingerprint take that off the table.

#Step two: a bounded summary per connection

Each socket gets one SessionAggregator. It keeps up to 16 shapes per direction. Once they are full, anything new goes into a shared __other__ slot, so memory stays flat no matter what the app sends. For each shape it keeps a count, a sketch of message sizes, and a sketch of the gap between consecutive messages of that shape. The sketches are explained in the next section.

Then there are request and reply pairs. Every outgoing message is pushed onto a list that only keeps the last 5 seconds. When a message arrives, it is paired with the most recent outgoing message still on that list, and that entry is removed so it cannot claim every later reply. The pair becomes an edge, like order.create to order.ack, with its own count and a sketch of round-trip times. Edges are capped at 8 source shapes with 4 replies each.

%% caption: Figure 2. Pairing replies with requests. Each incoming message is matched to the most recent unmatched send from the last 5 seconds.
sequenceDiagram
    participant App as App in the browser
    participant Srv as App server
    App->>Srv: subscribe (t = 0 ms)
    App->>Srv: chat.send (t = 12 ms)
    Srv-->>App: ack (t = 58 ms)
    Note over App: ack pairs with chat.send, round-trip 46 ms
    Srv-->>App: snapshot (t = 140 ms)
    Note over App: snapshot pairs with subscribe, round-trip 140 ms

When the socket closes, the whole summary is posted once, typically 4 to 8 KB. A socket that never carried a message sends nothing.

#Step three: percentiles you can add together

For every shape and every edge we want a p50 and a p99 across thousands of sessions. The tempting shortcut, storing each session's p99 and averaging them, is simply wrong. A session with 1,000 messages and a p99 of 200 ms, averaged with a session of 10 messages and a p99 of 900 ms, gives 550 ms. That number describes nothing. Percentiles do not average. To combine them you need the distributions, and shipping every raw value is exactly what we cannot afford.

We use DDSketch (Masson, Rim and Lee, VLDB 2019). It is a histogram whose buckets grow geometrically, so every bucket is the same width in relative terms:

alpha = 0.05                                    // target relative error
gamma = (1 + alpha) / (1 - alpha) ~= 1.10526    // each bucket is ~10.5% wider than the last
index(x)    = ceil( ln(x) / ln(gamma) )          // bucket i covers (gamma^(i-1), gamma^i]
estimate(i) = 2 * gamma^i / (gamma + 1)          // within 5% of every value in bucket i

A worked example: a 212 ms round-trip gives ln(212) / ln(1.10526) = 53.52, so it lands in bucket 54. That bucket covers 201.2 ms to 222.4 ms, and its estimate is 211.3 ms, 0.3% off. The whole range from 1 ms to 60 seconds takes about 110 buckets, and 1 byte to 1 MiB takes about 139. The sketch only stores buckets that were actually used. Zero gets a separate counter, because ln(0) is minus infinity.

To read a percentile, walk the buckets from smallest to largest, adding counts until you pass that fraction of the total, and return the bucket's estimate. The error bound holds for any percentile.

The property we chose it for: two sketches merge by adding their counts bucket by bucket. The same index always means the same range, so the merged sketch is identical to the one you would have built from all the data together. Merging adds no error at all. t-digest is the other popular choice, but merging t-digests is itself approximate, and we merge on every single session.

On the wire a sketch is tiny. This is an example of one:

{ "a": 0.05, "n": 214, "z": 0, "mn": 31, "mx": 912,
  "b": [[35, 12], [36, 40], [37, 61], [38, 44], [54, 2]] }

#A baseline that forgets

Each project keeps one rollup sketch per shape and per edge. When a session arrives, it is folded in with an exponential moving average, applied to every bucket count and to the totals:

baseline[i] = 0.95 * baseline[i] + 0.05 * session[i]

%% caption: Figure 3. Folding one session into the project baseline. Buckets line up by index, so the merge is a weighted sum.
flowchart TD
    V["One session's round-trip times: 38, 41, 44, 212 ms"] --> B["Bucket each value: i = ceil(ln x / ln gamma)"]
    B --> S["Session sketch: count per bucket"]
    P["Project baseline sketch"] --> E["baseline = 0.95 x baseline + 0.05 x session"]
    S --> E
    E --> Q["Read p50 and p99, each within 5%"]

This has two useful effects.

• Sessions count by traffic, not one vote each. A session with 1,000 messages adds 50 to the bucket totals, and one with 10 messages adds 0.5. The baseline stays a percentile over messages, not an average over sessions. That is the mistake from the previous section, avoided.
• Old traffic fades. A session's weight halves every ln(0.5) / ln(0.95), about 13.5 sessions later. After 100 sessions it carries 0.6%. Buckets that drop below 0.05 are removed, so rollups stay small. When a deploy changes a message format, the baseline follows the app instead of staying stuck on last month's numbers.

#Step four: deciding whether a session is weird

Nothing is scored until a project has seen 1,000 sessions. Below a hundred or so samples, a p99 is really just the maximum, and it takes hundreds before the tail means anything. 1,000 is a first guess that we will tune with real traffic. A shape or edge also has to appear in 20 sessions before it can be scored, so one-off shapes, such as those from random URLs, never raise alarms.

Each shape and edge in a closed session gets one of two kinds of score:

count score  = |c - mu| / sqrt(mu + 1)
spread score = |session median - baseline p50| / max(baseline p99 - baseline p50, 1)

The count score treats message counts as roughly Poisson, where the variance equals the mean, so sqrt(mu) is the natural yardstick. The +1 stops rare shapes, with an expected count near zero, from dividing by almost nothing. The spread score is used for sizes, gaps and round-trip times. It measures distance from the median in units of "baseline p99 minus baseline p50", so 1.0 means about as far out as the baseline's p99.

Why not standard deviations? Latency is heavy-tailed. A handful of huge outliers inflates the standard deviation so much that real problems look normal. The p50-to-p99 spread barely moves when outliers show up. It also means these scores are not sigmas, which we learned the hard way (more on that below).

Then the scores are combined. A shape's score is the worst of its count, size and gap scores, because one bad dimension is enough. An edge's score is the worse of its count and round-trip scores. Then:

F     = root mean square of shape scores that are >= 1.0
A     = root mean square of edge scores that are >= 1.0
joint = sqrt(F^2 + A^2)                  // flag the session when joint >= 3.0

%% caption: Figure 4. Scoring one closed session. Scores under 1.0 sit inside the normal range and are left out.
flowchart TD
    S["Closed session"] --> C{"Project past 1,000 sessions?"}
    C -->|"no"| L["Learn only: update baselines"]
    C -->|"yes"| E["Score each established shape and edge"]
    E --> F["Keep scores of 1.0 or more"]
    F --> J["joint = sqrt(F^2 + A^2)"]
    J --> T{"joint 3.0 or more?"}
    T -->|"yes"| X["Flag the session"]
    T -->|"no"| N["Normal"]

#Scoring at ingest instead of in a background job

The original design had a worker that picked up recently closed sessions every 30 seconds and scored them. It is the standard "don't slow down ingest" pattern. When we sat down to wire it up, the session summary was already in memory at the moment we saved it, and the baselines were one query away. Scoring right there removed a queue, a worker process and about 30 seconds of delay before an anomaly appears.

The cost is real, and we wrote it down in the decision record: summaries are discarded after folding, so when the formula changes, old sessions cannot be re-scored. If we ever need that, we will keep raw summaries for a few days. We have not needed it yet.

#Knowing when to give up

All of this depends on messages having a shape. Encrypted payloads, or JSON whose keys are random IDs, produce a new fingerprint for nearly every message, and everything piles into __other__.

After every fold, the backend computes the __other__ share over the project's lifetime counts. Above 80%, L2 and L3 switch off for that project, permanently. L1 keeps running, and the dashboard shows a banner saying why. We would rather say "we cannot see this traffic" than show confident noise.

#What we got wrong

#A month of silence

The SDK recorded message direction as in and out. The database enum and the scorer used inbound and outbound. Every summary insert failed on the enum, and the request returned a 500. The SDK deliberately never retries or surfaces telemetry errors, because monitoring must never break the app it is monitoring, so nothing looked wrong from the outside. Even if the insert had worked, the scorer's baseline keys would never have matched, and every joint score would have been zero.

A July audit that read the pipeline end to end found it. The fix was to normalise direction at the single point where data enters the backend. The lesson: each half worked against its own idea of the format, and nothing checked the SDK's real payload against the real ingest. That check is cheap, and it would have caught this on day one.

#Measured, stored, never scored

The SDK measured round-trip time for every edge from the start, and the backend kept a baseline for it. The scorer only ever looked at edge counts. The per-shape gap signal looked like latency, but the gap between two messages of the same type is not the time a reply takes. So the headline failure, "replies got slow", was only caught by accident. Edges now get a proper round-trip score.

#Averages hide outliers

The joint score used to average over every shape and edge, including healthy ones scoring around 0.01. Take a session with ten edges where one got ten times slower and scores 4.1:

before:  sqrt( (4.1^2 + nine scores near 0) / 10 )  =  1.3    not flagged
after:   sqrt(  4.1^2 / 1 )                          =  4.1    flagged

The more healthy traffic a session had, the better it hid a real problem. The 1.0 floor fixed that. Ten edges that are each only mildly high (about 1.4) still score 1.4 and are not flagged, so lots of small wobbles do not add up to an alarm.

The trade-off we kept: one edge at 4.0 alongside nine at 1.5 scores about 1.9 and is not flagged. Taking the maximum instead would catch it, but it would also flag any session where a single shape wobbles. We chose the average, documented the case, and will revisit it with real traffic.

#Only counting the sockets that closed

Summaries were sent on the close event. A socket that was still open when the tab closed never fired it, so it was never reported. Those are the long-lived sockets, which are usually the interesting ones, so the baselines were built only from connections that ended politely. SDK 1.4.0 now also sends summaries for still-open sockets on pagehide, marked as unloaded.

#Calling it sigma

The dashboard used to label scores with the sigma symbol and talk about "3σ". As the formulas above show, they are not standard deviations. The label made the numbers sound more rigorous than they are. They are now just called scores, with a note explaining what 1.0 and 3.0 mean.

#How we made these calls

• Constraints first. "No payloads", "synchronous per message" and "no bundle weight" ruled out most options before anyone argued about them, including sampling payloads and crypto hashes.
• Every decision gets a record, with the rejected options. The hash choice and inline scoring each have an architecture decision record saying what we picked, what we turned down and what it costs. The records are append-only. When we change our minds, we write a new one, so the reasoning survives.
• Constants are labelled as guesses. 1,000 sessions, 20 appearances, the 0.05 decay and the 3.0 threshold are all marked as first guesses, to be tuned from real traffic, not from theory.
• Audits that read the whole path. The three worst bugs here were found by reading the pipeline from SDK to dashboard in one sitting, not by adding features. We now do that for every data pipeline in Reliable.

#What to take away

• You can learn a lot about traffic from its shape without ever touching its content.
• Pick the simplest tool that meets the real requirement. We needed grouping, not security, so a tiny hash beat a crypto library.
• If you need percentiles across many sources, ship a mergeable sketch, never the percentile itself.
• Test the real payload against the real ingest. Two halves that each work can still disagree.