Averages hide outliers, maxima cry wolf: how Reliable combines anomaly scores

In September we found that Reliable's WebSocket anomaly engine could not see the one failure it was built for. A request and reply pair that had become ten times slower scored 4.1 on its own, comfortably over our flag threshold of 3.0. The session it belonged to scored 1.3, and nobody was told.

The individual score was fine. The bug was in the step after it: turning many small scores into one number. That step fits on one line of code and is surprisingly easy to get wrong in both directions. This post goes through the options with the math, the rule we chose, and the trade-off we accepted.

#Many scores, one decision

When a WebSocket connection closes, Reliable compares it with the project's baseline. (How we get baselines without ever reading a message is in the WebSocket deep dive.) Every message shape and every request/reply pair in the session gets a score:

spread score = |session median - baseline p50| / max(baseline p99 - baseline p50, 1)
count score  = |c - mu| / sqrt(mu + 1)

Both are unitless. On the spread score, 1.0 means "as far from the median as the baseline's p99", and a healthy entry sits close to 0. A session can have up to 16 shapes in each direction and up to 32 request/reply pairs, so one session can produce dozens of scores.

The dashboard needs one yes or no per session. Whatever function combines the scores quietly decides what "an anomalous session" means.

#Attempt one: the root mean square of everything

The first version combined scores with a root mean square over every entry:

joint = sqrt( (z1^2 + z2^2 + ... + zn^2) / n )

The reasoning was written in the code comment: dividing by n keeps the magnitude comparable across sessions, so a session with 16 message shapes does not score four times higher than one with 4 just because it has more of them.

Now put one bad score z among n - 1 healthy ones sitting near zero. The joint score becomes roughly z / sqrt(n), so to reach 3.0 the bad entry needs z >= 3 * sqrt(n):

entries in session    score one bad entry needs to flag
        1                        3.0
        4                        6.0
       10                        9.5
       25                       15.0

The more healthy traffic a session had, the better it hid a real problem. The apps with the richest protocols, the ones with the most to monitor, got the least protection.

We found it while adding a round-trip score for request/reply pairs. Before changing anything we wrote a throwaway script with synthetic baselines and a handful of scenarios. The scenario the feature exists for, "one pair ten times slower, nine healthy", came out at 1.3.

#The obvious fix, and why we did not take it

Use the maximum. joint = max(z1 ... zn) flags a single bad entry the moment it crosses 3.0, no matter how many healthy entries surround it.

The maximum has the opposite problem: multiple comparisons. Every extra score is another chance for ordinary noise to cross the line. Suppose each healthy score has a small, independent probability p of landing above 3.0. Then the chance that at least one of n does is:

P(false alarm) = 1 - (1 - p)^n

With an illustrative p of 1%:

scores per session    healthy sessions flagged by the max rule
         5                         4.9%
        10                         9.6%
        20                        18.2%
        40                        33.1%

Under the maximum, how often a healthy app gets flagged depends on how many message types it has, which says nothing about its health. A tool that flags a third of a busy app's healthy sessions gets muted within a week, and then it catches nothing either.

So both simple rules fail, in opposite directions. The average hides the one bad entry. The maximum finds it, and also finds noise in proportion to the size of the protocol.

#What we chose: a noise floor

The fix keeps the average but changes what goes into it. Only scores of 1.0 or more count:

F     = RMS of the shape scores that are >= 1.0
A     = RMS of the request/reply scores that are >= 1.0
joint = sqrt(F^2 + A^2)              flag when joint >= 3.0

The floor of 1.0 is not a tuned constant. It falls out of how the score is defined: 1.0 is the baseline's own p99, the edge of normal. Anything below it is inside the range the project produces every day, and averaging it in can only dilute.

%% caption: Figure 1. Combining one session's scores. Entries under 1.0 are inside the normal range and never reach the average.
flowchart TD
    S["All scores for one closed session"] --> Q{"Score 1.0 or more?"}
    Q -->|"no"| D["Dropped as normal noise"]
    Q -->|"yes, shape"| F["F = RMS of elevated shape scores"]
    Q -->|"yes, request/reply"| A["A = RMS of elevated pair scores"]
    F --> J["joint = sqrt(F^2 + A^2)"]
    A --> J
    J --> T{"joint 3.0 or more?"}
    T -->|"yes"| X["Flag the session"]
    T -->|"no"| N["Normal"]

That gives three properties we wanted:

1. A single bad entry is judged on its own. With one elevated score, the RMS of one value is that value, so the joint score equals z. One pair at 4.1 among nine healthy pairs scores 4.1 and is flagged.
2. Lots of small wobbles do not add up. If k entries all sit at the same mildly high value v, the RMS is v, whatever k is. Ten scores of 1.4 give 1.4, not flagged. The protection the original design wanted against cardinality survives.
3. Two independent kinds of evidence reinforce each other. Because sqrt(F^2 + A^2) is at least as big as either part, a shape at 2.2 and a request/reply pair at 2.2 give 3.11 and flag, even though neither crossed 3.0 alone. A message type behaving oddly and a reply slowing down in the same session is more suspicious than either one. That is deliberate.

#How we decided

We wrote the scenarios and the answer we wanted for each before comparing rules, then ran every candidate against them. It is the most useful habit we took from this bug:

scenario                          RMS of all   max    floor + RMS   wanted
one pair at 4.1, nine healthy        1.3       4.1       4.1        flag
ten entries at 1.4                   1.4       1.4       1.4        no flag
everything healthy (~0.01)           0.01      0.01      0.0        no flag
shape at 2.2 plus pair at 2.2        1.42      2.2       3.11       flag
one at 4.0, nine at 1.5              1.9       4.0       1.9        debatable

The floor rule matched every row we had an opinion on. The last row is the one we argued about.

#The trade-off we accepted

With one bad entry at z and m others moderately elevated at v, the joint score is sqrt((z^2 + m * v^2) / (m + 1)). One entry at 4.0 with nine at 1.5 gives 1.9. A genuinely bad reply can still hide, this time among neighbours that are all somewhat high rather than healthy.

We accepted it on purpose. When ten different message types in one session are all running above normal, the likeliest cause is the session itself, such as a slow mobile connection, rather than one broken server handler. The maximum would flag that session and blame one message type for it. We wrote the case down next to the code so nobody has to rediscover it.

#Why a single healthy spike rarely reaches 3.0

For a session with exactly one elevated score, the floor rule and the maximum agree, so it is worth asking what stops ordinary noise from producing that one score. Three parts of the design do:

• Calibration. Nothing is scored until a project has seen 1,000 sessions, and a shape or pair only gets a score once it has appeared in 20 sessions. Rare, noisy entries never get a vote.
• Medians, not maxima. The spread score compares the session median of a shape with the baseline. A median over many messages is stable. For a reply with a p50 of 40 ms and a p99 of 250 ms, reaching 3.0 needs the whole session's median at 670 ms or more. One slow message cannot do that on its own.
• A high bar. 3.0 is three times the baseline's own p50-to-p99 spread beyond the median, measured in the project's own units, so a naturally jittery app and a very steady one are each judged against themselves.

#We also stopped calling it sigma

The dashboard used to show these scores with a σ and call the threshold "3σ". They are not standard deviations. The spread score uses the p50-to-p99 distance precisely because latency is heavy-tailed, and a few huge outliers inflate a standard deviation enough to make everything look normal. The σ made the numbers sound more rigorous than they are, so the labels now just say "score" and explain what 1.0 and 3.0 mean.

#What to take away

• Averaging scores to find a problem dilutes the problem with every healthy neighbour. Check what the mean of many zeros does to your one real signal.
• Taking the maximum turns every extra score into another chance of a false alarm. Your false alarm rate will quietly depend on how many things you measure.
• Write the scenarios and the answers you want before choosing the rule. A small truth table beats an afternoon of intuition.
• Score with medians against the baseline's own spread, so one slow message cannot flag a session and every app is judged against itself.