How Reliable wakes an engineer at 3 a.m., and why the alarm can always be stopped

A pager has two jobs, and they pull in opposite directions. It has to be impossible to miss: loud, on the lock screen, through Do Not Disturb and battery savers. It also has to stop the moment the person holding it asks. For a while, the Reliable on-call app did the first job well and could fail the second completely. In a couple of ordinary situations, the alarm could not be stopped at all.

This post follows a page from the error that triggers it to the phone that rings, explains why Android makes that path harder than it looks, and walks through the trap we built by accident and the layered fix.

#From an error to a page

A page starts with an error arriving from the Reliable SDK. The backend fingerprints it with SHA-256 over the project, the error source, a source reference and the title. If an open or acknowledged incident already has that fingerprint, we only bump its occurrence count. An error that fires ten thousand times pages once. Only a new incident goes on to dispatch.

Dispatch decides who to wake:

• Who is eligible. Projects use either simple tiers (tier 1, tier 2, tier 3, then the lead) or an escalation policy made of ordered steps, where a step can require a minimum severity.
• Who is on call right now. If on-call schedules are enabled, the list is narrowed to whoever holds an active shift, and a covering override replaces that schedule's shifts. Critical incidents bypass the filter by default.
• Nobody on call? Then everyone eligible is paged. The comment in the resolver says it plainly: better than silently dropping the alert.

Every engineer who gets paged gets a row in incident_pages with an escalation deadline. The backend then schedules a delayed job in BullMQ that fires exactly at that deadline. The step timeout is the largest of the engineers' own preferences, then the project default, then 20 minutes.

When the job fires, the worker first checks that the incident is still open and still on the same step. If anyone acknowledged it in the meantime, the job does nothing. Otherwise it marks the unanswered pages as ignored, moves to the next step and dispatches again. Because of that check, the worker is idempotent, and BullMQ can safely retry it (it is allowed two attempts, five seconds apart).

%% caption: Figure 1. A page end to end. The server's escalation timer is the source of truth; the phone only makes noise.
sequenceDiagram
    participant SDK as Reliable SDK
    participant API as Backend
    participant Q as BullMQ
    participant FCM as Firebase
    participant P as On-call phone
    SDK->>API: error event
    API->>API: fingerprint, open a new incident
    API->>API: pick step, filter to who is on call
    API->>FCM: high-priority push
    API->>Q: escalate-step, delayed to the deadline
    FCM->>P: page arrives, alarm rings
    alt engineer acknowledges
        P->>API: POST /incidents/:uid/ack
        Q-->>API: job fires later, sees acknowledged, does nothing
    else nobody acknowledges before the deadline
        Q->>API: escalate-step
        API->>FCM: page the next step
    end

Pages go out on four channels: the Reliable app (a socket event plus a push notification), Slack, email through Brevo, and HMAC-signed webhooks.

#The push message, and a decision we reversed

The first version sent a data-only Firebase message. Data-only messages give the app full control: our own code decides what to show and whether to take over the lock screen. The catch is that our code has to run for anything to appear. When Android will not start the app's process, the alert can simply never show.

The current message is hybrid: a notification block that the operating system can display on its own, plus a data block with the incident details for the app. It is sent with priority: high on a dedicated notification channel, tagged with the incident ID. The comment next to it records the trade: we lose the lock-screen takeover in some states, "but we don't drop the alert, which was the bug."

#Why Android makes this hard

Android works hard to stop apps from waking the phone, because most apps that try are wasting battery. Doze mode, background execution limits and aggressive battery savers on some phone brands are all aimed at exactly what a pager has to do. The app relies on a stack of mechanisms, each covering a different gap:

• High-priority push, which is the one kind of message allowed to wake a dozing device promptly.
• A foreground service, a part of the app that Android lets keep running as long as it shows a notification. Ours is declared as media playback and set to survive the app being swiped away (stopWithTask="false").
• A full-screen intent in the "call" category, and an activity allowed to show over the lock screen and turn the screen on.
• Permission to draw over other apps, and a request to be exempted from battery optimisation. The app asks for both on launch.
• The alarm audio stream. The sound loops through the alarm stream with volume at maximum. On devices that respect the alarm stream policy, that plays even in silent mode.

#Three isolates, one alarm

Flutter apps run Dart code in isolates, separate runtimes that share no memory. A page touches three of them:

• The Firebase background isolate, which Firebase starts when a push arrives while the app is in the background or closed.
• The foreground service's task isolate, which lives as long as the service does, independently of any screen.
• The main UI isolate, which exists only while the app is open.

We made the service's task isolate the owner of the alarm, because it is the only one guaranteed to outlive everything else. That way, as the code puts it, sound is never lost between "the push arrived" and "the screen became visible". The incident screen deliberately plays nothing itself when the service is running, to avoid two alarms at once. The isolates talk through two channels: messages sent into the task isolate, and a small shared key-value store. Two keys matter below: reliable.pending_page, which holds the unresolved page, and reliable.alarm_active.

%% caption: Figure 2. Who owns what. The task isolate owns the sound because it is the only one that outlives the screen.
flowchart TD
    FCM["High-priority push"] --> BG["Firebase background isolate"]
    BG -->|"notifyPaging"| T["Foreground service task isolate: owns the alarm"]
    BG --> N["Full-screen notification"]
    T -->|"saves reliable.pending_page"| KV["Shared key-value store"]
    T -->|"launchApp"| UI["Main UI isolate: incident screen"]
    UI -->|"reads pending page on resume"| KV
    UI -->|"stop_alarm on acknowledge"| T

#The trap

With the alarm living in the service, the only thing that could stop it was the incident screen: tapping Acknowledge or Dismiss, or the screen being closed normally. Now list what a half-awake engineer actually does, and check whether a way out exists:

what the engineer does          what happened before the fix
taps Acknowledge                 alarm stops
presses Home                     screen hidden, not closed: alarm keeps ringing
swipes the app away              screen destroyed, service survives by design:
                                 alarm keeps ringing, no screen left to tap
locks the phone and waits        alarm keeps ringing

In two of those rows, the alarm looped forever with no screen and no button on the notification. The only way out was to find the app in Android settings and force-stop it. We had designed carefully for "the page must not be lost" and never enumerated the ways a person leaves a screen.

#The fix: independent ways out

We did not want one better stop button. We wanted several ways out that do not depend on each other, so that no single failure can trap anyone.

1. SILENCE ALARM on the notification. The button is added whenever the alarm is sounding, and it is handled inside the task isolate, so it works with no app screen at all. It deliberately does not acknowledge and makes no network call. The incident stays open, and the server's escalation timer still fires.
2. A hard five-minute cap (kMaxAlarmDuration), enforced by a timer in both the service and the screen. The reasoning is in the code comment: a pager that can ring forever is a broken pager, and unanswered pages are the escalation timer's problem, not the alarm's.
3. The page is remembered. The pending page used to be cleared when it was read. Now it stays until the incident is resolved, so reopening the app puts Acknowledge in front of you instead of a ringing phone with nothing to tap.
4. The notification is not overwritten mid-alarm. A routine refresh of the service notification used to replace it, silently removing the SILENCE button, which is the one guaranteed way to stop the sound. The reliable.alarm_active flag now blocks that while the alarm plays, and it is cleared on stop and on service shutdown so a stale flag cannot suppress later updates.

%% caption: Figure 3. Every path out of Ringing. Only Acknowledge stops escalation; the others only stop the noise.
stateDiagram-v2
    [*] --> Idle
    Idle --> Ringing: page arrives
    Ringing --> Idle: SILENCE ALARM (escalation continues)
    Ringing --> Idle: 5 minute cap (escalation continues)
    Ringing --> Acknowledged: Acknowledge
    Acknowledged --> Idle: server stops escalating

Acknowledging has its own small safety net. The app stops the alarm first, then sends the request. If the request fails, the alarm starts again, because a failed acknowledge that looked successful would be the worst outcome of all. On the server, acknowledging is idempotent. It does not even cancel the delayed escalation job, since we do not store its ID. The job still fires, sees the incident is acknowledged, and does nothing. Designing the worker to check state, rather than relying on cancellation, made this simple.

#The same kind of bug, the other way round

Reviewing this path again turned up a second problem. When a page arrived while the app was open, the incident screen saw the service running and stayed quiet, assuming the service was ringing. But only the background push handler ever told the service to ring. The foreground handler just opened the screen. The result was a red incident screen with no sound, no SILENCE button and no saved page.

The fix was a single call: when a page arrives in the foreground and the service is running, notify the service exactly as the background path does. Two paths that were meant to behave the same differed by one line, and nothing forced them to match.

#How we decided

• The server owns escalation, and the phone only makes noise. This is the decision everything else rests on. Because the escalation deadline lives in a server-side job, the phone is allowed to fail safe: a silenced or timed-out alarm never loses the incident, it just hands responsibility to the next step.
• Keep "stop the noise" separate from "I am handling it". Silencing and the five-minute cap stop sound. Only Acknowledge takes responsibility. Mixing the two would let a sleepy thumb cancel an escalation.
• Enumerate the exits, not just the entrances. The fix came from writing down every way a person leaves a screen and checking each one, the table above, rather than from testing the happy path harder.

#What to take away

• Design the ways out as carefully as the ways in. For every state, ask: if the person does nothing, or does the wrong thing, how does this end?
• Anything built to keep running needs an off switch that works when everything else has failed, including when there is no UI.
• Put the source of truth on the server and let the device be an unreliable, fail-safe messenger.
• When two code paths should behave the same, make them share code or test them together. Ours differed by one line.