Most teams wire up heartbeats and stop there. A job posts a ping on success. If the ping stops, something fires. That last part is where the design work actually lives, and it is also where most setups go quiet in the wrong way.
A missed heartbeat is not a monitoring event in the abstract. It is a person who needs to decide whether the job is late, broken, or still running somewhere the signal cannot see. If you have not named that person before the miss happens, the page lands wherever your default routing happens to point. Default routing is rarely the same as ownership.
Ownership before the miss
Scheduled work has owners even when nobody wrote them down. The nightly invoice export belongs to billing ops. The retention purge belongs to the data team. The certificate renewer belongs to whoever last touched the ACME client. When those jobs go silent, the useful page is the one that reaches the owner, not the one that reaches whoever is currently holding the ops phone.
That sounds obvious until you look at how most heartbeat alerts are wired. The signal goes into a shared channel. The shared channel pages the on-call rotation. The on-call rotation is staffed by people who keep the platform up, not by people who know whether a silent batch job is supposed to finish at 02:15 or 03:40. They can restart a host. They cannot tell you whether yesterday's ledger file was meant to be empty.
So the first design choice is not the grace window. It is the destination. For each monitored schedule, write down who is allowed to say "this miss is real" and who is allowed to say "this miss is expected." Those are often the same person. Sometimes they are not. Either way, the answer should not be "whoever is awake."
Platform on-call still has a role. Infrastructure failures that take out a whole class of jobs belong there. A single missed heartbeat for one tenant export usually does not. If your routing cannot tell those cases apart, every miss becomes an infrastructure ticket, and the people who can fix the job never see the alert until morning.
A single #cron-alerts channel is convenient to set up and expensive to run. Convenience is the problem. Once every heartbeat lands in the same place, the only routing that remains is human attention, and human attention does not scale past a handful of jobs. Better practice is boring. Each schedule has a destination that matches the owning team. Billing jobs page billing. Data jobs page data. Platform jobs page platform. The shared ops rotation gets the residue: jobs with no owner yet, or misses that look like scheduler or network failures rather than application ones.
You can still keep a read-only feed for the whole company. Visibility and paging are different jobs. A feed lets people watch patterns. A page asks someone to act within minutes. Mixing them is how you train people to ignore both.
Escalation belongs in the same design. If the owner does not acknowledge within a window that matches the job's actual risk, escalate to their backup, then to platform. Do not escalate on the first miss of a weekly report the same way you escalate on a payment settlement that must finish before banks open. The severity is a property of the work, not a property of the monitoring product.
What the person on the page needs
A page that says "heartbeat missed" is half a sentence. The other half is context the person will have to reconstruct under time pressure if you do not attach it.
They need the schedule identity, not just a friendly name that drifted six months ago. They need the expected cadence and the grace window you chose, so they can tell "late" from "gone." They need the last successful heartbeat time and, if you have it, the last known exit status or log pointer. They need a short runbook link that starts with what to check first, not a wiki homepage.
They also need a clear stop condition. Some misses resolve themselves when a long-running job finally finishes and posts late. Some do not. If the person on call cannot tell which case they are in, they will either wait too long or page too many people. Both outcomes are worse than a slightly noisier alert that includes the right facts up front.
One more detail that gets skipped: who can silence the alert, and for how long. Temporary silence is useful when a known outage is underway. Permanent silence without an owner change is how a critical schedule disappears from monitoring while still running in production. Treat silence as a change that needs a name next to it.
When the signal lies
Sometimes the wrong person gets paged because the signal was never truthful. A job posts its heartbeat at the start of the run instead of the end. A container restarts and posts success before the work finishes. A shared runner posts on behalf of several schedules and one of them dies quietly. In those cases the routing looks fine and the page still lands on the wrong desk, because the miss arrived late or not at all.
Fix the signal first when you see that pattern. Heartbeats should mean "the work completed within the rules we agreed," not "the process started" or "the host is alive." Once the signal is honest, ownership routing starts to make sense again.
Schedule drift creates a related failure. If a job slowly slides later each night, the grace window may still catch it, but the person who gets paged at 04:00 may not be the person who owns the morning business process that depends on the output. Drift detection helps here because it surfaces the problem before a hard miss. Route drift warnings to the same owner as the hard miss. They are the same ownership question asked earlier.
If you are setting this up from scratch, start with a short inventory rather than a clever escalation graph. List the schedules that matter when they fail. For each one, name a primary owner, a backup, and a severity that matches the business impact. Wire the heartbeat to that primary. Put the runbook URL in the alert body. Keep platform on-call as the escalation path for true infrastructure failures and for orphans.
Then review the inventory when people change teams. Ownership rot is quieter than signal rot and just as dangerous. A page that still goes to last quarter's owner is a page that will bounce until the window closes.
You do not need a perfect org chart to make this work. You need a place where "who gets paged" is an explicit field next to "what is monitored." Most silent-job incidents are not mysterious. They are ownership problems that waited until 2 a.m. to introduce themselves.