What Happens When It Breaks?

The Fear Is Correct

The fear goes like this: I automate the report, everyone stops doing it by hand, then one day the automation quietly dies and nobody notices until the numbers are wrong in front of a customer. That fear is correct. Silent failure is the default state of automation, not the exception.

Software rarely announces its own death. A credential expires, a system update revokes a permission, a machine is asleep at the scheduled hour, and the job simply does not run. No error lands on anyone's desk, because nobody built a desk for errors to land on. An automation you cannot audit is worse than the manual process it replaced, because the manual process at least had a human who would notice.

Supervision Is Architecture

The fix is architectural, and it costs a fraction of the automation itself. Every automated job needs a place to report, and the report has to include success, because a job that only speaks up when something goes wrong looks exactly like a job that has died.

On top of the reporting sits a supervisor: one process whose entire purpose is checking that every other job ran recently and paging a human when anything goes stale. We run nine production agents this way, and one of them exists only to watch the others. Unsupervised automation is a rumor, not a system.

• Every job reports to a channel a human already reads, on success and on failure.
• Silence counts as failure. A missed check-in is itself an alert, never just an absence of news.
• A supervisor process verifies every job's freshness daily and pages a human when anything goes stale.
• Expect breakage at the edges: expired credentials, changed permissions, a machine asleep at the scheduled hour.

The supervision loop

Every automated job reports somewhere a human actually looks, and something checks the checkers.

Our Outage Ran Silent for Weeks

We learned this the expensive way. Early in our own agent operations, a single configuration change broke authentication on the machine that runs our scheduled jobs. Every one of them began failing, quietly, and kept failing for weeks before anyone noticed. Nothing crashed visibly. The work just stopped arriving, and we had grown used to not watching.

The supervisor agent we run today exists because of that outage. It checks every agent's freshness each morning and pages our ops channel the moment anything goes stale. Recently it caught a permissions change that had killed one agent overnight: one red, eight green, fixed before lunch. Supervised systems still fail. They just cannot hide it.

Audit Yours With One Question

If you already run automation, audit it this week with one question: if this broke last night, how would I know, and when? Any answer longer than a sentence means you would not know.

When you buy automation, make supervision a purchase requirement rather than a wishlist item. Ask the vendor where failures surface, whether success is confirmed or merely assumed, and what watches the watcher. Budget for the edges too, because that is where real systems break: expired logins, changed permissions, a machine that was off at the scheduled hour. The logic is usually the reliable part. The plumbing around it is what fails, and the plumbing is what needs eyes.