Running 60-plus local autonomous agents on a single Windows machine means you are your own SRE team. On 2026-08-11 we found a bug in our own master scheduler’s error handler that had been silently swallowing job crashes — no log line, no dead-letter entry, no alert — for days. It only surfaced because two unrelated agents (a video-promotion job and a livestream-replay job) had been failing on every single scheduled run with zero visibility into why. In this dispatch we walk through the exact one-line dict.get() mistake that caused it, why it hid specifically from our CronTrigger-scheduled jobs and not our IntervalTrigger ones, and the real fix. If you’re running your own APScheduler-based multi-agent daemon and trust that ‘no alerts’ means ‘everything is fine,’ this one is worth ten minutes of your time.
🔒 Members-Only: The AI Fleet Architect #12: The Silent-Failure Bug That Hid Two Real Production Crashes
ByteSize Basic ($7/mo) members get the full root-cause trace, the actual before/after code from our production scheduler (loop_a_sentinel.py), and the two real downstream bugs this silent failure was hiding.
Deep Production Dive: The dict.get() Bug That Cost Us Days of Blind Scheduling
1. The Real Incident (2026-08-11)
Our fleet’s master scheduler is a single Python process built on APScheduler, hosting roughly 40 of our ~65 autonomous agents as either CronTrigger jobs (fixed daily/weekly times) or IntervalTrigger jobs (every N seconds/minutes). Every job goes through one shared `_on_job_error()` listener that’s supposed to log the failure, write a dead-letter entry for operator review, and back off the job’s next run so a broken agent can’t hot-loop.
We went looking for it after noticing two agents — a YouTube-video promotion job and a livestream-replay job — had stale heartbeats with zero corresponding failure logs anywhere. Not one dead-letter entry. Not one alert. As far as our own monitoring was concerned, both agents simply hadn’t been asked to run. In reality, they’d been crashing on every scheduled tick for days.
2. The Root Cause
Our interval-tracking dict stores a real number of seconds for every IntervalTrigger job, and stores `None` for every CronTrigger job (crons don’t have a fixed interval to back off against — they just retry at their next naturally scheduled time). The error handler read that dict like this:
interval = _intervals.get(name, 900)
backoff_seconds = interval * backoff_mult
The bug: `dict.get(key, default)` only returns the default for a *missing* key. For a CronTrigger job, the key exists and its value is `None` — so `.get()` correctly returns `None`, not `900`. `None * backoff_mult` then raises a `TypeError`, and it raises it while the exception handler is still building the arguments for its own logging call — before the log line executes, before the dead-letter write executes, before the health-file write executes. Every one of those safety nets was downstream of a line that itself crashed.
The only trace of any of it was APScheduler’s own generic “Error notifying listener” line, buried in a different log stream than the one we actually watch.
3. The Fix
interval = _intervals.get(name)
backoff_seconds = (interval * backoff_mult) if interval is not None else None
One line. CronTrigger jobs now correctly skip the backoff calculation entirely and just wait for their next scheduled fire time, exactly as designed — but now the log line, the dead-letter entry, and the health-file update all execute first, so a crash is visible instead of invisible.
4. What It Was Actually Hiding
With the handler fixed and actually logging again, two real bugs surfaced immediately:
– One promotion agent’s scheduler entry called its bare `main()` function, which parses CLI arguments against the daemon’s empty argv and exits with `SystemExit(2)` on every tick. It turned out to be a pure duplicate of an already-working, independently scheduled job — so the fix was deleting the redundant entry outright, not patching it.
– A second agent’s scheduler entry had the identical bare-`main()` mistake, but this one wasn’t a duplicate — it needed a real code path that didn’t touch argument parsing at all, so we extracted its dispatch logic into a `run_cycle()` function the scheduler could call directly.
Neither bug was newly introduced. Both had been silently failing since the day they were added to the scheduler. The error-handling bug is what let them run undetected.
5. The Lesson for Multi-Agent Schedulers
If your error handler can itself throw, your monitoring has a hole exactly the shape of that exception. We now treat every exception-handling path in our fleet the same way we treat the agents themselves: it needs its own test, because it’s the thing standing between a real crash and total silence.
Key Takeaways for Builders
- dict.get(key, default) only applies its default for a MISSING key — if the key exists with value None, you get None back, not the default. This is an easy, easy trap once you’re intentionally storing None as a real value.
- If your error/exception handler itself can raise, every downstream safety net in that handler (logging, dead-letter queues, alerts, backoff) never runs. Test the failure path, not just the happy path.
- Silent failures compound: this one bug was actively hiding two separate, unrelated production crashes for days with zero operator visibility.
Leave a Reply