Scheduling and capacity
Scheduler circuit breaker trips during low traffic and stops dialing
Short answer
A scheduler circuit breaker that monitors an event stream for freshness will eventually manufacture its own outage. When traffic naturally drops, the webhook stream goes quiet. If the breaker trips on this silence, it suspends dialing, which guarantees no new webhooks arrive, locking the system in a false outage. You fix this by requiring two independent staleness signals before engaging the breaker. When you combine webhook freshness with a reconciliation poll from a separate system, a single stale signal becomes a warning and dispatch continues normally.
Symptoms
- The dialler drops into conservative mode and stops placing outbound calls entirely.
- This drop happens during a perfectly healthy quiet hour with zero underlying infrastructure faults.
- System event logs point to a webhook stream or an event consumer that has gone stale or silent.
- The misleading observation is assuming the webhook endpoint itself is broken when the stream is simply starved of calls.
What is actually happening
Staleness detectors on event streams have a built-in circularity when they govern a distributed scheduler. The event stream is fed by the very activity the circuit breaker is designed to suspend.
If the freshness of a webhook endpoint is your only health input, low traffic looks identical to a broken endpoint. When call volume naturally drops below the rate required to satisfy the freshness threshold, the circuit breaker trips.
Tripping the breaker suspends the dialler. A suspended dialler produces zero new calls. Zero new calls produce zero new webhooks. The breaker has now manufactured the exact outage it was built to contain. It then keeps itself tripped indefinitely because a suspended dialler will never generate the events required to clear the fault and resume traffic.
How to confirm it
- Query your system logs for the exact moment the circuit breaker engaged. Expect this timestamp to correlate with a natural drop in user demand or scheduled call volume, not with an error spike or a failed deployment.
- Inspect the outbound queue dispatch logs for the window immediately preceding the trip. Expect to see a completely empty queue with no active jobs, rather than a queue full of jobs failing to dispatch.
- Verify that regular dispatch clears the breaker automatically as soon as call activity produces a fresh event. Expect the circuit breaker to clear automatically without human intervention once either signal reports fresh.
How it works
I require two independent staleness signals before engaging the circuit breaker.
In my production setup, these two signals are webhook freshness and the freshness of the reconciliation poll. These two inputs come from different systems and travel completely different network paths. The media server pushes webhooks to an HTTP endpoint, validating the external network ingress. Meanwhile, a separate task queue drives a self-rescheduling handler for internal state reconciliation. If this internal loop stops silently, it is usually because a self-rescheduling task chain stops silently without logging any errors.
When both signals go stale, that is real evidence something structural is broken. If only one goes stale, it serves as a warning line and nothing more. The system records the single-signal staleness as a warning with the specific signal named, making these near-misses countable. Call dispatch continues normally without dropping into conservative mode.
Recovery is automatic. When either signal returns, the breaker clears instantly. There is no manual reset step required. Dispatch resumes without requiring an operator to clear state flags or restart scheduler processes.
What I would check first
- Check the media server event stream freshness against scheduled outbound calls to distinguish an idle quiet hour from a failure. If the stream is silent purely because zero calls were dispatched, the candidate cause is call starvation rather than media server failure. Expect media server health checks to pass normally even when the webhook event stream is completely silent.
- Check the reconciliation poll task queue to determine whether internal state updates have stalled independently. If the self-rescheduling task chain stopped running, the reconciliation poll signal will go stale while media server webhooks remain unaffected. Expect task queue logs to show dropped executions or task timeouts if the reconciliation handler is the failing component.
- Check whether both signals are stale simultaneously to separate a structural failure from a single-path warning. If only one signal is stale, dispatch should continue normally while recording a single-signal warning. Expect an actual system outage only when both the HTTP webhook stream and the reconciliation task queue are stale.
Why a single signal is just a heuristic
This architectural vulnerability generalizes to any circuit breaker whose input is fed by the traffic it would suspend. Job runners, webhook consumers, and outbound message queues all share this identical failure mode if configured poorly.
If you can only think of one signal to monitor, you are building a heuristic. It will fail on weekends, holidays, or overnight when traffic drops. You must pick two signals that fail for different reasons and require both to be stale before acting.
If you lengthen the freshness threshold to survive a quiet night, the breaker will take just as long to respond to a genuine daytime outage.
Instrumentation that prevents manual reconstruction
When a breaker does legitimately trip, diagnosing it later requires highly specific instrumentation. I log the freshness age of each signal independently. I never combine them into a single aggregate health value, because doing so destroys the forensic evidence of which path failed first.
I also record the maximum time spent in the engaged state per flip. This specific metric is written directly to the event log. By recording the maximum time spent flipped as an explicit duration, a breaker that engaged and never cleared can be recovered after the fact as a raw number. If you omit this, you are forced to reconstruct the outage duration by manually crawling the database for an absence of call records, which is slow and error-prone.
For distinguishing scheduler-owned states from media server states, see the scheduler holds stale capacity after a call ends or kills a live session prematurely.
Common questions
Why does the scheduler circuit breaker trip when there are no errors?
It trips because the staleness detector interprets a quiet traffic period as a broken webhook endpoint. The single signal cannot distinguish between low call volume and a failed delivery path.
How do I recover a dialler stuck in conservative mode?
Wait for regular call dispatch to produce an event, as recovery is automatic when either signal returns. Then reconfigure the breaker to require a second independent signal before it trips again.
What two signals should a scheduler use for health checks?
Use webhook freshness from the media server and the freshness of an internal reconciliation poll driven by a task queue. These travel different paths and fail for entirely different reasons.