Scheduling and capacity
A self-rescheduling task chain stops silently without logging any errors
Short answer
When a self-rescheduling task chain dies, it leaves no trace. The loop exists as a handler that does work and then enqueues its own next execution. If an instance is killed mid-deploy or a queue rejects the enqueue, the chain breaks. Because the chain is its own retry mechanism, nothing brings it back. To fix this, put the revival outside the chain. An independent heartbeat function re-enqueues the polling task on a cadence, using a deterministic task name derived from the target window. The queue rejects duplicates, making the heartbeat a free no-op while healthy.
Symptoms
- The scheduler stops placing outbound calls entirely, and the first person to notice is a customer asking why nobody was called.
- Error rates remain perfectly flat because there are no running executions left to fail.
- There are no failed tasks, no rejected dials, and no stack traces in your logging pipeline.
- Task queue depth drops to zero for the polling handler, while other queues remain active.
- Dial activity stops completely while the queue sits idle, giving no indication of failure until a customer reports that calls were missed.
What is actually happening
A loop implemented as "handler runs, then enqueues its own next execution" holds its own liveness in a single link. The chain perpetuates itself sequentially. Every execution must succeed in placing the next execution onto the queue before it finishes processing.
If any execution dies between doing its work and enqueueing the successor, the chain ends. This exact timing window is small, but it is hit constantly in a busy system. An instance might be killed mid-deploy. The queue broker might temporarily reject the enqueue payload. An unhandled exception might trigger on a new payload shape, crashing the worker process before the final line of code executes. An operator might even delete a stuck task by hand during an incident.
When that link breaks, the silence is total. Nothing retries the loop, because the thing that would have retried it was the loop itself. You do not see an error rate climb because there is no execution left to fail. The task queue simply empties out its remaining jobs and sits idle. The system thinks it has successfully completed all its work.
How to confirm it
- Query the task queue for the current active or pending executions of the polling handler. Expect to see exactly zero tasks if the chain is dead, confirming that the loop has stopped entirely.
- Query the execution history for the timestamp of the last successful execution of the self-rescheduling task. Expect the time elapsed since that run to vastly exceed your normal polling cadence, pinpointing the exact moment the chain broke.
- Filter your infrastructure logs for deployment restarts, broker enqueue rejections, or manual task deletions that occurred at that exact timestamp. Expect to find a hard termination event that aligns perfectly with the last successful poll, explaining why the next enqueue never happened.
How it works
To make a self-rescheduling loop resilient, you put the revival mechanism outside the chain. I run a separate scheduled function on an independent trigger that attempts to re-enqueue the poll on its own cadence. This external reviver acts as a safety net.
The critical detail is that this heartbeat must use a deterministic task name, derived from the target window rather than a random identifier. The heartbeat computes a task name using the current timestamp truncated to the boundary of the target window.
When the heartbeat fires, it attempts to enqueue this deterministically named task. If the chain is alive, the handler has already enqueued that exact name for the current window. The task queue evaluates the payload, rejects the enqueue as a duplicate, and the heartbeat becomes a free no-op.
If the chain is dead, the queue accepts the heartbeat’s enqueue. The task lands, the worker picks it up, the handler runs, and the self-rescheduling loop restarts on its own.
Deterministic naming is what makes this safe to run continuously. Without it, the heartbeat bypasses deduplication and forks a second chain every time it fires. You end up with multiple loops racing to process the same schedules, pulling the same calls from the database. I wrote about the consequences of runaway concurrent execution in Task Queue Executions Climb Indefinitely With No Visible Errors.
On queues without native name-based deduplication, you achieve the exact same effect using a conditional insert on a unique key in your database. The heartbeat tries to write a lock record for the window. If the write fails with a uniqueness constraint, the loop is alive. If the write succeeds, the heartbeat fires the enqueue command to the broker.
Because the heartbeat runs independently, it needs a way to be turned off. I put a stored flag kill switch on it in the database. The heartbeat checks this flag before attempting the enqueue. This allows the heartbeat to be silenced during planned maintenance windows without requiring a deployment.
What I would check first
- Observe the task queue depth for the polling handler. Expect zero depth if the chain is broken. If the depth is exactly one but the task is pending indefinitely, you have starved or disconnected workers, not a broken scheduling chain.
- Inspect the age of the most recent poll execution in your metrics. Expect a gap of minutes or hours. If the poll ran seconds ago but no calls were placed, the chain is perfectly healthy, and your database schedule query is simply returning empty.
- Check worker logs and queue broker events around the last execution timestamp. Expect an unhandled exception on a new payload shape or a broker rejection if the handler crashed before scheduling its successor. If no error appears in the logs and the queue is empty, inspect host deployment events for an unhandled kill signal that stopped the process mid-run.
Monitoring the revival mechanism
Once the external heartbeat is in place, it becomes the primary lens into the health of the scheduling loop. I instrument this by tracking the count of heartbeat enqueues rejected as duplicates versus those accepted. In a healthy system, the accepted count stays at zero indefinitely. Every heartbeat is rejected because the loop is already running and has claimed the deterministic time window ahead of the external trigger.
A single accepted heartbeat is a non-issue. It means the chain died, the heartbeat caught it, and the system self-healed exactly as designed. However, a sustained rise in accepted heartbeats means the chain is dying repeatedly. This indicates a poisonous payload in the schedule or a persistent infrastructure failure that is crashing the handler on every run.
The age of the most recent poll execution is another vital metric. Because the polling loop is the heartbeat of the scheduling layer, its staleness is a high-confidence signal that the layer is failing. I use this staleness as one of two independent signals for my outbound dialing circuit breaker. Because poll freshness is one of two independent signals required by the circuit breaker, a stale polling loop arms the breaker without tripping it on low call traffic alone.
Why self-reschedule at all
A self-rescheduling handler controls its own concurrency. It only schedules the next tick after the current one finishes processing the batch. If the database is slow or a third-party API degrades, the polling cycle naturally stretches. The system degrades gracefully by checking less frequently, maintaining a concurrency of exactly one active polling worker.
A pure cron trigger fires blindly on the clock. If the handler takes longer than the interval to complete, a pure cron overlaps executions. Multiple workers wake up, query the same pending scheduled calls, and race to place them.
Common questions
Why did my self-rescheduling task stop without logging any errors?
The task process was likely interrupted after doing its work but before enqueueing its successor. Because the loop relies on itself for retries, a single interruption breaks the chain permanently and silently.
How do I restart a dead polling loop safely?
Use an external heartbeat function that runs on an independent trigger. It must attempt to enqueue the loop using a deterministic task name so the queue deduplicates it if the loop is already running.
Why use deterministic task names for a heartbeat?
Deterministic naming allows the task queue to reject duplicates. Without it, the heartbeat would spawn a new concurrent loop every time it fires, leading to runaway execution and overlapping tasks.
Can I just use a cron job instead of a self-rescheduling loop?
A self-rescheduling loop controls its own concurrency, preventing overlap if tasks run long. A pure cron job fires blindly, which can cause multiple workers to race over the same data if the system degrades.