Infrastructure

Read-Only Ops CLI Tooling Using Exit Codes for Production Response

Operations scripts that mutate state by default are too dangerous to run during a live incident. To fix this, split read from write at the command line level and make the exit code the interface. The default mode must be a read-only scan that exits 1 when a stuck set is found and 0 when clear. This allows a standard cron job to schedule the exact same script as an automated detector. Gate the repair logic behind an explicit flag that requires a TTY confirmation or a force override for automation.

Symptoms

  • A job or active call session is stuck in the telephony pipeline at 2am.
  • The only diagnostic tools immediately available are a web console and a hunch.
  • A script exists that would definitively tell you what is broken, but no engineer will run it during an active incident because they fear it might also change something.
  • A diagnostic script prints that a queue is blocked but exits zero, preventing automated schedulers from detecting the failure.

What is actually happening

Tooling that mutates state by default is tooling nobody dares run under pressure. When the system is broken and an operator is operating on low sleep, they will actively avoid running a command if they are not entirely sure what side effects it carries. Because the script does not get run, the incident lasts longer. You end up diagnosing the system through a web console instead of the command line interface built exactly for this purpose.

Separately, ops scripts frequently print a verbose description of a problem to standard output and then exit 0. A script that exits zero cannot be consumed by anything automatic. A standard scheduler evaluates success or failure based on the exit code. When the script exits zero, the scheduler assumes the system is healthy. This tooling only helps when a human is already looking directly at the terminal window. That is precisely the situation you are trying to stop depending on. Automation requires an unambiguous non-zero exit code to initiate remediation without waiting for human observation.

How to confirm it

  1. Run your existing diagnostic script against a known stuck queue without passing any arguments. Expect the command to either mutate the state or prompt for an immediate destructive action. If it does either, you have a mutating-by-default tool.
  2. Execute the script against a broken pipeline and immediately run echo $? in the shell. Expect the output to be a 0. This confirms the script speaks to humans via standard output but lies to automation via the exit code.
  3. Search the script source code for a dry-run flag. Expect to find none, or find one that prints a summary rather than the exact database queries or API calls it intends to make. This confirms operators cannot safely preview a repair.

How it works

The fix is to split read from write at the command level and make the exit code the primary interface. One script provides three distinct postures, and the safe posture is always the default.

The default mode is strictly read-only. A one-shot scan queries the system for stuck jobs. If the stuck set is non-empty, the script exits 1. If the pipeline is clear, it exits 0. Because the exit code accurately reflects the health of the system, scheduling this script in cron immediately turns it into an automated detector. No extra infrastructure is built. You do not need to stand up a metrics pipeline to get alerted on a stuck queue.

For live debugging, the script provides a live tail mode. This mode subscribes to the exact same query logic used by the one-shot scan. You can watch the pipeline drain or back up in real time while you are actively working the incident, using the exact same code path that cron uses to evaluate health.

The mutating repair logic sits behind an explicit, separate flag. Running the repair requires a TTY confirmation prompt. The engineer must type a confirmation to proceed. For automation, a force escape flag bypasses the TTY check, but it must be passed explicitly.

Finally, the repair mode includes a dry-run variant. Instead of printing a generic summary, the dry-run resolves the explicit mutation payload. It prints the exact, literal write set it would make against the state store, and then it performs none of them. Inspecting the write set beforehand confirms the mutation targets only the identified stuck set.

What I would check first

  1. Run the read-only scan repeatedly across successive intervals. A stuck set that is actively draining is a fundamentally different problem from a set that is growing. The trajectory dictates whether you need to intervene immediately or let the system catch up.
  2. Check whether the stuck set is growing or draining across successive runs. A draining set indicates a temporary processing backlog, whereas a growing set points to a stalled pipeline or blocked downstream dependency.
  3. Check the contents of the stuck set in live tail mode while incoming calls arrive. Observe whether new telephony sessions immediately stick or if the issue is confined to a fixed batch of historical sessions. If only historical sessions are stuck, investigate stale state left behind by an earlier restart; if new calls continue sticking, inspect real-time routing.

The value of cron-friendly exit codes

Every stateful system needs a detector before it needs a repair. On a small self-hosted deployment, standing up a full metrics pipeline carries its own operational cost. You have to monitor the monitoring.

By making a single script speak in cron-friendly exit codes, you gain an immediate, zero-dependency detector. The script runs on an interval. If it exits 1, cron or systemd can immediately fire off an alert or execute a separate mitigation task. This pattern is often the right first move before a dedicated observability stack is worth the engineering cycles.

Evaluating successive runs

The current limitation of this pattern is that it evaluates the size of the stuck set at a single instant. The exit code tells you if the queue is blocked right now.

To know if the problem is getting worse, you need to measure the size of the stuck set across successive runs. A stuck set that shrinks across consecutive runs indicates that workers are actively processing the backlog. A stuck set that expands across runs indicates that processing is stalled or that incoming traffic exceeds capacity. Tracking this delta requires the runs to be retained somewhere. Nothing in this script retains them today, which is the honest gap in this specific implementation. Comparing the queue across successive scans requires retaining run output over time, though current tooling does not persist prior runs.

Common questions

Why does a diagnostic script need to exit 1 on failure?

A script that prints a failure but exits zero cannot be monitored by standard automation. Setting the exit code to 1 allows cron or systemd to detect the failure automatically without human intervention.

How do you prevent accidental state changes during incident response?

Make the command line tool read-only by default. Put any mutating operations behind an explicit repair flag that requires a TTY confirmation prompt to execute.

What makes a useful dry-run implementation?

A dry-run must print the exact writes and database updates it intends to make, rather than a generic summary. It then skips executing those writes, allowing operators to verify the exact mutations before execution.

How do you track if a stuck queue is draining or growing?

You must measure the size of the stuck set across successive runs. This requires saving the output of previous scans, as evaluating a queue at a single instant cannot reveal its trajectory.

Infrastructure

LiveKit SIP Trunk Does Not Exist After A Container Recreate Or Host Reboot

After a reboot or container recreate, outbound dialling fails silently. How to persist Redis state, configure systemd, and self-heal stale trunk identifiers.

Infrastructure

Deploying a LiveKit Agent Worker Without Dropping Active Calls

Restarting a LiveKit voice agent drops every live call. How to deploy long-lived workers by coexistence, drain them safely, and snapshot host state for rollback.