Infrastructure
Deploying a LiveKit Agent Worker Without Dropping Active Calls
Short answer
A voice agent is a long-lived stateful worker holding live websocket sessions. Restarting it in place is not a deployment; it is a mass disconnect that a client cannot transparently retry. To deploy safely, treat the host as the source of truth, snapshot the exact running image and unit file, and start the new worker alongside the old one. You must size the deployment script’s drain window to exceed the LiveKit SDK’s default 1800 second drain timeout. If the deploy script exits and kills the old unit early, it severs the exact calls the drain existed to protect.
Symptoms
- Deploying a new agent build drops every active call currently in progress.
- You need to revert to the previous build but cannot, because the Docker tag was reused (like a branch name) and nobody recorded the exact digest the previous container ran with.
- You look at the host server and are simply not certain what code is running on the box right now.
- A misleading observation: The deployment pipeline reports a successful rollback after a
git revert, but the broken behaviour continues in production because the unit file and environment variables on the host were left entirely untouched.
What is actually happening
An agent worker is a long-lived process holding live realtime websocket sessions. Restarting it in place is not a deployment; it is a mass disconnect. Unlike an HTTP request, a dropped WebRTC call cannot be transparently retried by the client. When a standard deployment script updates a systemd unit or a container, it stops the old process and starts the new one. The stop command sends a termination signal, and if the process does not exit fast enough, the supervisor sends a kill signal.
The LiveKit Agents SDK has a built-in drain mechanism. When it receives a termination signal, it stops accepting new jobs and waits up to 1800 s by default for existing sessions to finish. The failure happens because the deployment script or the process supervisor is completely unaware of this 1800 s window. If your deploy script waits less than the SDK’s 1800 s timeout, it kills exactly the calls the drain existed to protect.
The second half of the problem is rollback failure. What is running in production is not fully captured in version control. The running state consists of an image tag, a runtime environment, and a unit file, all located on a host machine. Rolling back the code repository leaves the other three components untouched. A git revert is not a rollback. If you deploy by overwriting the existing unit and the new build breaks, you have no local record of the exact configuration that was working five minutes ago.
How to confirm it
- Read the running unit file’s
TimeoutStopSecdirective (or equivalent Docker stop timeout). Expect the configured timeout to be set lower than the SDK’s 1800 s drain window, confirming the supervisor kills the draining process prematurely. - Inspect the LiveKit agent worker logs during a deployment. Expect to see the SDK drain initiation message, followed shortly by an abrupt termination signal before the active session ends, confirming the process is not crashing on its own but being forcefully stopped.
- Compare the image tag of the running container on the host against the tag defined in version control. Expect to find a floating tag like
mainorlateston the host, confirming that a code revert in version control does not guarantee the host will pull or run the correct historical container.
How it works
I treat the host as the single source of truth and make the deployment tool prove what is actually there.
I inventory the system first. The deployment script enumerates the systemd units on the host and reads the running image out of each service definition. You have to look at the box itself rather than relying on your assumptions about what the pipeline last did.
I snapshot the state before touching anything. The tool captures the exact image reference, the runtime configuration (capturing variable names only, with secret values redacted), the source tree, and the unit file. It writes these into a timestamped, access-restricted directory on the host alongside a manifest file. Doing this means a rollback is simply selecting an entry from a local list rather than attempting to reconstruct history from version control.
I deploy by coexistence. The tool starts the new worker alongside the old one and lets the old one drain instead of restarting it. The new unit connects to the LiveKit server alongside the old one to listen for new job dispatches. The LiveKit server handles the routing, sending new calls to the newly deployed worker while the old worker finishes its active sessions.
I size that deployment drain window to exceed the SDK’s own drain timeout. The agents SDK will wait up to 1800 s by default for sessions to finish. The deployment script must wait at least that long before it considers the old unit safe to remove.
What I would check first
- Find the termination signal and exit code on the old worker process during a deployment. An exit code of 0 means the agent finished its calls and decided to leave, ruling out an abrupt kill. A forced termination signal means your deploy script or supervisor killed the worker before the drain finished, pointing directly to your deployment timeouts.
- Compare your deploy script’s wait budget against the SDK’s 1800 s drain timeout. If the deploy script enforces a timeout shorter than 1800 s, it cancels the drain and kills the active sessions prematurely. This rules out the worker machine as the source of the timeout and points to the deploy script limits.
- Inspect the environment variables on the host versus version control. Finding configuration drift on the host rules out a purely code-based rollback working correctly, pointing instead to the lack of a local snapshot mechanism.
Why long lived stateful workers require coexistence
This deployment pattern generalizes to any long-lived stateful worker. Media workers, websocket gateways, streaming consumers, and game servers all share the same operational constraints. Blue/green deployment in this context is not about achieving zero-downtime HTTP routing. A load balancer can easily hold an HTTP request for a few milliseconds while it swaps a backend target.
Voice agents are different. They hold stateful WebRTC sessions that clients cannot transparently re-establish. If an HTTP request fails, the browser retries it and the user never knows. If a voice agent drops a websocket connection, the caller hears dead air, the call drops, and they have to dial the phone number again. Deploying by coexistence is the only way to respect the lifespan of these connections.
Instrumenting build attribution
When you deploy by coexistence, you will inevitably have two different builds of your agent answering calls at the exact same time. The old build handles the calls that started before the deployment, and the new build handles the calls that start after.
If a caller reports a failure during this crossover period, you need to know which build they were talking to. I capture the image tag that each running unit reports at deploy time. Capturing the resolved digest instead of just the tag would be an improvement, though it is not something the tool does today.
Build attribution for a specific call is a property of the evaluation harness here, not a default feature of production call records. You have to treat per-call build attribution as something you need to build into your logging pipeline, not something you can assume the SDK will provide out of the box. In this architecture, per-call build attribution is instrumented inside the evaluation harness rather than production call records, so production call logs cannot be assumed to record the running build out of the box.
Common questions
Why do active LiveKit calls drop during deployment?
A deployment replaces the running process. If the deployment script sends a kill signal before the LiveKit SDK finishes draining its active sessions, the websocket connections are severed instantly. WebRTC clients cannot transparently reconnect a dropped call, so the user experiences a failure.
How do I roll back a Docker deployment on a plain VM?
You cannot rely on version control alone to roll back a VM. You must snapshot the running image reference, runtime configuration variable names, source tree, and unit file into a local directory before deploying. To roll back, you restore those exact files and restart the service.
What is the default drain timeout for LiveKit agents?
The LiveKit Agents SDK waits up to 1800 s by default for active sessions to finish after receiving a termination signal. Your deployment scripts and systemd unit files must be configured with a timeout exceeding 1800 s to prevent killing active calls.
How do I deploy a voice agent with zero downtime?
You must deploy by coexistence rather than replacing the process in place. Start the new worker alongside the old one, bound to the same LiveKit server. Let the old process drain its existing calls while the new process picks up all incoming jobs.