Skip to main content
The control plane deployment service creates deployment records and delegates execution to Restate workflows.

​
Virtual object keying

Each Restate VO uses the narrowest key that gives the serialization it needs:
  • DeployWorkflow is keyed by deployment_id. Each deployment is one workflow run, so multiple deployments in the same environment can build in parallel. It is a workflow rather than a virtual object because Deploy has no operations to serialise.
  • EnvironmentService is keyed by environment_id. Promote and rollback run here. Both change which deployment is live in one environment, and both check the live pointer before they swap it. One key holds the check and the swap under one lock. With separate keys, two operations on the same environment can pass their checks against the same live pointer and then swap one after the other.
  • RoutingService is keyed by env_id. Frontline route assignment and the live-deployment swap serialize here. It is the write primitive that DeployWorkflow and EnvironmentService call.
  • DeploymentService (desired-state transitions, scheduled and immediate) is keyed by deployment_id.
Build concurrency has no virtual object. Build is a second handler on DeployWorkflow that Deploy calls on its own key, in the Restate scope builds with the workspace id as the limit key, and Restate’s concurrency rules cap how many run at once per workspace. See Build concurrency. Key components:

​
Flow: create deployment

​
Flow: cancel deployment

​
Flow: promote

​
Flow: rollback

​
Build concurrency

Restate limits how many builds run at once. Deploy calls Build for a git source only, in the scope builds with the workspace id as the limit key, and a rule for builds/<workspace_id> or builds/* says how many such calls may run at the same time. An exact pattern beats *; no matching rule means no limit. The database is the source of truth: a rule set by hand lasts at most 15 minutes. To raise a workspace’s cap, raise builds_concurrent_max on its limits row. Deploy waits for Build (Request, not Send), so cancelling Deploy cancels the Build too. A Build still waiting is removed and never runs. A running one has its image build aborted and frees its slot at once. That last part needs Build to stay unsuspended for the whole build, which is why it is bound with WithInactivityTimeout(deploy.BuildKeepAliveWindow). Restate suspends an invocation after about a minute of inactivity, and a suspended Build cannot be told it was cancelled: it would run to completion and hold the workspace’s slot until it did. Keep that window above buildBackendDeadline, or cancel stops working for any build that outlives it.
  • One queue per workspace, production and preview together, in the order builds became ready. No priority, no timeout: a waiting deployment stays pending until a build in its workspace finishes or it is cancelled.
  • Build refuses to run outside scope builds or with a limit key other than its workspace. Restate picks the rule before it dispatches, so these checks report a mis-queued Build, they do not stop it from taking the wrong slot.
  • A build that still fails after all retries is stopped for good, and Deploy marks the deployment failed.
  • All builds share one Restate partition. When its leader changes, every build in progress restarts from its last recorded step.
Operating the rules on Restate Cloud, reading them, forcing a sync, listing waiting builds, is covered by runbook 0003 in the infra repo.

​
Commit deduplication

When a new deployment is created, Workflow.cancelOlderSiblings looks for older deployments on the same (app, environment, branch) that are still in the build queue (pending or awaiting_approval) and cancels them. Once Restate lets a deployment’s Build run and it transitions to building, it is committed: newer commits will not supersede it. This avoids the pathological case where rapid pushes keep cancelling builds and nothing ever finishes. Cancellation happens in three steps, all batched:
  1. One SELECT — list older queued sibling deployments with their invocation IDs.
  2. One batch UPDATE — stamp every sibling’s in-flight steps with "Superseded by newer commit" (first-write-wins via WHERE ended_at IS NULL).
  3. One batch UPDATE — transition every sibling to status=superseded.
  4. N HTTP calls — restateAdmin.CancelInvocation for each sibling that has an invocation ID.
Only git-sourced deployments with a branch are deduplicated; Docker-image redeploys bypass this path.

​
Instance readiness (promise-based)

After createTopologies, the Deploy run enters waitForDeployments, which awaits a durable promise named instances_ready until krane reports pod readiness. Krane installs the per-deployment Cilium network policy itself when it applies each deployment, so the run does not create one. waitForDeployments flow:
  1. Build per-region min replicas from the topologies just inserted.
  2. Require numRegions - 1 healthy regions (minimum 1, tolerating one regional outage).
  3. Check the database once for instances that are already healthy.
  4. Otherwise await the promise, racing it against regionReadyTimeout.
The promise is resolved by DeployWorkflow.NotifyInstancesReady, a SHARED handler that runs concurrently with the suspended run. A report that arrives before the run awaits is kept by the promise, so there is no awakeable to stash and no state to clear. Caller: cluster.Service.ReportDeploymentStatus (the RPC krane calls to report instance state) runs a thundering-herd gate after the upsert transaction:
  1. Deployment must be in an active status (starting | building | deploying | network | finalizing).
  2. Look up per-region min replicas via FindDeploymentTopologyMinReplicas.
  3. Count running instances per region; require numRegions - 1 healthy regions (minimum 1 — tolerates one regional outage).
  4. Dedup the notification via an in-process sync.Map so we don’t re-fire on every subsequent status report once the threshold is met.
  5. Skip if the deployment’s Deploy invocation is no longer live in Restate. Wake also reports with status deploying, and a notify for a finished run would leave a promise Restate never cleans up.
  6. If threshold met (and not yet notified), send NotifyInstancesReady(deployment_id) to DeployWorkflow.

​
Self-skip (belt-and-suspenders dedup)

In addition to the proactive cancel above, the Deploy handler checks HasNewerActiveDeployment at the top of its workflow. If a newer sibling on the same (app, env, branch) is already pending, starting, building, deploying, network, finalizing, ready, or awaiting_approval, the current deployment self-skips. This catches races where the proactive cancel didn’t land (e.g. the newer deployment hadn’t persisted its invocation ID yet).

​
State serialization (desired state)

Scheduled state changes are serialized via a Restate virtual object keyed by deployment ID in svc/ctrl/worker/deployment. The object stores a nonce for the most recent transition so older delayed requests no-op.

​
Retry policy

DeployWorkflow is registered with an exponential-backoff retry policy: 2s → 4s → 8s → 16s → 30s (capped), 15 attempts total (about 5 minutes). Short intervals keep cancel latency low, since a cancel lands at the next attempt boundary. On exhaustion the invocation pauses rather than being killed, so the compensation stack can still run when a restate.Run returns a terminal error. This replaces an older 150-attempt policy that could leave a deploy stuck retrying for about 24 hours. Build uses the same intervals and attempt count but is stopped for good on exhaustion, as described under Build concurrency.

​
Compensation stack

The Deploy handler maintains a LIFO compensation stack registered via Compensation.Add, each step wrapped in restate.RunVoid. The stack fires on any error or cancellation:
  • Mark the deployment as failed (only if still in an active status — the conditional UpdateDeploymentStatusIfActive query prevents overwriting superseded or ready)
  • Undo topology inserts, route assignments, etc.