Pull-Based Provisioning

Unkey's infrastructure orchestration implements a pull-based model where autonomous Krane instances poll the control plane for state changes and continuously reconcile desired state with actual state. This architecture follows the Kubernetes List+Watch pattern, using a globally unique, monotonically increasing version number embedded directly in each resource row to track changes and enable efficient incremental synchronization.

The architecture's core principle is to enable autonomous reconciliation and self-healing without requiring the control plane to track connected clients or push events.

Architecture

VersioningService

The VersioningService is a Restate virtual object that generates globally unique, monotonically increasing version numbers. It is a singleton (keyed by empty string) that maintains a durable counter in Restate's state store.

Before any mutation to a deployment topology or sentinel, the control plane calls NextVersion() to obtain the next version. This version is then stored directly on the resource row. The virtual object guarantees:

• Monotonically increasing values: Versions always increase, with no gaps under normal operation
• Exactly-once semantics: Retries of the same Restate invocation return the same version
• Single-writer: The singleton virtual object serializes all version generation

This eliminates the need for a separate changelog table—the version embedded in each resource row is the source of truth for synchronization.

Watch Protocol

The synchronization protocol uses WatchDeployments and WatchSentinels RPCs that handle both initial bootstrap and incremental updates. This design eliminates the complexity of managing separate "synthetic" and "live" modes, and removes the need for the control plane to track connected clients in memory.

Version-Based Tracking

Every resource (deployment topology, sentinel) carries a version column set at insert/update time via the VersioningService. Krane tracks its last-seen version (versionLastSeen) and requests only resources with version > versionLastSeen. The version is globally unique across all resource types, enabling a single watermark for all synchronization.

Bootstrap Phase

When Krane connects with version=0 (fresh start or after data loss), the control plane streams the complete desired state for the region. Resources are ordered by version. Krane tracks the maximum version seen across all received resources. Stream close without error signals bootstrap completion—only then does Krane commit its version watermark.

After receiving the full bootstrap, Krane garbage-collects any Kubernetes resources that were not mentioned in the stream. This ensures consistency even if the control plane's view of desired state has changed significantly.

Incremental Sync

After bootstrap, Krane periodically reconnects with its last-seen version. The control plane queries resource tables directly:

SELECT * FROM deployment_topology 
WHERE region = ? AND version > ? 
ORDER BY version LIMIT 100;
 
SELECT * FROM sentinels 
WHERE region = ? AND version > ? 
ORDER BY version LIMIT 100;

Changes are streamed to Krane, which processes each and tracks the maximum version seen.

Soft Deletes

Resources are never hard-deleted during sync. Instead, "deletes" are represented as state changes: setting desired_replicas=0 or desired_status=stopped. Krane sees this change (the row still exists with a new version) and deletes the corresponding Kubernetes resource.

Reconnection

If the connection drops, Krane reconnects with its last-seen version. Since versions are embedded in resource rows and not in a separate changelog with retention, there's no risk of the version being "too old"—the query simply returns any resources with versions newer than Krane's watermark.

Krane Agent

Krane agents act as autonomous controllers that reconcile desired state with actual Kubernetes resources in their respective regions. Each Kubernetes cluster runs a single Krane agent.

Each controller maintains its watch stream, reconnecting with jittered backoff (1-5 seconds) on failure. Controllers process received state messages and track versionLastSeen, updating it only after successfully processing state changes and receiving a clean stream close.

Status updates flow back to the control plane through unary RPCs (ReportDeploymentStatus, ReportSentinelStatus) with buffering, retries, and circuit breakers for reliability.

Deployment Workflow

The deploy workflow obtains a version from the VersioningService before writing each topology row. It does not push events directly to Krane. The workflow then polls the instances table waiting for Krane to report that pods are running, with a timeout for failure handling.

Why Polling Over Push

We chose polling-based synchronization over push-based event streaming for several reasons.

The control plane becomes stateless with respect to connected clients. It doesn't need to track which Krane instances are connected, buffer events during disconnections, or handle the complexity of fan-out to multiple subscribers. This simplifies horizontal scaling and eliminates a class of bugs around connection state management.

Polling naturally handles backpressure. If Krane falls behind processing, it simply polls less frequently. With push-based streaming, the control plane would need to implement flow control or risk overwhelming slow clients.

The version-based approach provides exactly-once delivery semantics. Each resource has a unique version, and Krane's watermark ensures no changes are missed or processed twice, even across restarts.

The tradeoff is latency. With 1-5 second polling intervals, there's a delay between a state change and Krane receiving it. For our use case (infrastructure provisioning measured in seconds to minutes), this latency is acceptable.

Database Schema

The deployment_topology table defines desired state for multi-region deployments. Each row specifies the desired replica count for a deployment in a specific region. The version column (unique, indexed with region) is set via VersioningService on insert/update.

The sentinels table combines desired and actual state with an embedded version. Desired fields (cpu, memory, replicas, desired_state) are set by the control plane with a new version; actual fields (available_replicas, health) are updated by Krane without version changes.

The instances table tracks actual state reported by Krane. We only write to it in response to Kubernetes events. The workflow polls this table to determine when deployments are ready.

Both deployment_topology and sentinels have a composite index on (region, version) for efficient sync queries.