# Resource permission catalog
Source: https://engineering.unkey.com/architecture/authorization/resource-permission-catalog
Canonical resource hierarchy, paths, and actions
The resource permission catalog lists every canonical resource path and its
supported actions. GitHub apps are workspace-scoped. Every other catalog
resource is project-scoped. Portals remain outside the canonical catalog.
Read [Unkey Resource Names](/architecture/resources/unkey-resource-names) for
URN format and pattern rules. Read
[Resource permissions](/architecture/authorization/resource-permissions) for
permission matching and action rules.
## Canonical catalog
Each resource shows its actions in brackets. The next line shows its canonical
resource path. Add `unkey:v1:{workspace_id}:` before the path and `#{action}`
after it to form a permission.
```plaintext theme={"theme":"kanagawa-wave"}
Workspace
│
├── GitHub app [read, write, delete]
│ github/apps/{github_app_id}
│
└── Project [read, write, delete]
projects/{project_id}
│
├── App [read, write, delete]
│ projects/{project_id}/apps/{app_id}
│ │
│ └── Environment [read, write, delete]
│ projects/{project_id}/apps/{app_id}/environments/{environment_id}
│ │
│ ├── Deployment [read, write, delete]
│ │ projects/{project_id}/apps/{app_id}/environments/{environment_id}/deployments/{deployment_id}
│ │ │
│ │ └── Logs [read]
│ │ projects/{project_id}/apps/{app_id}/environments/{environment_id}/deployments/{deployment_id}/logs
│ │
│ ├── Domain [read, write, delete]
│ │ projects/{project_id}/apps/{app_id}/environments/{environment_id}/domains/{domain_id}
│ │
│ ├── Environment variable [read, write, delete]
│ │ projects/{project_id}/apps/{app_id}/environments/{environment_id}/variables/{variable_id}
│ │
│ └── Gateway
│ projects/{project_id}/apps/{app_id}/environments/{environment_id}/gateway
│ │
│ ├── Logs [read]
│ │ projects/{project_id}/apps/{app_id}/environments/{environment_id}/gateway/logs
│ │
│ └── Policy [read, write, delete]
│ projects/{project_id}/apps/{app_id}/environments/{environment_id}/gateway/policies/{policy_id}
│
├── Identity [read, write, delete]
│ projects/{project_id}/identities/{identity_id}
│
├── Keyspace [read, write, delete]
│ projects/{project_id}/keyspaces/{keyspace_id}
│ │
│ ├── Logs [read]
│ │ projects/{project_id}/keyspaces/{keyspace_id}/logs
│ │
│ └── Key [read, write, delete, decrypt, verify]
│ projects/{project_id}/keyspaces/{keyspace_id}/keys/{key_id}
│
├── Rate limit namespace [read, write, delete, limit]
│ projects/{project_id}/ratelimits/namespaces/{namespace_id}
│ │
│ ├── Logs [read]
│ │ projects/{project_id}/ratelimits/namespaces/{namespace_id}/logs
│ │
│ └── Override [read, write, delete]
│ projects/{project_id}/ratelimits/namespaces/{namespace_id}/overrides/{override_id}
│
└── RBAC
projects/{project_id}/rbac
│
├── Role [read, write, delete]
│ projects/{project_id}/rbac/roles/{role_id}
│
└── Permission [read, write, delete]
projects/{project_id}/rbac/permissions/{permission_id}
```
`gateway` and `rbac` are path containers. They organize child paths. They are
not concrete resources and cannot be permission targets.
## Examples
These examples show how concrete IDs and wildcards use the same catalog.
### Create a key
Use `write` with a wildcard key ID because the key does not exist yet.
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:projects/proj_123/keyspaces/ks_123/keys/*#write
```
### Promote or roll back a deployment
Promoting or rolling back a deployment changes which deployment serves the
environment. Both actions require `write` on the environment. A deployment
permission does not grant either action.
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:projects/proj_123/apps/app_123/environments/env_123#write
```
### Start or stop a deployment
Deployment `write` covers creation, updates, starts, and stops.
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:projects/proj_123/apps/app_123/environments/env_123/deployments/dep_123#write
```
### Read deployment logs
Logs are first-class resources. Deployment `read` does not grant access to
deployment logs. Grant `read` on the log path.
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:projects/proj_123/apps/app_123/environments/env_123/deployments/dep_123/logs#read
```
### Read gateway logs
Gateway logs belong to an environment's gateway.
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:projects/proj_123/apps/app_123/environments/env_123/gateway/logs#read
```
### Verify a key
Key verification uses `verify`, not `read`.
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:projects/proj_123/keyspaces/ks_123/keys/key_123#verify
```
## Related pages
Use these pages for URN rules, permission rules, and WorkOS role expansion.
* [Unkey Resource Names](/architecture/resources/unkey-resource-names)
* [Resource permissions](/architecture/authorization/resource-permissions)
* [WorkOS roles](/architecture/authorization/workos-permissions)
# Resource permissions
Source: https://engineering.unkey.com/architecture/authorization/resource-permissions
Permission format, matching, and action rules
Resource permissions combine a URN pattern with an action. The resource
path identifies the resource type and scope. The action identifies the
operation.
Read [Unkey Resource Names](/architecture/resources/unkey-resource-names) for
URN format and pattern rules. Use the
[resource permission catalog](/architecture/authorization/resource-permission-catalog)
to find each canonical resource path and its supported actions.
## Permission format
A permission has two parts separated by `#`.
```plaintext theme={"theme":"kanagawa-wave"}
{resource_urn_pattern}#{action}
```
This permission grants read access to one key:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:projects/proj_123/keyspaces/ks_123/keys/key_456#read
```
Standard grants use an exact canonical resource path. Use `*` only in ID
segments when a grant covers multiple resource instances:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:projects/proj_123/keyspaces/*/keys/*#read
```
The path makes this a key permission. The same `read` action on an environment
path grants environment access instead.
## Permission matching
A permission grants access when both of these conditions are true:
1. The permission resource pattern matches the request resource URN.
2. The permission action matches the request action.
Match resource patterns with the URN rules. Match actions with exact text. The
only action wildcard is the global admin permission
`unkey:v1:{workspace_id}:**#*`.
```plaintext theme={"theme":"kanagawa-wave"}
Permission: unkey:v1:ws_123:projects/proj_123/keyspaces/*/keys/*#read
Request: unkey:v1:ws_123:projects/proj_123/keyspaces/ks_123/keys/key_456#read
Result: granted
```
An exact app path does not grant access to its descendants:
```plaintext theme={"theme":"kanagawa-wave"}
Permission: unkey:v1:ws_123:projects/proj_123/apps/app_123#write
Request: unkey:v1:ws_123:projects/proj_123/apps/app_123/environments/env_123#write
Result: denied
```
Use a trailing `/**` only for an intentional subtree grant. It covers the base
resource and every current or future descendant for the same action:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:projects/proj_123/apps/app_123/**#write
```
This permission authorizes `write` on the app, its environments, deployments,
and other descendants.
A `/**` grant also covers descendant resource types added to the catalog
later. Use an exact canonical path unless future descendants belong in the
grant.
## Action rules
Canonical actions use generic operation names. The resource path identifies the
target resource type.
| Action | Grants access to |
| --------- | -------------------------------------------- |
| `read` | Get or list the resource |
| `write` | Create or update the resource |
| `delete` | Delete the resource |
| `decrypt` | Decrypt key data |
| `verify` | Verify a key |
| `limit` | Check or use a rate limit |
| `*` | Perform any action as a global administrator |
`decrypt`, `verify`, and `limit` are special actions. Use them only on catalog
resources that support them.
## Create and update rules
Use `write` for create and update operations. Do not define separate create or
update actions.
Use the concrete resource ID when the resource exists:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:projects/proj_123/keyspaces/ks_123/keys/key_456#write
```
Use `*` in the target resource ID position when the resource does not exist:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:projects/proj_123/keyspaces/ks_123/keys/*#write
```
The wildcard applies only to that path segment. It does not grant access to
other keyspaces or projects.
## Resource-owned operations
Check an operation against the resource that owns the resulting state change.
Do not add an action for each API endpoint.
| Operation | Resource | Action |
| ---------------------------------------------- | ----------- | ------- |
| Connect or disconnect a GitHub repository | App | `write` |
| Promote or roll back a deployment | Environment | `write` |
| Start or stop a deployment | Deployment | `write` |
| Restart domain verification | Domain | `write` |
| Reroll a key or update its credits | Key | `write` |
| Assign or remove a role or permission on a key | Key | `write` |
| Assign or remove a permission on a role | Role | `write` |
| Create or edit a permission definition | Permission | `write` |
## Log rules
Logs are first-class resources. Each log resource supports `read`. Its path
identifies the log source.
Read access to a parent resource does not grant read access to its logs. Grant
the exact log resource path or use an intentional subtree grant.
Find all log resource paths in the
[resource permission catalog](/architecture/authorization/resource-permission-catalog).
## Global admin
This permission grants every action on every resource in a workspace:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:{workspace_id}:**#*
```
Use this permission only for workspace administrators.
## Related pages
Use these pages for URN rules, the resource catalog, and WorkOS role expansion.
* [Unkey Resource Names](/architecture/resources/unkey-resource-names)
* [Resource permission catalog](/architecture/authorization/resource-permission-catalog)
* [WorkOS roles](/architecture/authorization/workos-permissions)
# WorkOS permissions
Source: https://engineering.unkey.com/architecture/authorization/workos-permissions
How WorkOS permission strings map to Unkey resource permissions
WorkOS is authoritative for user roles and permission strings. Unkey uses WorkOS
for authentication and role assignment, then translates the WorkOS permission
strings from the access token into canonical Unkey resource permissions before
the API constructs the principal.
This translation exists because WorkOS permission strings cannot represent
Unkey resource permissions directly. WorkOS permission slugs are capped at 48
characters and cannot contain `/`, so they cannot encode canonical Unkey
resource paths. The WorkOS permission model is intentionally much smaller than
Unkey's resource permission model.
The exact list of supported WorkOS permissions and their Unkey translations is
defined in
[`pkg/auth/workos/permissions.go`](https://github.com/unkeyed/unkey/blob/main/pkg/auth/workos/permissions.go).
That Go file is the source of truth.
## Permission shape
WorkOS recommends clear, concise permission slugs with a resource and action
delimiter. Unkey follows that shape and keeps WorkOS permissions broad:
```plaintext theme={"theme":"kanagawa-wave"}
{area}:{action}
```
For example:
```plaintext theme={"theme":"kanagawa-wave"}
keys:create
```
The WorkOS slug is intentionally not a Unkey resource permission. It is stable
input from the identity provider. The API translates it after token verification
because only Unkey owns the resource-name contract.
## Translation
The WorkOS resolver wraps the generic JWT JWKS resolver. The generic resolver
verifies issuer, audience, signature, and time claims, and builds the initial
JWT principal. The WorkOS wrapper then replaces the raw WorkOS permission
strings with canonical permissions in this format:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:{workspace_id}:{resource_path}#{action}
```
A WorkOS permission string such as:
```plaintext theme={"theme":"kanagawa-wave"}
deployments:create
```
becomes a resource permission for the principal's workspace:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:{workspace_id}:projects/**#create_deployment
```
The translated permission is the only value handlers see. Handler authorization
does not depend on WorkOS permission names.
## Unknown permissions
Unknown WorkOS permission strings are ignored during translation. They don't
grant access, and they don't fall back to legacy wildcard permissions.
This makes WorkOS safe to contain permissions that are not yet understood by
the API. Adding a new permission requires updating the mapping table in
`pkg/auth/workos/permissions.go` and adding tests for the resulting Unkey
resource permission.
## Ownership boundaries
The generic JWT package must stay provider-neutral. It verifies JWTs and
returns claims as-is.
WorkOS-specific behavior belongs in `pkg/auth/workos`:
* the WorkOS issuer constant,
* the WorkOS JWKS resolver wrapper,
* the WorkOS permission mapping table, and
* tests for WorkOS permission translation.
API configuration uses the WorkOS wrapper only for JWKS-backed WorkOS access
tokens. HMAC JWT auth continues to use the generic JWT resolver directly.
# Architecture
Source: https://engineering.unkey.com/architecture/index
System architecture and design references
This section documents system architecture, service workflows, and RFCs. Content is maintained separately from service-level runbooks and configuration guides.
## Service map
```mermaid theme={"theme":"kanagawa-wave"}
flowchart LR
Frontline[Frontline]
API[API]
Vault[Vault]
ControlAPI[Control plane API]
ControlWorker[Control plane worker]
Krane[Krane]
Preflight[Preflight]
Frontline -->|A| API
API -->|B| Vault
Frontline -->|B| Vault
ControlAPI -->|C| ControlWorker
ControlWorker -->|C| Krane
ControlWorker -->|B| Vault
Preflight -->|A| Krane
```
### Legend
* **A**: HTTP or HTTPS request.
* **B**: RPC or service call.
* **C**: Asynchronous workflow trigger.
## Sections
* [Services](/architecture/services)
* [RFCs](/architecture/rfcs)
# Consistency model
Source: https://engineering.unkey.com/architecture/ratelimiting/consistency-model
How rate limit state converges across processes and regions
Rate limiting is intentionally not a single synchronous global counter. The system converges in layers so the request path stays fast even when shared dependencies are slow or unavailable.
## Convergence layers
Each layer handles a different scope.
| Layer | Scope | Purpose |
| --------------- | ----------- | -------------------------------------------------------- |
| Local memory | One process | Make the immediate decision without a network round trip |
| Regional origin | One region | Converge multiple processes serving the same region |
| Global counters | All regions | Share meaningful regional usage with other regions |
The lower layer always remains useful when a higher layer lags. A process can continue making local decisions if regional convergence is delayed. A region can continue enforcing regional limits if global convergence is delayed.
## Regional convergence
After a request is accepted, the process buffers a replay event for the regional origin. Replay merges the regional value back into local memory with `max` and marks the entry fresh. Processes in the same region converge toward the same count without waiting on every request.
```mermaid theme={"theme":"kanagawa-wave"}
sequenceDiagram
participant Request as Request path
participant Local as Local counter
participant Replay as Replay buffer
participant Origin as Regional origin
Request->>Local: Commit accepted increment
Request->>Replay: Buffer replay event
Replay->>Origin: Increment regional count
Origin-->>Replay: Regional total
Replay->>Local: Merge max(local, regional total)
```
Cold counters, stale counters, and strict-mode counters read the regional origin synchronously before deciding. This makes the first decision for a key, decisions after stale local state, and decisions after a denial use a fresher regional baseline.
Warm entries carry a freshness deadline. Regional-origin reads and successful replays extend that deadline. While the entry is fresh, the request path can use local memory without blocking on origin. After the deadline passes, the next request refreshes from origin before deciding.
The freshness interval is intentionally short. Active identifiers normally stay fresh through replay, while idle or lagging identifiers re-read origin before they can keep serving an old local view for the rest of a long window. Concurrent stale requests for the same window cell share one origin read, then continue from the same refreshed value.
## Strict mode
Strict mode is regional. When a request is denied, the service records a deadline for the `(workspace, namespace, identifier, duration)` tuple. Until that deadline passes, later requests for the same tuple refresh the current window from the regional origin before evaluating the limit.
The strict-mode key excludes the sequence. A denial in one fixed window can still affect the weighted previous-window term in the next fixed window, so strict mode survives the sequence rollover.
Strict mode does not publish cross-region state. It refreshes the current window only. Previous windows use the normal cold and stale refresh path because they no longer receive new accepted increments.
## Global convergence
Global convergence is eventual. A region publishes its own regional count when the count becomes meaningful for remote decisions. Other regions import the sum of foreign regional counts and include that imported count in future decisions. The publishing region may also import its own published count as a lower bound for local regional state on nodes that have not yet seen the same regional origin value.
```plaintext theme={"theme":"kanagawa-wave"}
Region A accepts traffic
│
▼
Region A converges its local nodes
│
▼
Region A publishes its regional count
│
▼
Region B imports Region A's count
│
▼
Region B includes that count in later decisions
```
This model means simultaneous traffic in multiple regions can briefly pass before every region has imported the latest remote usage. The tradeoff is deliberate: request-serving processes do not wait for cross-region coordination on the hot path.
Own-region imports are a safety net, not a replacement for the regional origin. They can only raise local regional counts. They do not refresh the local entry's regional-origin freshness deadline, and foreign counts still stay separate so they cannot be published again.
Windows shorter than 60 seconds are effectively regional because the global convergence cadence is too coarse to provide useful cross-region accuracy. Longer windows can include global convergence before the window expires.
## Failure behavior
Failures degrade toward local decisions and recover when the affected layer becomes available again.
| Failure | Behavior |
| -------------------- | ----------------------------------------------------------------------------------------------------------- |
| Regional read fails | The process continues from its local count and retries soon; failed reads do not make the local entry fresh |
| Regional replay lags | Other nodes in the region converge later, or refresh when their local entry becomes stale |
| Global publish lags | Other regions do not see the new count yet |
| Global import lags | The region continues with its existing imported counts |
Correctness does not depend on making every layer synchronous. The invariant is that accepted local work is monotonic within a window cell, so delayed convergence can merge later without subtracting or rewriting history.
# Global counters
Source: https://engineering.unkey.com/architecture/ratelimiting/global-counters
How G-Counters power cross-region rate limit convergence
Global counters let regions share rate limit usage without turning every request into a synchronous global write. The model is a grow-only counter, or G-Counter, with one component per region and window cell.
## G-Counter model
A G-Counter is a conflict-free replicated data type for counts that only increase. It splits one logical count into independently owned components. The global value is the sum of all components.
For rate limiting, each region owns one component for each window cell. The component identity is:
```plaintext theme={"theme":"kanagawa-wave"}
(workspace, namespace, identifier, duration, sequence, region)
```
Each region writes only its own component. Region A never writes Region B's component. A component only moves forward inside a sequence.
```mermaid theme={"theme":"kanagawa-wave"}
flowchart TB
subgraph regionA[Region A]
aLocal[Regional count: 8]
aComponent[Component A: max(old, 8)]
aLocal --> aComponent
end
subgraph regionB[Region B]
bLocal[Regional count: 5]
bComponent[Component B: max(old, 5)]
bLocal --> bComponent
end
sum[Global value: A + B]
aComponent --> sum
bComponent --> sum
```
The merge rule is simple:
```plaintext theme={"theme":"kanagawa-wave"}
component = max(component, observedRegionalCount)
global = sum(all components)
```
That gives the counter four useful properties.
| Property | Effect |
| ----------- | --------------------------------------------------------- |
| Monotonic | An older delayed publish cannot move a component backward |
| Idempotent | Publishing or importing the same value twice is harmless |
| Commutative | Regions can publish and import in different orders |
| Associative | Components can be aggregated before import |
## Importing without feedback loops
A region imports components in two different ways. Its own component is a regional lower bound and can raise the local regional count. Foreign components are summed separately and added to decisions, but they are never published again.
```plaintext theme={"theme":"kanagawa-wave"}
Region A decision = A regional count + imported count from B and C
Region A publish = A regional count only
Region A own-row import = max(A regional count, A published component)
```
This separation prevents double counting. If Region A published `A + B`, then Region B could import that value and count its own traffic again. If Region A folded its own component into imported global count instead of regional count, it would count its own traffic twice during local decisions.
Own-row import is only a safety net. The regional origin remains the source of truth for in-region convergence, and importing an own component does not make the local entry fresh against that origin.
## Why it fits rate limiting
Usage inside one fixed window cell is grow-only. Requests add cost. The cell does not need to decrement while it is active.
Sliding-window behavior comes from reading two cells, weighting the previous one, and letting old cells expire.
```plaintext theme={"theme":"kanagawa-wave"}
current = currentRegionalCount + currentGlobalCount
previous = previousRegionalCount + previousGlobalCount
effective = current + (previous * previousWindowWeight) + cost
```
The G-Counter resets by identity. A new fixed window has a new sequence, so it has a new set of components. Old components stop mattering when they can no longer contribute to the sliding-window calculation.
## Publish threshold
The `limit` is not part of the G-Counter merge rule. It only decides when a regional count is worth publishing.
Small counts are usually irrelevant to remote decisions, so they stay regional. Once a region uses a meaningful fraction of the limit, the count is published and can affect later decisions elsewhere.
This keeps cross-region writes proportional to useful signal rather than total request volume. The request path still makes local decisions immediately, and global convergence catches up for identifiers that are approaching their limit.
## Invariants
Global counters rely on these invariants:
* Each region owns only its own component.
* Component values are merged with `max`, never replacement by an older value.
* A region's own imported component can only raise regional count.
* Foreign imported count is read by the request path but never republished.
* Own components and foreign components stay separate during import.
* A sequence is immutable once it ages out. New windows use new component identities.
# Rate limiting
Source: https://engineering.unkey.com/architecture/ratelimiting/overview
Architecture overview for Unkey's distributed rate limiter
Rate limiting is a shared runtime subsystem used by API and Frontline policy execution. The implementation lives in [`internal/services/ratelimit`](https://github.com/unkeyed/unkey/tree/main/internal/services/ratelimit).
The system is designed to protect request-serving paths first. Accuracy improves as state converges, but no normal request depends on a synchronous global counter.
## Design goals
The rate limiter optimizes for these goals, in this order:
1. Keep the hot path local. A process must be able to make a decision from memory without waiting on cross-region state.
2. Preserve availability. Shared dependencies can improve accuracy, but a dependency failure must degrade toward local enforcement instead of taking down requests.
3. Converge where it matters. Nodes in one region converge quickly. Regions share counts when usage is high enough to affect remote decisions.
4. Avoid double counting. A region's own count and imported foreign counts stay separate so remote usage is never republished as local usage.
5. Smooth reset boundaries. Fixed window cells are evaluated as a sliding window so callers cannot spend a full limit on both sides of a boundary.
## Tradeoffs
These goals create deliberate tradeoffs:
| Choice | Benefit | Cost |
| ------------------------------------------ | ------------------------------------------------------ | --------------------------------------------------------------------------------- |
| Local-memory decision first | Low latency and high availability | Simultaneous traffic in different processes can briefly see different views |
| Async regional convergence | Requests don't wait on the regional origin | A neighboring node may lag until replay, stale refresh, or strict mode catches up |
| Async global convergence | No request waits on cross-region coordination | Multi-region bursts can briefly pass before imported counts arrive |
| Publish only meaningful regional usage | Cross-region writes stay proportional to useful signal | Low regional usage may remain regional only |
| Sliding-window evaluation over fixed cells | Smooths boundary bursts without per-request histories | Requires reading the current and previous window cells |
The intended shape is shared-nothing on the hot path, with shared systems used as convergence accelerators. Regional and cross-region state improve accuracy, but they are not critical dependencies for serving the request.
## Core idea
The rate limiter stores fixed window cells and evaluates them as a sliding window. Each request updates the current cell if the effective count is still under the limit.
State converges in layers:
```mermaid theme={"theme":"kanagawa-wave"}
flowchart TB
request[Request path]
subgraph process[One process]
memory[Local atomic counters]
originReads[Origin reads]
end
subgraph region[One region]
regional[Regional counter origin]
end
subgraph global[All regions]
counters[Global counter components]
end
request --> memory
memory --> regional
regional --> memory
memory --> counters
counters --> memory
originReads --> regional
```
Local counters keep the hot path fast. The regional origin converges nodes inside one region through asynchronous replay, stale-entry refreshes, and strict-mode reads after denials. Global counters converge regional observations across regions for longer windows and can also raise same-region local counts as a safety net when a node has not yet seen the latest regional-origin value.
## Pages
* [Request path](/architecture/ratelimiting/request-path) explains how one request is evaluated, including sliding-window math and batch semantics.
* [Consistency model](/architecture/ratelimiting/consistency-model) explains what converges locally, regionally, and globally.
* [Global counters](/architecture/ratelimiting/global-counters) explains the G-Counter model used for cross-region convergence.
## Invariants
These invariants shape the implementation:
* The request path must not wait on cross-region state.
* A local count represents only this region's own observations.
* Imported global count represents other regions and must not be pushed back out.
* Own-region global-counter imports may raise local count but must not mark regional origin state fresh.
* A fixed window cell is grow-only while it is active.
* Sliding-window behavior comes from weighting the previous cell, not from decrementing the current cell.
* Batch requests must preserve all-or-nothing semantics.
## Scope boundaries
This subsystem owns counting, convergence, and the rate limit decision. It does not own how callers choose identifiers, configure limits, or translate denial responses into protocol-specific errors.
API uses the subsystem for standalone rate limits, key verification limits, and workspace API throttling. Frontline uses it for policy execution. Sharing the subsystem keeps these paths on the same counter semantics instead of creating service-specific rate limit behavior.
# Request path
Source: https://engineering.unkey.com/architecture/ratelimiting/request-path
How one rate limit request is evaluated
The rate limit request path turns a `(workspace, namespace, identifier, duration)` tuple into a sliding-window decision. The normal path is local-memory first and reaches regional state when a counter is cold, stale, or in strict mode.
## Single request flow
Each request builds two counter keys: the current fixed window cell and the previous fixed window cell. The previous cell is needed because the public behavior is a sliding window.
```mermaid theme={"theme":"kanagawa-wave"}
flowchart TD
start[Ratelimit request] --> validate[Validate request fields]
validate --> keys[Build current and previous keys]
keys --> hydrate[Hydrate cold or stale counters]
hydrate --> strict{Strict mode active?}
strict -->|Yes| fetch[Refresh from regional origin]
strict -->|No| read[Read local and imported counts]
fetch --> read
read --> math[Compute sliding-window count]
math --> decision{Effective count exceeds limit?}
decision -->|Yes| deny[Deny and set strict deadline]
decision -->|No| commit[CAS increment current counter]
commit -->|Retry| read
commit -->|Success| replay[Queue regional replay]
replay --> allow[Return success]
```
The compare-and-swap loop protects the local counter from concurrent accepted requests in the same process. If another goroutine changes the current counter between the read and the commit, the request recomputes the effective count before trying again.
## Sliding-window math
The implementation stores fixed window cells, but the decision behaves like a sliding window.
For a request at time `t`, the current sequence is:
```plaintext theme={"theme":"kanagawa-wave"}
currentSequence = floor(t / duration)
previousSequence = currentSequence - 1
```
The current cell contributes its full count. The previous cell contributes only the fraction that still overlaps the sliding window.
```plaintext theme={"theme":"kanagawa-wave"}
current = currentRegionalCount + currentGlobalCount
previous = previousRegionalCount + previousGlobalCount
effective = current + (previous * previousWindowWeight) + cost
```
The weight starts near `1` at the beginning of a new fixed window and moves toward `0` as the current fixed window advances.
```plaintext theme={"theme":"kanagawa-wave"}
previous window current window
┌────────────────┐┌────────────────┐
▲
request near boundary
Most of the previous window still overlaps the sliding window.
previous window current window
┌────────────────┐┌────────────────┐
▲
request near end
Very little of the previous window still overlaps the sliding window.
```
This prevents a caller from using the full limit at the end of one fixed window and immediately using the full limit again at the start of the next one.
## Counter entries
`counterEntry` is the in-memory state for one window cell. It is intentionally small because it sits on the request path.
| State | Purpose |
| --------------------- | ------------------------------------------------------------ |
| Regional count | This region's count for the window cell |
| Speculative count | In-flight `RatelimitMany` increments that may roll back |
| Hydration state | Cold-start coordination for regional origin reads |
| Origin freshness | Last regional-origin read or replay time for stale detection |
| Stale refresh mutex | Single-flight guard for concurrent stale origin reads |
| Global count | Imported count from other regions |
| Global push threshold | Minimum regional count worth sharing globally |
| Last pushed count | Last regional count published for global convergence |
The key invariant is that regional count and global count stay separate. Regional count can be published outward. Global count is imported from other regions and must not be published again.
## Batch requests
`RatelimitMany` evaluates multiple limits with all-or-nothing semantics. The method temporarily increments each requested counter, evaluates the full batch, then either keeps every increment or rolls every increment back.
```mermaid theme={"theme":"kanagawa-wave"}
sequenceDiagram
participant Batch as RatelimitMany
participant Entry as counterEntry
Batch->>Entry: speculative += cost
Batch->>Entry: regional += cost
Batch->>Batch: Evaluate every requested limit
alt batch passes
Batch->>Entry: speculative -= cost
else batch fails
Batch->>Entry: regional -= cost
Batch->>Entry: speculative -= cost
end
```
Global publishing reads regional count minus speculative count. That prevents temporary batch state from leaking into cross-region convergence before the batch is committed.
# Unkey Resource Names
Source: https://engineering.unkey.com/architecture/resources/unkey-resource-names
Canonical resource names for public Unkey resources
Unkey Resource Names, or URNs, identify public Unkey resources in a stable,
parseable format. URNs are used anywhere Unkey needs to refer to the same
resource across product surfaces, audit logs, permission checks, traces, and
internal events.
This document defines the `v1` resource-name contract. Future versions can add
path shapes or change parsing rules, but `v1` URNs must keep the behavior
defined here. A permission attaches an action to a URN, but the URN itself names
only the resource.
The Go implementation lives in
[`pkg/urn`](https://github.com/unkeyed/unkey/tree/main/pkg/urn).
## Format
A URN has four colon-separated fields.
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:{workspace_id}:{resource_path}
```
Each field has a fixed meaning.
| Field | Description |
| ----------------- | ---------------------------------------------------- |
| `unkey` | Fixed prefix for every Unkey Resource Name. |
| `v1` | Resource-name grammar version. |
| `{workspace_id}` | Workspace that owns the resource. |
| `{resource_path}` | Canonical path to the resource inside the workspace. |
For example:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:projects/proj_123
unkey:v1:ws_123:projects/proj_123/apps/app_456
unkey:v1:ws_123:projects/proj_123/apps/app_456/environments/env_789/deployments/d_abc
```
The workspace ID is part of the URN even when the caller already has workspace
context. Audit logs, background jobs, and support tools must be able to copy a
URN and identify the owning workspace without extra state.
## Path rules
Resource paths are part of the contract. Code that creates, parses, or matches
URNs must follow these rules.
* Concrete resource URNs must use the full canonical path from the catalog.
* Collection segments are plural, for example `keyspaces`, `keys`, and
`projects`.
* ID segments use the existing public Unkey ID for that resource.
* `:` is reserved for top-level URN fields and must not appear in a resource
path.
* `#` is reserved for permissions and must not appear in a URN.
* A resource path must not start or end with `/`.
Concrete URNs identify one resource. Resource-name patterns identify a set of
resources and are part of the `v1` URN grammar. Callers that need one exact
resource, such as audit logs and authorization requests, must use concrete
URNs. Stored authorization grants can use patterns.
## Resource-name patterns
Resource-name patterns use the same four-field URN format as concrete resource
names. The difference is in the resource path.
| Operator | Meaning |
| -------- | ---------------------------------------------------- |
| `*` | Matches exactly one complete path segment. |
| `/**` | Matches the base path and every descendant below it. |
The `*` operator must be the whole path segment. A pattern such as `key_*` is
invalid because it would make prefix matching ambiguous.
The `/**` operator must be the final path segment. A pattern such as
`projects/**/deployments/*` is invalid because descendant matching has to stop
at the end of the path.
After a path uses `*` for an ID selector, descendant ID selectors must also use
`*`. The path can still name child collections, but it can't narrow back to a
specific child. This keeps wildcard paths canonical and avoids permissions that
pretend to select a child without selecting the parent that owns it.
Valid:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:projects/*/apps/*
unkey:v1:ws_123:projects/*/apps/*/environments/*/deployments/*
unkey:v1:ws_123:projects/proj_123/apps/*/environments/*
```
Invalid:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:projects/*/apps/app_123
unkey:v1:ws_123:projects/proj_123/apps/*/environments/env_123
unkey:v1:ws_123:projects/proj_123/apps/*/environments/*/deployments/dep_123
```
For example, this pattern:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:keyspaces/*/keys/*
```
matches this concrete URN:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:keyspaces/ks_123/keys/key_456
```
This descendant pattern:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:projects/proj_123/**
```
matches the project itself and every public descendant below that project.
Patterns never cross workspace boundaries, including the global workspace
pattern:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:**
```
`pkg/urn` parses concrete names and patterns. It also decides whether one URN
covers another. The permission system adds the action suffix and decides what a
covered resource authorizes. It doesn't define its own path matching.
## Resource catalog
The public catalog defines every concrete resource path that can appear in a
`v1` URN. Implementation code must reject concrete URNs that don't match one of
these path shapes. Pattern grants must still be built from these path shapes,
with `*` replacing complete ID segments or trailing `/**` covering descendants.
### Team
Team resources are rooted under `team`.
| Resource | Path |
| ---------- | ---------------------------------- |
| Membership | `team/memberships/{membership_id}` |
| Invitation | `team/invitations/{invitation_id}` |
Examples:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:team/memberships/mbr_123
unkey:v1:ws_123:team/invitations/inv_456
```
### Billing
Billing resources are rooted under `billing`. Workspace quota is a singleton
resource because quota applies to the workspace billing state.
| Resource | Path |
| ------------- | ------------------------------- |
| Billing state | `billing` |
| Invoice | `billing/invoices/{invoice_id}` |
| Quota | `billing/quotas` |
Examples:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:billing
unkey:v1:ws_123:billing/invoices/inv_123
unkey:v1:ws_123:billing/quotas
```
### Keyspaces
Key resources are rooted under the keyspace that owns the key.
| Resource | Path |
| -------- | --------------------------------------- |
| Keyspace | `keyspaces/{keyspace_id}` |
| Key | `keyspaces/{keyspace_id}/keys/{key_id}` |
Examples:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:keyspaces/ks_123
unkey:v1:ws_123:keyspaces/ks_123/keys/key_456
```
### Identities
Identity resources are rooted under `identities`.
| Resource | Path |
| -------- | -------------------------- |
| Identity | `identities/{identity_id}` |
Example:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:identities/id_123
```
### Rate limits
Standalone rate limiting resources are rooted under `ratelimits`. Overrides
belong to the namespace they modify.
| Resource | Path |
| --------- | -------------------------------------------------------------- |
| Namespace | `ratelimits/namespaces/{namespace_id}` |
| Override | `ratelimits/namespaces/{namespace_id}/overrides/{override_id}` |
Examples:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:ratelimits/namespaces/rlns_123
unkey:v1:ws_123:ratelimits/namespaces/rlns_123/overrides/rlor_456
```
### RBAC
RBAC resources are rooted under `rbac`. Relationship changes, such as adding a
role to a key, are audited against both affected resources rather than by
creating a separate join-table URN.
| Resource | Path |
| ---------- | ---------------------------------- |
| Role | `rbac/roles/{role_id}` |
| Permission | `rbac/permissions/{permission_id}` |
Examples:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:rbac/roles/role_123
unkey:v1:ws_123:rbac/permissions/perm_456
```
### Deploy
Deploy resources use the full product hierarchy. A deployment belongs to one
environment, which belongs to one app, which belongs to one project.
| Resource | Path |
| ------------------- | ----------------------------------------------------------------------------------------------------------------------- |
| Project | `projects/{project_id}` |
| App | `projects/{project_id}/apps/{app_id}` |
| Environment | `projects/{project_id}/apps/{app_id}/environments/{environment_id}` |
| Deployment | `projects/{project_id}/apps/{app_id}/environments/{environment_id}/deployments/{deployment_id}` |
| Deployment instance | `projects/{project_id}/apps/{app_id}/environments/{environment_id}/deployments/{deployment_id}/instances/{instance_id}` |
| Domain | `projects/{project_id}/apps/{app_id}/environments/{environment_id}/domains/{domain_id}` |
| Variable | `projects/{project_id}/apps/{app_id}/environments/{environment_id}/variables/{variable_id}` |
| Gateway policy | `projects/{project_id}/apps/{app_id}/environments/{environment_id}/gateway/policies/{policy_id}` |
Examples:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:projects/proj_123
unkey:v1:ws_123:projects/proj_123/apps/app_456
unkey:v1:ws_123:projects/proj_123/apps/app_456/environments/env_789
unkey:v1:ws_123:projects/proj_123/apps/app_456/environments/env_789/deployments/d_abc
```
### Portal
Portal resources are rooted under `portals`. Session tokens and sessions belong
to the portal that created them.
| Resource | Path |
| -------------------- | -------------------------------------------------------------- |
| Portal | `portals/{portal_id}` |
| Portal session token | `portals/{portal_id}/session_tokens/{portal_session_token_id}` |
| Portal session | `portals/{portal_id}/sessions/{portal_session_id}` |
| Portal branding | `portals/{portal_id}/branding` |
Examples:
```plaintext theme={"theme":"kanagawa-wave"}
unkey:v1:ws_123:portals/portal_123
unkey:v1:ws_123:portals/portal_123/sessions/ps_456
```
## Internal resources
Runtime and implementation resources are not part of the public `v1` catalog
unless a product feature explicitly promotes them.
This includes:
* Frontline routes
* Regional counters
* ClickHouse outbox rows
* Cache entries
* Join-table rows
Internal systems can still log implementation IDs in metadata. They must not
mint public URNs for these resources unless the catalog is updated first.
## Invalid examples
These strings are invalid URNs or invalid concrete URNs.
| Value | Reason |
| ------------------------------------------------ | ------------------------------------------------- |
| `urn:unkey:v1:ws_123:keyspaces/ks_123` | Uses the wrong prefix. |
| `unkey:v1:ws_123` | Missing the resource path. |
| `unkey:v1:ws_123:keyspaces/ks_123#read_keyspace` | Contains a permission action. |
| `unkey:v1:ws_123:keyspaces/ks_*` | Uses `*` inside a path segment. |
| `unkey:v1:ws_123:projects/**/deployments/*` | Uses `**` before the end of the path. |
| `unkey:v1:ws_123:projects/*/apps/app_123` | Selects a specific child under a wildcard parent. |
| `unkey:v1:ws_123:keyspace/ks_123` | Uses an unknown path shape. |
## Related pages
* [Resource permissions](/architecture/authorization/resource-permissions)
defines how actions attach to URNs for authorization.
# 0000 Template
Source: https://engineering.unkey.com/architecture/rfcs/0000-template
You may copy this as a starting point, but it's not required
## Summary
One paragraph explanation of the feature.
## Motivation
Why are we doing this? What use cases does it support? What is the expected outcome?
## Detailed design
This is the bulk of the RFC. Explain the design in enough detail for somebody familiar with the network to understand, and for somebody familiar with the code practices to implement. This should get into specifics and corner-cases, and include examples of how the feature is used.
## Drawbacks
I Why should we not do this?
## Alternatives
What other designs have been considered? What is the impact of not doing this?
## Unresolved questions
What parts of the design are still to be done?
# 0001 RBAC
Source: https://engineering.unkey.com/architecture/rfcs/0001-rbac
To reduce the scope and time to implementation, we will be reducing our initial permission model to RBAC instead of ReBAC. This has fewer moving parts and can be implemented with just 1 new table.
There are many ways to store this data. Initially I had a 2 table setup, one for roles and one for an M:N relation between roles and keys, but there are some operational issues with that, mainly around planetscale’s foreignkeys and the lack of an easy “upsert” method. This requires us to query all roles of a workspace, then figuring out which ones are missing and creating them. For every key creation… That’s not amazing.
Here’s a much simpler proposal using a single table:
### Table Schema
* `id`
unique id for this row
* `workspaceId`
Every role is scoped to a workspace, no role sharing between tenants
* `keyId`
The key holding this role
* `role: string`
the actual name of a role, ie: `finance` (or more elaborate, see below)
This is completely up to the user, the only limitation is a length ≤ 512chars
for our own roles, we'll likely do some schema for roles, like `api::\{id\}::create_key`
This single table design is the simplest form of doing roles. By adding indices for these queries, it should scale far enough:
* roles by key
* roles by workspace
* keys by role
### Unkey internal role schema
`*` denotes either an id or a wildcard
```tsx theme={"theme":"kanagawa-wave"}
root_key::*::read_root_key
root_key::*::create_root_key // a root key MUST NOT be allowed to create another key with more permissions than itself
root_key::*::delete_root_key
root_key::*::update_root_key
api::*::create_api
api::*::delete_api // either wildcard or a specific id -> api::api_123::delete_api
api::*::read_api
api::*::update_api
api::*::read_key
api::*::create_key
api::*::update_key
api::*::delete_key
```
Some of these internal roles (`api::*::create_api`) seem overly complicated, because the wildcard will always be present, since it’s impossible to write this role in advance without knowing the api\_id that will get generated later, but by sticking with this schema, we stay consistent and can build our types and tooling more easily.
We could go deeper like `api::*::keys::*::read_key` but I’m not convinced anyone needs this and it just adds complexity for now. It’s trivial to add more roles later, let’s wait it out.
### Examples
1. A key should be allowed to create new apis, modify them and be able to perform all actions on keys.
```tsx theme={"theme":"kanagawa-wave"}
api::*::create_api
api::*::update_api
api::*::read_key
api::*::create_key
api::*::update_key
api::*::delete_key
```
2. Update access to one api and its keys, read access to all apis and their keys
```tsx theme={"theme":"kanagawa-wave"}
api::api_123::update
api::api_123::update_key
api::*::read_api
api::*::read_key
```
# 0002 Secret Scanning
Source: https://engineering.unkey.com/architecture/rfcs/0002-github-secret-scanning
## Changelog
* Unkey now offers automated alerts when root keys are checked in to Github.
* Unkey has partnered with Github to offer secret scanning for root keys.
* Root keys all take the shape `unkey_xxx`. Whenever you check in code to a public Github repository, Github will scan for this pattern and notify us.
* Unkey will automatically invalidate the key, notify you that this has taken place, and ask you to generate a new key to replace it.
* Currently, this is only offered for root keys – not for API keys that you generate in Unkey for your users. Stay tuned as we look to expand our partnership with Github in future.
## Context
Github offer a secret scanning program. Whenever code is checked in to a public repository, Github will scan the code for secrets that match a list of patterns published by official Github partners. If they detect that a secret matches that pattern, they notify the partner. The provider validates the string and decides if they should revoke the secret, issue a new secret, or contact the customer that published the secret.
{
if (!payload) {
throw new Error('Invalid payload');
}
if (!signature) {
throw new Error('Invalid signature');
}
if (!keyID) {
throw new Error('Invalid keyID');
}
const response = await fetch(GITHUB_KEYS_URI);
const data = await response.json();
const keys = data.public_keys;
if (!(keys instanceof Array) || keys.length === 0) {
throw new Error('No public keys found');
}
const publicKey = keys.find((k: { key_identifier: string }) => k.key_identifier === keyID) ?? null;
if (!publicKey) {
throw new Error('No public key found matching key identifier');
}
const verifier = crypto.createVerify('SHA256').update(payload);
if (!verifier.verify(publicKey.key, Buffer.from(signature, 'base64'))) {
throw new Error('Signature does not match payload');
}
}
```
Sample POST request from Github:
```tsx theme={"theme":"kanagawa-wave"}
POST / HTTP/2
Host: HOST
Accept: */*
Content-Length: 104
Content-Type: application/json
Github-Public-Key-Identifier: bcb53661c06b4728e59d897fb6165d5c9cda0fd9cdf9d09ead458168deb7518c
Github-Public-Key-Signature: MEQCIQDaMKqrGnE27S0kgMrEK0eYBmyG0LeZismAEz/BgZyt7AIfXt9fErtRS4XaeSt/AO1RtBY66YcAdjxji410VQV4xg==
[{"source":"commit","token":"some_token","type":"some_type","url":"https://example.com/base-repo-url/"}]
```
### Implement key revocation and user notification in the service
```tsx theme={"theme":"kanagawa-wave"}
// /api/v1/github_secrets.ts
export async function handler(request: Request) {
try {
const payload = await extractPayload(request);
log("Github secret scanning webhook", payload);
await verifySignature(payload);
const hash = await hash(payload.token);
const user = await getUserByKeyHash(hash);
const team = await getTeam(user);
await deleteKey(hash);
await notifyTeam(team);
log("User automatic key revocation event", user);
} catch (error) {
log("Github webhook verification error", error)
}
}
```
## Questions
* Should this run in a [trigger.dev](http://trigger.dev) background job? Github say that it needs to be able to “handle requests with a large number of matches without timing out”
# 0003 Key shape
Source: https://engineering.unkey.com/architecture/rfcs/0003-key-shape
[https://unkey.slack.com/archives/C05SYU6P2DC/p1705420411688469](https://unkey.slack.com/archives/C05SYU6P2DC/p1705420411688469)
This is the current shape of our keys:`\{prefix?\}_\{version\}\{length\}\{base58_randomness\}`
I chose this for backwards compatibility if we ever wanted to encode something into the key, for example a semi-public identifier, think `client_id` and `client_secret` in OAuth or other implementations. If the version would match, we knew it was a key with separate client\_id and client\_secret and could split and parse it correctly.
However there is a problem with this.
We are encoding the version as well as the byte-length at the *beginning* of the key, this makes all of our keys look very similar when you’re only seeing the first few characters.
Other complications come from the fact that we want to be able to onboard other keys without the end user noticing, which means we can not force them to reroll their key, but need to accept it as is, we can only change the hash function to our own (sha256). This means, we can never enforce a strict schema on keys, we can only try to shift the key landscape, by asking users to reroll.
Example:
Resend have keys in the shape of `re_\{user_id\}_\{secret\}_\{checksum\}`
When we want to win them as a customer, we do not want to ask all of their users to create new keys, existing keys must keep working without our customer having to maintain their own system as well as unkey.
We do this by adding their hashes to our db and marking them as well as noting how we derive the hash from the key (see [https://linear.app/unkey/issue/ENG-119/migrating-keys-to-unkey](https://linear.app/unkey/issue/ENG-119/migrating-keys-to-unkey)). When they now create new keys through unkey, they’ll have our own keyshape, but old keys do not change.
At this point when we receive a key, we do not know what shape it’ll be, it could be a resend shape, it could be one of our own, or it could be something entirely different from another customer.
## Possible Solution
Having a version encoded is still a good idea I think, but there’s nothing stopping us from moving it to the end, instead of parsing a key from the start, we can just start reading from the back.
## Other thoughts
* We should also think of adding a checksum, which would allow us to clearly identify “this key was created by unkey”, as well as it would help with github secret scanning stuff.
* A problem with github secret scanning is, that we allow our users to choose their prefix, so it’s not as simple as matching `/^unkey_.{16}$/`, we’ll have to look a bit deeper, but a checksum may help.
* our future key shape could look like this:
```tsx theme={"theme":"kanagawa-wave"}
{prefix}_{base58_randomness}{meta}{version}_{checksum}
```
I don’t really like the second delimiter, but I haven’t thought deeply enough how we can match that in a way that doesn’t accidentally match a migrated key
# 0004 COSS Starter
Source: https://engineering.unkey.com/architecture/rfcs/0004-coss-starter
### **Problem**:
One of our biggest hurdles is penetrating the market because you need to be building either an API or you have one and need something we offer. This is fine but what if you don’t know if you need one? Or if you are building an application for the first time and have never considered it.
### **Proposed solution**:
Open Source starter that is highly opinionated and includes everything you need to build a new product.
1. Web app with internal API
2. Public API with open API spec protected by Unkey
3. Docs
4. Stripe integrations
Imagine typing `npx create-coss-app`, and there you have it. All you need to do is sign up for 1 or 2 services, including Unkey. We want it to be DX-driven and super easy to extend and modify, but keep it so easy that an FE engineer can do it.
We can get a domain with a landing page that points to the npx command and some documentation for how it works.
## Tech stack proposed
We can create a monorepo using turbo repo for the entire stack to make it as easy as possible to use.
### Web application:
* Next.js
* tRPC
* Lucia auth
* Drizzle
* Planetscale
* Stripe
* Unkey
### Public API
* Hono v4
* Unkey
## Docs
* Mintlify using the open API spec built using Hono v4.
The proposed solution uses the smallest amount of 3rd party services (Planetscale, Mintlify, and Unkey), allowing maintenance and contributions a small hurdle.
## Work to be done
Come up with a simplified project to show how everything works. An example could be a to-do list or something similar.
1. Web app has a UI interface that allows for creating, updating, and deleting todos, but you can only have a maximum of 5 on the free tier. The web app also allows for the creation of API keys to provide API access (The free tier gets limited uses or rate limiting or whatever we pick) and a way for the user to upgrade to the paid tier with unlimited.
2. API has keys from Unkey passed in and can be used to CRUD the data via the API.
3. Document everything as a real product to show how documentation works.
4. Basic landing page with the command and link to docs and a demo
5. The demo is just a deployed version of this.
Inspiration : [https://create.t3.gg/](https://create.t3.gg/)
# 0005 Analytics API
Source: https://engineering.unkey.com/architecture/rfcs/0005-analytics-api
Unkey exposes APIs to retrieve all required data to build end-user facing dashboards and drive our customer's usage-based billing.
## Motivation
Consumption based billing for APIs is getting more and more popular, but it's tedious to build in house.
For low frequency events, it's quite possible to emit usage events directly to Stripe or similar, but this becomes very noisy quickly.
Furthermore if you want to build end-user facing or internal analytics, you need to be able to query the events from Stripe, which often does not provide the granularity required.
Most teams end up without end-user facing analytics, or build their own system to store and query usage metrics.
Since Unkey already stores and aggregates verification events by time, outcome and identity, we can offer this data via an API.
## Detailed design
In order to charge for usage, our users need information of **who** used their API **when** and **how often**.
For end-user facing analytics dashboards, it would also be relevant to differentiate between different outcomes (`VALID`, `RATE_LIMITED`, `USAGE_EXCEEDED`, `INSUFFICIENT_PERMiSSIONS` etc.)
### Available data
We already store events for every verification in ClickHouse and have materialized views for aggregations.
```sql theme={"theme":"kanagawa-wave"}
`request_id` String,
`time` Int64,
`workspace_id` String,
`key_space_id` String,
`key_id` String,
`region` LowCardinality(String),
`outcome` LowCardinality(String),
`identity_id` String
```
We can return this data in different granularities:
* hourly
* daily
* monthly
In order be scalable, we will not expose individual events in the beginning, nor allow you to filter by exact timestamps. If we can't query a materialized view, it would be too compute-intensive to query.
If needed, a per-minute granularity materialized view could be created, but is not currently planned.
And filtered by:
* identity\_id
* key\_space\_id (which we can derive from the api\_id)
* key\_id
* outcome
* start and end time
### Request
We will create a new endpoint `GET /v1/analytics.getVerifications`, protected by a root key in the `Authorization` header.
The root key will require specific permissions tbd.
Calling the endpoint will return an array of verification counts, aggregated by time and provided filters.
All required and optional arguments are passed via query parameters. Some parameters may be specified multiple times, either as You may specify multiple ids such as `?param=value_1,value_2` or `?param=value_1¶m=value_2`
#### start
(integer, required)
Unix timestamp in milliseconds to specify the start of the interval to retrieve.
We will return all datapoints with a timestamp greater or equal to `start`.
There may be restrictions depending on the granularity chosen and the retention quota of the customer
#### end
(integer, required)
Unix timestamp in milliseconds to specify the end of the interval to retrieve.
We will return all datapoints with a timestamp less than or equal to `end`.
There may be restrictions depending on the granularity chosen and the retention quota of the customer
#### granularity
(enum \["hour", "day", "month"], required)
Selects the granularity of data. For example selecting `hour` will return one datapoint per hour.
#### apiId
(string, optional, may be provided multiple times)
Select the API for which to return data.
When you are providing zero or more than one API ids, all usage counts are aggregated and summed up.
Send multiple requests with one apiId each if you need counts per API.
#### externalId
(string, optional, may be provided multiple times)
Filtering by externalId allows you to narrow down the search to a specific user or organisation.
When you are providing zero or more than one external ids, all usage counts are aggregated and summed up.
Send multiple requests with one externalId each if you need counts per identity.
#### keyId
(string, optional, may be provided multiple times)
Only include data for a speciifc key or keys.
When you are providing zero or more than one key ids, all usage counts are aggregated and summed up.
Send multiple requests with one keyId each if you need counts per key.
#### groupBy
(enum \["key", "identity"], optional)
By default, all datapoints are aggregated by time alone, summing up all verifications across identities and keys.
However in certain scenarios you want to get a breakdown per key. For example finding out the usage spread across all keys for a specific user.
### limit
(integer, optional)
Limit the number of returned datapoints. This may become useful for querying the top 10 identities based on usage.
#### orderBy
(enum \["total", "valid", ..], optional)
This is a rough idea.
We're leaning towards `?orderBy=valid&order=asc`, but have not decided what this API should look like.
#### order
(enum \["asc", "desc"], optional, default="asc", only allowed in combination with `orderBy`)
See above.
### Example Access Patterns
> A chart of an enduser's usage over the past 24h, showing the outcomes
```bash theme={"theme":"kanagawa-wave"}
?start={timestamp_24h_ago}&end={timestamp_now}&externalId=user_123&granularity=hour
[
// 24 elements, one per hour
{ time: 123, valid: 10, ratelimited: 2, ..., total: 30 },
]
```
> A daily usage breakdown for a user per key in the current month
```bash theme={"theme":"kanagawa-wave"}
?start={timestamp_start_of_month}&end={timestamp_now}&granularity=day&externalId={user_123}&groupBy=key
[
// One row per keyId and time
{ keyId: "key_1", time: 123, valid: 10, ..., total: 30 },
{ keyId: "key_1", time: 456, valid: 20, ..., total: 52 },
...
{ keyId: "key_2", time: 123, valid: 0, ..., total: 10 },
{ keyId: "key_2", time: 456, valid: 1, ..., total: 2 },
...
]
```
> A monthly cron job creates invoices for each identity:
```bash theme={"theme":"kanagawa-wave"}
?start={timestamp_start_of_month}&end={timestamp_end_of_month}&granularity=month&externalId={user_123}
[
// one element for the single month
{ time: 123, valid: 10, ..., total: 30 }
]
```
> A user sees a gauge with their quota, showing they used X out of Y API calls in the current billing period:
```bash theme={"theme":"kanagawa-wave"}
?start={timestamp_start_of_billing_cycle}&end={timestamp_end_of_billing_cycle}&granularity=day&externalId={user_123}
[
{ time: 123, valid: 10, ..., total: 30 }
]
```
Sum up the `valid` or `total`, however you want to count, and display it to the user.
> An internal dashboard shows the top 10 users by API usage over the past 30 days
```bash theme={"theme":"kanagawa-wave"}
?start={timestamp_30_days_ago}&end={timestamp_now}&granularity=day&groupBy=identity&limit=10&orderBy=total&order=desc
```
### Response
Successful responses will always return an array of datapoints. One datapoint per granular slice, ie: hourly granularity means you receive one element per hour within the queried interval.
```json title="200 OK Body" theme={"theme":"kanagawa-wave"}
[
Datapoint,
Datapoint,
Datapoint
]
```
```ts title="Datapoint" theme={"theme":"kanagawa-wave"}
type Datapoint = {
/**
* Unix timestamp in milliseconds of the start of the current time slice.
*/
time: number
/**
* For brevity, I will not explain every outcome here.
* There will be one key and count for every possible outcome, so you may
* choose what to display or not.
*/
valid: number
rateLimited: number
usageExceeded: number
// ...
/**
* Total number of verifications in the current time slice, regardless of outcome.
*/
total: number
/**
* Only available when specifying groupBy in the query.
* In this case there would be one datapoint per time and groupBy target.
*/
keyId?: string
apiId?: string
identity?: {
id: string
externalId: string
}
}
```
## Drawbacks
Our current serverless architecture costs money per invocation. Our customer's users could generate a decent amount of requests.
## Alternatives
Offering a prometheus `/metrics` endpoint would be interesting, however I believe most of our users don't have the infra in place to adopt this easily.
***
Instead of aggregating multiple keyIds together, we could disallow specifying them multiple times and instead ask the user to create one request per id and then merge them together on their side.
## Unresolved questions
* What cache times are acceptable? We probably don't want to hit ClickHouse for every single query, especially for fetching monthly aggregations.
* When we return keyIds as part of groupBy queries, the user needs to make another call to our API in order to fetch details such as the name for each key. That doesn't feel great.
* What are the retention quotas tier and granularity?
# 0006 Auth Migration
Source: https://engineering.unkey.com/architecture/rfcs/0006-auth-migration
Migrate everything to WorkOS, despite their bad APIs..
## Motivation
We need to migrate our users and organisations from Clerk to WorkOS.
## Detailed design
In order to apply the changes safely and, more importantly, roll back in case we need to, we'll need to do it in 3 steps.
Each step should be its own PR, so we can roll back safely.
### 1. Migrate Data
1. Create a new non-nullable varchar column in our `workspaces` table with default `""`.
2. Migrate all users from Clerk to WorkOS. They will receive a new `user_id`, which is fine.
3. Migrate all organisations and personal workspaces from Clerk to WorkOS. These will also have a new `org_id`, which we will store in the `workspaces.organisation_id` column.
### 2. Switch Reads
(Meg's PR)
We deploy the dashboard changes to production, which will now receive an `orgId` from the WorkOS SDK and change our db queries to match against the `organisation_id` field, instead of `tenant_id`.
If we really need to, we can roll this back at any time and use clerk again. The only problem here is that users that were created after the switch, would not be in Clerk.
### 3. Remove old columns
After we're happy with everything and it's been running smoothly for a few weeks, we can remove the old `tenant_id` column.
# 0007 Client-side file structure
Source: https://engineering.unkey.com/architecture/rfcs/0007-client-file-structure
File structure for our client apps
## Executive Summary
This RFC proposes restructuring our client components from their current flat organization into a feature-based architecture, grouping related components, hooks, and utilities within feature-specific directories. Each Next.js page will be treated as a distinct feature module, ensuring clear boundaries and colocation of related code. The migration can be implemented incrementally, with each feature module being refactored independently without disrupting ongoing development.
Key benefits include:
* Improved developer onboarding through intuitive code organization
* Reduced coupling between features
* Faster feature development through clear patterns and conventions
* Better code maintainability through consistent structure
* Easier code reviews through predictable file locations
* **Standardized contribution patterns for our open source community**
## Problem Statement
### Current Situation
Our Next.js application's flat directory structure has led to several challenges:
1. Related code is scattered across different directories, making it difficult to understand feature boundaries
2. New team members spend excessive time locating relevant components and understanding relationships
3. Lack of consistent patterns leads to inconsistent implementations
4. Code reuse is hindered by poor discoverability of existing components
5. Utilities often end up far from the components they support
A critical issue in our open-source project is the lack of standardized patterns. Currently:
* Different contributors implement features using their own organizational preferences because they don't know our pattern.
* This creates inconsistency across the codebase
* Code reviews take longer as reviewers need to understand each contributor's unique approach
* New contributors lack clear examples to follow
* Integration of community contributions requires significant refactoring
For example, our `/authorization` page demonstrates these issues...
```bash theme={"theme":"kanagawa-wave"}
├── authorization/
│ ├── permissions/
│ │ ├── [permissionId]/
│ │ │ ├── client.tsx
│ │ │ ├── delete-permission.tsx
│ │ │ └── page.tsx
│ │ ├── create-new-permission.tsx
│ │ └── page.tsx
│ └── roles/
│ ├── [roleId]/
│ │ ├── delete-role.tsx
│ │ ├── page.tsx
│ │ ├── permission-toggle.tsx
│ │ ├── tree.tsx
│ │ └── update-role.tsx
│ ├── create-new-role.tsx
│ └── page.tsx
├── constants.ts
└── layout.tsx
```
We could turn this into this:
```bash theme={"theme":"kanagawa-wave"}
├── authorization/
│ ├── permissions/
│ │ ├── [permissionId]/
│ │ │ ├── components/
│ │ │ │ └── permission-details.tsx
│ │ │ ├── actions/
│ │ │ │ └── delete-permission.ts
│ │ │ ├── hooks/ # Page-specific query hooks
│ │ │ │ └── use-permission.ts # Single permission queries
│ │ │ └── page.tsx
│ │ ├── components/
│ │ │ ├── create-new-permission/
│ │ │ │ ├── index.tsx
│ │ │ │ └── permission-form.tsx
│ │ ├── schemas/ # New validation schemas folder
│ │ │ ├── permission-form.schema.ts # .schema or -schema suffix are both fine.
│ │ │ └── permission.schema.ts
│ │ ├── types/
│ │ │ └── permission.ts
│ │ ├── utils/
│ │ │ └── permission-validator.ts
│ │ ├── hooks/
│ │ │ ├── use-permission-form.ts
│ │ │ └── queries/ # Shared permission query hooks
│ │ │ ├── use-permissions-list.ts
│ │ │ ├── use-create-permission.ts
│ │ │ └── use-update-permission.ts
│ │ ├── constants.ts # Permission wide constants
│ │ └── page.tsx
├── constants/
│ └── shared.ts # Authorization wide constants
```
And, actual page files will look like this. Note this is audit component refactored from this [Old Audit Page](https://github.com/unkeyed/unkey/blob/46878c232b3e57372f43141816e508f63c6570fd/apps/dashboard/app/\(app\)/audit/%5Bbucket%5D/page.tsx) to this:
```ts theme={"theme":"kanagawa-wave"}
import { Navbar } from "@/components/navbar";
import { PageContent } from "@/components/page-content";
import { getOrgId } from "@/lib/auth";
import { InputSearch } from "@unkey/icons";
import { type SearchParams, getWorkspace, parseFilterParams } from "./actions";
import { Filters } from "./components/filters";
import { AuditLogTableClient } from "./components/table/audit-log-table-client";
type Props = {
params: {
bucket: string;
};
searchParams: SearchParams;
};
export default async function AuditPage(props: Props) {
const orgId = await getOrgId();
const workspace = await getWorkspace(orgId);
const parsedParams = parseFilterParams({
...props.searchParams,
bucket: props.params.bucket,
});
return (
Audit
{workspace.ratelimitNamespaces.find((ratelimit) => ratelimit.id === props.params.bucket)
?.name ?? props.params.bucket}
);
}
```
Contributors and our team will be able to easily locate functions and components, and get a general feel for the component immediately.
### Impact
This problem affects multiple stakeholders in our ecosystem:
Developer Community:
* Open source contributors face a learning curve when trying to understand where to place new code
* Community developers spend extra time in code review discussions about file organization rather than functionality
* First-time contributors often need multiple revision cycles just to match project structure
Core Team:
* Maintainers spend significant time providing structural guidance in PRs
* Code review efficiency is reduced by inconsistent file organization
* Integration of community contributions requires extra refactoring effort
End Users:
* Feature delivery is slowed by organizational overhead
* Bug fixes take longer as developers navigate inconsistent structures
* New features may be delayed due to time spent on structural debates
### Motivation
Solving this organizational challenge is critical for several reasons:
Project Scalability:
* As our project grows, the cost of inconsistent structure compounds
* More contributors means more potential for divergent patterns
* Larger features become increasingly difficult to maintain without clear boundaries
Community Growth:
* Clear conventions lower the barrier to entry for new contributors
* Standardized patterns help contributors focus on value-add features rather than structure
* Predictable organization improves documentation and knowledge sharing
Development Velocity:
* Consistent patterns reduce cognitive load during development
* Feature implementation time decreases as conventions become second nature
* Code reviews can focus on logic and functionality rather than organization
* Faster onboarding for new contributors who can follow established patterns
Code Quality:
* Well-organized code is easier to test and maintain
* Clear boundaries prevent unwanted coupling between features
* Consistent structure makes it easier to identify and fix architectural issues
## Proposed Solution
### Overview
We propose implementing a feature-based architecture where each distinct feature (Next.js page) is treated as a self-contained module with its own component hierarchy. The structure follows these key principles:
1. Feature Isolation
* Each feature (page) gets its own directory
* All related components, hooks, and utilities live within the feature directory
* Shared code is clearly separated from feature-specific code
2. Consistent Internal Structure
Each feature directory follows a standard organization:
* `/components`: Feature-specific React components
* `/hooks`: Custom hooks for the feature
* `/actions`: Server actions and API calls
* `/types`: Types and interfaces
* `/schemas`: Zod schemas
* `/utils`: Helper functions and utilities
* `/constants`: Feature-specific constants
3. Clear Dependencies
* Shared components live in a global `@components` or `/components` directory
* Feature-specific components shouldn't be imported by other features
* Common utilities, types and components are placed in root-level shared directories
As demonstrated in the example of the `/authorization` feature above.
## Alternatives Considered
### Alternative 1: Features folders
#### Description
A simpler feature-based structure where all features are in a `/features` directory:
```bash theme={"theme":"kanagawa-wave"}
├── features/
│ ├── authorization/
│ │ ├── components/
│ │ ├── hooks/
│ │ └── utils/
│ ├── audit/
│ └── billing/
├── shared/
│ ├── components/
│ └── utils/
└── pages/
```
#### Pros
* Clear separation between features and shared code
* Simpler top-level organization
* Common pattern in React applications
* Less nesting compared to proposed solution
* Easier to colocate tRPC and page related code
#### Cons
* More difficult to colocate route-specific code
* Less granular organization within features
* Harder to implement incrementally
* Mixing of page-specific and feature-wide code
#### Why we didn't choose this
Going from what we have to this one is really hard to do incrementally.
### Alternative 2: Flat structure (current)
#### Description
Our current flat structure where files are organized by type:
#### Pros
* Simple to understand
* No complicated nesting
* Easier to move components between features
#### Cons
* Related code is scattered
* No clear feature boundaries
* Poor scalability as app grows
* Difficult to understand feature scope
* Hard for new contributors to know where to put things
* Mixed responsibilities in directories
* No clear ownership of code
* Harder to refactor single features
* OSS contributors tend to put files in random places
#### Why we didn't choose this
The flat structure has proven problematic as our project grows and receives more open source contributions. The lack of clear conventions leads to inconsistent implementations and makes it harder for new contributors to understand where their code should go. The proposed solution provides clearer boundaries and better guides contributors toward consistent patterns.
## Future Work
After implementing the initial structure, we can gradually move towards a more framework-agnostic `/features` organization:
* Move framework-independent code (components, hooks, utils) into `/features`
* Keep Next.js specific files (page.tsx, loading.tsx, error.tsx) in the App Router structure
* This separation will make our codebase more portable
## Questions and Discussion Topics
* Are there too many levels of nesting in the proposed structure?
* Maybe we should adopt Remix.js-style suffixes for better clarity? Examples:
* `.type.ts` for type definitions
* `.schema.ts` for validation schemas
* `.client.tsx` for client-specific components
* `.server.ts` for server-only code
* `.action.ts` for server actions
***
## Document History
| Version | Date | Description | Author |
| ------- | ---------- | ------------- | ------ |
| 0.1 | 2024-12-20 | Initial draft | @Oz |
# 0008 Dataplane
Source: https://engineering.unkey.com/architecture/rfcs/0008-dataplane
Global Unkey Deployment Architecture
We need to design a globally distributed architecture for Unkey where the dataplane can operate independently of the primary database for improved availability.
## Goals
* Achieve 100% dataplane availability independent of primary database
* Provide fast access to dynamic data across global regions
* Propagate data across the system quickly
* Minimize load on expensive storage
* Enable efficient cache invalidation
* Can run on any cloud or on premise
## Options
### 1. Direct S3 + In-Memory Cache with SWR
```ascii theme={"theme":"kanagawa-wave"}
┌─────────────────────┐ ┌─────────┐
│ Frontline │ │ │
│ ┌───────────────┐ │────►│ S3 │
│ │ Memory Cache │ │ │ │
│ └───────────────┘ │ │ │
└─────────────────────┘ └─────────┘
```
#### Pros
* Simple, straightforward design
* Low architectural complexity
#### Cons
* Cache invalidation requires communication with all machines or really low TTLs (\<10s)
* Inefficient cache refresh patterns
* High load on S3 due to concurrent SWR requests from multiple machines
### 2. S3 + In-Memory Cache with Gossip Protocol
```ascii theme={"theme":"kanagawa-wave"}
┌─────────────────────┐
│ Frontline │
│ ┌───────────────┐ │──┐
│ │ Memory Cache │ │ │
│ └───────────────┘ │ │
└─────────────────────┘ │
▲ │
│ Gossip │ ┌─────────┐
▼ ├───►│ │
┌─────────────────────┐ │ │ S3 │
│ Frontline │ │ │ │
│ ┌───────────────┐ │──┘ └─────────┘
│ │ Memory Cache │ │
│ └───────────────┘ │
└─────────────────────┘
```
If we had global fast and efficient eviction, we could set much higher TTLs.
#### Pros
* Efficient cache invalidation through gossip, allowing higher TTLs
* Reduced load on primary storage
* Only need to notify one node for changes
#### Cons
* Need to implement ordering mechanism (timestamps/Lamport clocks)
* More complex system architecture
* Global gossip cluster management overhead
### 3. S3 + Dedicated Cache Layer
```ascii theme={"theme":"kanagawa-wave"}
┌─────────────────┐
│ Frontline 1 │───┐
└─────────────────┘ │
│
┌─────────────────┐ │ ┌────────────┐
│ Frontline 2 │───┼───►│ Load │ ┌────────────┐
└─────────────────┘ │ │ Balancer │───►│ Cache │──┐
│ │ │ │ Node 1 │ │
┌─────────────────┐ │ │ │ └────────────┘ │
│ Frontline 3 │───┤ │ │ │ ┌─────────┐
└─────────────────┘ │ │ │ ├───►│ S3 │
├───►│ │ ┌────────────┐ │ └─────────┘
┌─────────────────┐ │ │ │───►│ Cache │──┘
│ Frontline 4 │───┤ │ │ │ Node 2 │
└─────────────────┘ │ └────────────┘ └────────────┘
│
┌─────────────────┐ │
│ Frontline n │───┘
└─────────────────┘
```
Frontline instances still use an in-memory SWR cache. The cache nodes help reduce the cost
and latency of S3, but will likely have the same freshness/staleness as the
Frontline instances. TBD..
Everything here, except the S3 bucket, would be duplicated per region.
#### Pros
* Better cache retention due to less frequent reboots
* Optional global eviction via gossip/kafka later
* Maybe we only need 1 S3 region now instead of replicating it
#### Cons
* Additional infrastructure to manage
### 4. DynamoDB Global Tables + Caching
Option 4A: Direct DynamoDB + Frontline Memory Cache
* Each Frontline instance maintains a local memory SWR cache with a TTL of 10s
* DynamoDB serves as source of truth
* Automatic multi-region replication handled by AWS
```ascii theme={"theme":"kanagawa-wave"}
┌─────────────────────┐ ┌──────────────────┐
│ Frontline (US) │ │ DynamoDB │
│ ┌───────────────┐ │────►│ (US-WEST-1) │
│ │ Memory Cache │ │ │ │
│ └───────────────┘ │ └──────────────────┘
└─────────────────────┘ ▲
│
│ Replication
│
┌─────────────────────┐ ▼
│ Frontline (EU) │ ┌──────────────────┐
│ ┌───────────────┐ │────►│ DynamoDB │
│ │ Memory Cache │ │ │ (EU-WEST-1) │
│ └───────────────┘ │ │ │
└─────────────────────┘ └──────────────────┘
```
Option 4B: With Dedicated Cache Layer
A dedicated cache layer could be added to reduce the load on DynamoDB and improve read performance as well as cost.
Whether this actually saves money is debatable, we'll have to try.
These cache nodes would be dumb, they only cache reads for 10s and don't have any manual eviction possibilities.
```ascii theme={"theme":"kanagawa-wave"}
┌─────────────────┐
│ Frontline 1 │───┐
└─────────────────┘ │
│ ┌────────────┐
┌─────────────────┐ │ │ Load │ ┌────────────┐
│ Frontline 2 │───┼───►│ Balancer │───►│ Cache │──┐
└─────────────────┘ │ │ │ │ Node 1 │ │ ┌──────────────┐
│ │ │ └────────────┘ ├───►│ DynamoDB │
│ │ │ │ │ Global │
│ │ │ ┌────────────┐ │ │ Tables │
┌─────────────────┐ │ │ │───►│ Cache │──┘ └──────────────┘
│ Frontline n │───┘ └────────────┘ │ Node 2 │
└─────────────────┘ └────────────┘
```
#### Pros
* Built-in multi-region replication with strong consistency
* No need to manage complex replication logic
* Lower latency reads from local region
* Automatic conflict resolution
* Serverless and fully managed by AWS
* Cheaper for small reads than S3
* 99.999% availability (s3 only has 99.99%)
#### Cons
* Vendor lock-in to AWS -> we need to have an abstraction
* Higher storage cost compared to S3 due to replication
* Cost of replication
* Replication lag is controlled by AWS, not us
* More expensive for large >13kb reads than S3
# 0009 Pricing refresh for 2025
Source: https://engineering.unkey.com/architecture/rfcs/0009-pricing-updates
We need to update our pricing for 2025
## Summary
Our pricing has been the same for the past 12 months, while we want stability we also want to ensure we are competitive in the market. We are proposing a refresh of our pricing and free tier to ensure we are competitive and can continue to grow our business.
## Motivation
Our whole ethos has always been "Making developers lives easier" and we want to continue to do that.
We want to ensure that we are competitive in the market and that we are providing value to our customers, and currently our pricing has two main issues:
1. Unexpected bills due to the way we charge for requests, can be scary.
2. Confusing pricing structure that is only getting more confusing as we add more features
a. Ratelimiting
b. API key verification
c. What consisitutes a valid request?
We have seen a lot of growth in the past 12 months and we want to ensure that we can continue to grow by making it easier for developers to adopt, and scale with us.
## Details of the pricing refresh
Firstly we are moving away from "verifications" and moving towards using the "requests", this will make it easier to transition into our new features we rely directly on requests.
#### For reference our current pricing is:
| Free | Pro | Enterprise |
| ------------------------ | ------------------------ | ---------------------- |
| 2.5k Valid Verifications | 150k Valid Verifications | XM Valid Verifications |
| 100k Valid Ratelimits | 2.5M Valid Ratelimits | XM Valid Ratelimits |
| 7 days Logs | 90 Days Logs | X Days Logs |
| 30 days Audit | 90 Days Audit | X Days Audit |
| 1k API keys | 1M API Keys | 1M API Keys |
| Unlimited APIs | Unlimited APIs | Unlimited APIs |
| \$0 | \$25 | \$? |
#### Charge for additional requests:
* \$1 per 10k
* \$1 per 100k (Ratelimits)
### Proposal for 2025
When we met in October 2024, we discussed moving to buckets of requests, where a developer can have a certain number of requests per month. This is similar to other SaaS products like `Resend` or `Dub`.
| Free | Pro Tier 1 | Pro Tier 2 | Pro Tier 3 | Pro Tier 4 | Pro Tier 5 | Pro Tier 6 | Pro Tier 7 | Enterprise |
| -------------- | ---------------------- | ---------------------- | ----------------- | ------------------------ | ------------------ | ------------------ | ---------------- | -------------- |
| 150k Requests | Up To 250,000 Requests | Up To 500,000 Requests | UP To 1M Requests | Up To 2,000,000 Requests | Up to 10M Requests | Up to 50M Requests | Up 100M Requests | XM requests |
| 7 Days Logs | 90 Days Logs | 90 Days Logs | 90 Days Logs | 90 Days Logs | 90 Days Logs | 90 Days Logs | 90 Days Logs | X Days Logs |
| 30 days Audit | 90 Days Audit | 90 Days Audit | 90 Days Audit | 90 Days Audit | 90 Days Audit | 90 Days Audit | 90 Days Audit | X Days Audit |
| 1k API keys | 1M API Keys | 1M API Keys | 1M API Keys | 1M API Keys | 1M API Keys | 1M API Keys | 1M API Keys | 1M API Keys |
| Unlimited APIs | Unlimited APIs | Unlimited APIs | Unlimited APIs | Unlimited APIs | Unlimited APIs | Unlimited APIs | Unlimited APIs | Unlimited APIs |
| \$0 | \$25 | \$50 | \$75 | \$100 | \$250 | \$500 | \$1000 | \$? |
To provide context 100M API Key verifications on the old pricing would cost $10,010.00 but on the new pricing would cost $1,000.00.
#### What is counted as a successful request?
A successful request means everything is fine and you should grant access to the user. Requests may be unsuccessful due to exceeding limits, keys being expired or disabled, or other factors.
To protect your business from abuse, we do not charge for unsuccessful requests.
#### What happens if I exceed my tier?
If you exceed your tier, we will continue to allow traffic to flow and won’t charge you for the additional usage. If you API continues to exceed the tier we will advise you to upgrade to the next tier up.
## Drawbacks
Standalone ratelimiting becomes more expensive as a SaaS scales however adding more requests to the free tier will help offset this. This seems like a win for scaling into the product, and getting more eyes on it, rather than a negative.
## Alternatives
The only alternative considered so far:
1. Keeping the pro tier as Pay as you go, while also adding bucket pricing. This could cause additional confusion, or even more support requests as they realize bucketing is much cheaper.
# 0010 Splitting the monorepo
Source: https://engineering.unkey.com/architecture/rfcs/0010-split-monos
Splitting the monorepo into multiple smaller, more focused repositories / monorepos.
## Summary
As our codebase continues to grow, the need for scalability, maintainability, and improved development workflows has become more critical. This RFC proposes splitting our current monorepo into multiple smaller, more focused repositories / monorepos.
## Motivation
We have on several occasions encountered issues with our current Monorepo structure, such as:
* Increased build times due to large codebases
* Complex dependency management across multiple projects
* Difficulty onboarding new contributors due to the size and complexity of the monorepo
* Conflicts between different projects
There is nothing worse than pushing a change that affects a single project only to have a separate project affected as well. While monorepos make sense, splitting the monorepo into smaller, more focused repositories can help address these issues.
## Detailed design
The proposal is to transition from a monorepo to a multi-repo / monorepo architecture, breaking up our codebase into several repositories based on functionality and project scope. Each repository would encapsulate its dependencies, tools, and build processes.
### Key Repositories
1. **Core**: The Core API code, dashboard, and engineering documentation is contained.
2. **SDKs**: Houses all our SDKs, this makes autogeneration of SDKs easier.
3. **Infra**: This includes all infra codes in its repo today.
4. **Documentation**: Dedicated documentation for Unkey.
5. **Marketing site**: It's not dependent on anything else, so we should just keep it separate. This includes the playground project which should be maintained by the same developers.
## Benefits
* **Improved Build Performance**: Smaller repositories lead to faster incremental builds and no more waiting for unrelated builds to fire off.
* **Improved GH Actions**: Only run relevant actions for each repository.
* **Simplified Dependency Management**: Each repository will manage its dependencies, making upgrading and maintaining easier.
* **Easier Onboarding**: New developers can clone and work on a smaller codebase relevant to their tasks.
## Drawbacks
The drawback is that there are more repositories to clone and maintain. However, grouping core product functionality into a single repository can lead to a more cohesive and unified codebase, making it easier to maintain and develop.
## Alternatives
The alternative is to keep the mono repo structure as is and continue to bloat it with unrelated projects.
## Unresolved questions
* **Migration Plan**: A clear, step-by-step plan will need to be established for moving code and setting up each new repository.
* **Communication**: Ensure all team members are updated on changes and can adapt to the new structure, including when we should pull the main and checkout codes.
* **Tooling**: Set up the necessary tools for managing multiple repositories if needed.
# 0008 URNs
Source: https://engineering.unkey.com/architecture/rfcs/0011-unkey-resource-names
Implementing Uniform Resource Names (URNs) and Structured Error Codes at Unkey
This RFC records the original proposal. The implemented `v1` resource-name
contract is documented in [Unkey Resource Names](/architecture/resources/unkey-resource-names)
and [Resource permissions](/architecture/authorization/resource-permissions).
The implemented contract uses `unkey:v1:{workspace_id}:{resource_path}` and
includes resource-name patterns with `*` and trailing `/**` for authorization
grants.
## 1. Background
As we grow Unkey from a single API authentication product into a comprehensive API infrastructure platform with multiple services, we face increasing complexity in resource identification and error handling. Our current approach lacks a consistent, scalable system for uniquely identifying resources across services and providing structured error information to developers.
I'm proposing two complementary systems:
1. A hierarchical URN schema for all Unkey resources
2. A structured error code system that provides detailed, actionable error information
Both systems are designed to scale with our expanding product surface while providing developers with a consistent, intuitive experience.
## 2. Resource Identification: URN Structure
### 2.1 Format
I propose adopting a URN structure for all Unkey resources:
```
urn:unkey:{service}:{workspace_id}:{environment}:{resource-type}/{resource-id}
```
Where:
* `urn:unkey` - Fixed prefix indicating this is a Unkey resource identifier
* `service` - The Unkey service (auth, ratelimit, deploy, etc.)
* `workspace_id` - ID of the workspace containing the resource
* `environment` - Environment within the workspace (production, staging, etc.)
* `resource-type` - Type of resource (key, api, identity, etc.)
* `resource-id` - Unique identifier for the specific resource
For customer cloud deployments, we can extend this pattern:
```
urn:{customer-cloud-id}:{service}:{workspace_id}:{environment}:{resource-type}/{resource-id}
```
### 2.2 Examples
```
# API resource
urn:unkey:auth:ws_123456:production:api/api_abcdef
# Key resource
urn:unkey:auth:ws_123456:staging:key/key_xyz123
# Rate limit namespace
urn:unkey:ratelimit:ws_123456:production:namespace/ns_abc123
# Customer cloud deployment
urn:customer_abc:auth:ws_123456:production:key/key_xyz123
```
### 2.3 Service Namespace
Based on our product roadmap, I propose the following service namespaces:
* `auth` - Authentication and authorization
* `ratelimit` - Rate limiting service
* `identity` - Identity management
* `deploy` - Deployment service
* `observe` - Observability platform
* `audit` - Audit logging service
* `secrets` - Secrets management
* `billing` - API monetization/billing
## 3. Error Identification: Error Code Structure
### 3.1 Format
For error codes, I propose a separate namespace with a more concise format:
```
err:{service}:{category}:{specific-error}
```
Where:
* `err` - Fixed prefix indicating this is an error code
* `service` - The Unkey service where the error occurred
* `category` - Broad category of the error
* `specific-error` - Specific error type
### 3.2 Error Categories
I recommend standardizing on the following error categories across all services:
1. `state` - Existence, status, and lifecycle issues
* Resource not found
* Resource already exists
* Resource disabled/expired
2. `validation` - Format and content issues
* Invalid formats
* Schema violations
* Constraint violations
3. `permissions` - Access control issues
* Insufficient permissions
* Unauthorized access
* Role requirements
4. `limits` - Quota and capacity issues
* Rate limits exceeded
* Storage limits exceeded
* Quota limits exceeded
5. `configuration` - Setup problems
* Invalid settings
* Incompatible configurations
* Missing required settings
### 3.3 Error Code Examples
```
# Authentication error - key not found
err:auth:state:key_not_found
# Rate limiting error
err:ratelimit:limits:exceeded
# Deployment error
err:deploy:validation:schema_violation
```
## 4. Implementation Strategy
### 4.1 API Version Approach
I propose implementing these systems using a clean API versioning strategy:
* **V1 API (Current)**: Maintains existing ID formats and error responses for complete backward compatibility
* **V2 API (New)**: Fully implements the URN and error code systems as the standard approach
This creates a clean separation between versions and avoids complex migration paths for existing clients. Developers can choose when to adopt the new systems by migrating to the V2 API.
### 4.2 Implementation Focus
For the V2 API implementation:
* All resources will be identified using the URN schema
* All errors will follow the structured error code format
* Documentation will be built around these new systems
* Client libraries will support the new formats natively
### 4.3 Documentation and Tooling
To support this approach:
* Clear migration guides for moving from V1 to V2
* Automatic URN generation for resources created via the V2 API
* Comprehensive documentation of the new URN and error systems
* Developer tools to help work with and parse URNs
## 5. API Response Examples
### 5.1 Resource Response with URN
```json theme={"theme":"kanagawa-wave"}
{
"id": "key_xyz123",
"urn": "urn:unkey:auth:ws_123456:production:key/key_xyz123",
"name": "Production API Key",
"enabled": true,
"created_at": "2023-01-01T00:00:00Z"
}
```
### 5.2 Error Response with Error Code
```json theme={"theme":"kanagawa-wave"}
{
"error": {
"code": "RATE_LIMITED",
"err": "err:ratelimit:limits:exceeded",
"message": "You have exceeded your rate limit of 100 requests per minute",
"docs": "https://unkey.dev/docs/errors/ratelimit/limits/exceeded",
"requestId": "req_1234567890"
}
}
```
## 6. Benefits
### 6.1 Technical Benefits
* **Consistency**: Uniform identification across all services
* **Self-documenting**: Resource identifiers and error codes are descriptive and hierarchical
* **Future-proof**: Structure scales to accommodate new services and resource types
* **Improved Debugging**: Detailed error information speeds troubleshooting
* **Better Logging**: Structured identifiers improve log searchability and correlation
### 6.2 Developer Experience Benefits
* **Intuitive Navigation**: Hierarchical structure makes relationships clear
* **Better Documentation**: Documentation can be organized to match URN structure
* **Error Actionability**: Structured errors guide developers to solutions
* **Consistent Patterns**: Same patterns work across all Unkey services
* **Familiar Model**: Developers familiar with similar systems will recognize the approach
## 7. Migration Considerations
### 7.1 Database Impact
No schema changes are required to existing resources. URNs will be computed dynamically based on existing IDs, services, workspaces, and environments.
### 7.2 API Impact
All V2 APIs will need to:
* Include URNs in responses
* Support the new error code format
* Be thoroughly documented with examples
### 7.3 Documentation Impact
Documentation will need to be reorganized to:
* Explain the URN and error code systems
* Provide examples of both systems in use
* Possibly reorganize API reference documentation to align with the URN hierarchy
## 8. Conclusion
Implementing standardized URNs and error codes across Unkey's growing product surface will significantly improve both our internal systems and developer experience. These systems provide a foundation for scale as we expand our product offerings while maintaining a consistent, intuitive interface for developers.
This approach balances immediate needs with long-term scalability, ensuring we can grow our platform without introducing inconsistencies or complexity for developers.
## Appendix A: Common Error Codes
Below is an initial set of error codes for key services. This list will expand as new services and features are added.
### Authentication Service (`auth`)
* `err:auth:credentials:invalid_key` - API key is invalid or malformed
* `err:auth:credentials:missing_key` - Required API key is not provided
* `err:auth:permissions:insufficient_scope` - Key lacks required permissions
* `err:auth:state:key_disabled` - API key exists but is disabled
* `err:auth:state:key_not_found` - Referenced key doesn't exist
* `err:auth:state:already_exists` - Resource already exists (conflict)
### Rate Limiting Service (`ratelimit`)
* `err:ratelimit:limits:exceeded` - Rate limit has been exceeded
* `err:ratelimit:limits:quota_exceeded` - Monthly quota has been exceeded
* `err:ratelimit:configuration:invalid_limit` - Invalid rate limit configuration
* `err:ratelimit:state:namespace_not_found` - Rate limit namespace doesn't exist
### Identity Service (`identity`)
* `err:identity:validation:invalid_external_id` - External ID format is invalid
* `err:identity:state:disabled` - Identity is disabled
* `err:identity:state:not_found` - Identity doesn't exist
* `err:identity:state:already_exists` - Identity already exists with this external ID
## Appendix B: Related Work
This proposal draws inspiration from several established systems:
* RFC 8141 (URN Syntax)
* Amazon Web Services ARNs
* Azure Resource IDs
* Google Cloud Resource Names
# 0012 Stricter Linter
Source: https://engineering.unkey.com/architecture/rfcs/0012-stricter-linter
Adding more strict lint rules to minimize issues in our codebase.
## Summary
Our current linter rules are a bit loose, allowing us to make mistakes like using `as any`, non-null assertions (`shouldntBeNull!`), and redundant conditionals (`true ? true : false`).
Although we are careful when reviewing PRs, these things can still slip through. Therefore, we need to introduce a few more rules to make our configuration stricter.
## Motivation
To improve code predictability and maintainability using stricter, automated lint rules. This will also make PR reviews more efficient by automatically catching issues like `as any`, saving developers from repeatedly providing the same feedback manually.
## Solution
To achieve the goals, we need some new rules. Warnings will highlight potential issues without blocking commits initially, allowing for gradual adoption.
```json theme={"theme":"kanagawa-wave"}
"linter": {
"enabled": true,
"rules": {
"recommended": true,
"a11y": {
"noSvgWithoutTitle": "off",
"useSemanticElements": "warn",
"useFocusableInteractive": "warn"
},
"correctness": {
"noUnusedVariables": "error",
"useExhaustiveDependencies": "warn",
"noUnusedImports": "warn",
"noChildrenProp": "off",
"useJsxKeyInIterable": "warn",
"noUnsafeOptionalChaining": "warn"
},
"security": {
"noDangerouslySetInnerHtml": "warn" // -> Changed from "off" to "warn"
},
"style": {
"useConst": "warn",
"useBlockStatements": "error", // -> "Old"
"noNonNullAssertion": "warn",
"noUselessElse": "warn",
"useImportType": "warn",
"useFragmentSyntax": "warn",
"useDefaultSwitchClause": "warn",
"useAsConstAssertion": "warn",
"useTemplate": "warn",
"useNamingConvention": "warn",
"noYodaExpression": "warn",
"noUnusedTemplateLiteral": "warn",
"noNegationElse": "warn",
"useSelfClosingElements": "warn",
"useShorthandAssign": "warn"
},
"performance": {
"noDelete": "off" // -> "Old"
},
"suspicious": {
"recommended": true,
"noDoubleEquals": "warn",
"useIsArray": "warn",
"useAwait": "warn",
"noFallthroughSwitchClause": "warn",
"noExplicitAny": "warn",
"noConsoleLog": "warn"
},
"complexity": {
"noForEach": "off", // -> "Old"
"noUselessTernary": "warn",
"noUselessTypeConstraint": "warn",
"useSimplifiedLogicExpression": "warn",
"noUselessStringConcat": "warn",
"useOptionalChain": "warn",
"useDateNow": "warn",
"noExtraBooleanCast": "warn"
}
}
}
```
### Accessibility Rules (Making Web Content Usable for Everyone)
* **`useSemanticElements`** (`warn`)
* **Why:** Enforces the use of HTML elements that convey meaning (like ``, ``, ``) over generic `` or `
` elements for accessibility and SEO benefits. Screen readers and other assistive technologies rely on semantic markup.
* **Problematic Example:**
```jsx theme={"theme":"kanagawa-wave"}
Click Me
// Is it a button? A link?
```
* **Preferred:**
```jsx theme={"theme":"kanagawa-wave"}
Click Me // Clear semantic meaning
// or if navigating:
Click Me
```
* **Docs:** [Biome: useSemanticElements](https://biomejs.dev/linter/rules/use-semantic-elements/)
* **`useFocusableInteractive`** (`warn`)
* **Why:** Ensures that interactive elements that are focusable (can receive keyboard focus) are accessible via keyboard navigation. Elements with click handlers should generally be focusable or contained within focusable elements.
* **Problematic Example:**
```jsx theme={"theme":"kanagawa-wave"}
Action
// Interactive but not easily keyboard accessible
```
* **Preferred:** Use inherently focusable elements like `` or ensure custom interactive elements have appropriate `tabIndex` (usually `0`) and ARIA roles if needed.
```jsx theme={"theme":"kanagawa-wave"}
Action
// Or for custom elements (simplified):
Action
```
* **Docs:** [Biome: useFocusableInteractive](https://biomejs.dev/linter/rules/use-focusable-interactive/)
### Correctness Rules (Avoiding Errors & Improving Reliability)
* **`noUnusedVariables`** (`error`)
* **Why:** Prevents declaring variables that are never used, which clutters the code and can sometimes indicate incomplete logic or typos.
* **Problematic Example:**
```typescript theme={"theme":"kanagawa-wave"}
function greet(name: string) {
const message = "Hello"; // Unused variable
return `Hi ${name}`;
}
```
* **Preferred:** Remove the unused variable.
```typescript theme={"theme":"kanagawa-wave"}
function greet(name: string) {
return `Hi ${name}`;
}
```
* **Docs:** [Biome: noUnusedVariables](https://biomejs.dev/linter/rules/no-unused-variables/)
* **`useExhaustiveDependencies`** (`warn`)
* **Why:** Specifically for React's Hooks (`useEffect`, `useCallback`, etc.), this rule checks that all variables from the surrounding scope used inside the hook are included in the dependency array. Missing dependencies can lead to stale closures and unexpected behavior.
* **Problematic Example:**
```jsx theme={"theme":"kanagawa-wave"}
function Counter({ step }) {
const [count, setCount] = useState(0);
useEffect(() => {
const id = setInterval(() => {
setCount(c => c + step); // Uses 'step' but not listed below
}, 1000);
return () => clearInterval(id);
}, []); // Missing 'step'
}
```
* **Preferred:** Include all dependencies identified by the linter.
```jsx theme={"theme":"kanagawa-wave"}
// ...
useEffect(() => {
const id = setInterval(() => {
setCount(c => c + step);
}, 1000);
return () => clearInterval(id);
}, [step]); // Added 'step'
// ...
```
* **Docs:** [Biome: useExhaustiveDependencies](https://biomejs.dev/linter/rules/use-exhaustive-dependencies/)
* **`noUnusedImports`** (`warn`)
* **Why:** Flags imported variables, types, or modules that are not used anywhere in the file. This keeps the import list clean and avoids unnecessary dependencies.
* **Problematic Example:**
```typescript theme={"theme":"kanagawa-wave"}
import { useState, useEffect } from 'react'; // useEffect is not used
function MyComponent() {
const [value, setValue] = useState(0);
return {value}
;
}
```
* **Preferred:** Remove the unused import.
```typescript theme={"theme":"kanagawa-wave"}
import { useState } from 'react'; // Removed useEffect
// ...
```
* **Docs:** [Biome: noUnusedImports](https://biomejs.dev/linter/rules/no-unused-imports/)
* **`useJsxKeyInIterable`** (`warn`)
* **Why:** Requires a unique `key` prop when rendering lists of elements in JSX using iterators like `map`. React uses these keys to efficiently update the list and maintain component state.
* **Problematic Example:**
```jsx theme={"theme":"kanagawa-wave"}
{items.map(item => {item.name} )} {/* Missing key prop */}
```
* **Preferred:** Add a unique and stable `key` to the outermost element returned by the map.
```jsx theme={"theme":"kanagawa-wave"}
{items.map(item => {item.name} )} {/* Added key */}
```
* **Docs:** [Biome: useJsxKeyInIterable](https://biomejs.dev/linter/rules/use-jsx-key-in-iterable/)
* **`noUnsafeOptionalChaining`** (`warn`)
* **Why:** Prevents optional chaining (`?.`) in contexts where it doesn't provide safety, such as arithmetic operations or assignments, where a `null` or `undefined` result would likely cause a runtime error anyway.
* **Problematic Example:**
```typescript theme={"theme":"kanagawa-wave"}
const length = data?.array?.length * 2; // TypeError if data?.array is null/undefined
let count = 0;
count += obj?.value; // TypeError if obj?.value is null/undefined
```
* **Preferred:** Use nullish coalescing (`??`) or explicit checks to handle potential `null`/`undefined` before the operation.
```typescript theme={"theme":"kanagawa-wave"}
const length = (data?.array?.length ?? 0) * 2;
let count = 0;
count += obj?.value ?? 0;
```
* **Docs:** [Biome: noUnsafeOptionalChaining](https://biomejs.dev/linter/rules/no-unsafe-optional-chaining/)
### Security Rules
* **`noDangerouslySetInnerHtml`** (`warn`)
* **Why:** Prevents the use of `dangerouslySetInnerHTML` in JSX, which can expose your application to cross-site scripting (XSS) attacks if the injected HTML comes from user input.
* **Problematic Example:**
```jsx theme={"theme":"kanagawa-wave"}
```
* **Preferred:** Use safer methods to render dynamic content, like directly rendering text or using libraries that sanitize HTML.
* **Docs:** [Biome: noDangerouslySetInnerHtml](https://biomejs.dev/linter/rules/no-dangerously-set-inner-html/)
### Style Rules (Code Consistency & Readability)
* **`useConst`** (`warn`)
* **Why:** Encourages using `const` for variables that are never reassigned after their initial declaration. This improves readability by signaling the variable's immutability.
* **Problematic Example:**
```javascript theme={"theme":"kanagawa-wave"}
let userId = fetchUserId(); // userId is never reassigned
console.log(userId);
```
* **Preferred:**
```javascript theme={"theme":"kanagawa-wave"}
const userId = fetchUserId();
console.log(userId);
```
* **Docs:** [Biome: useConst](https://biomejs.dev/linter/rules/use-const/)
* **`noNonNullAssertion`** (`warn`)
* **Why:** Discourages the use of the non-null assertion operator (`!`), which tells TypeScript a value is not `null` or `undefined` without actual checks. Overuse can hide potential runtime errors. This relates to the `shouldntBeNull!` example mentioned earlier.
* **Problematic Example:**
```typescript theme={"theme":"kanagawa-wave"}
const name = user!.name; // Assumes user is not null/undefined
```
* **Preferred:** Use type guards, default values, or optional chaining (`user?.name`).
* **Docs:** [Biome: noNonNullAssertion](https://biomejs.dev/linter/rules/no-non-null-assertion/)
* **`noUselessElse`** (`warn`)
* **Why:** Prevents `else` blocks when the `if` block contains a `return`, `throw`, `continue`, or `break` statement, making the code less nested and easier to read.
* **Problematic Example:**
```typescript theme={"theme":"kanagawa-wave"}
if (condition) { return value1; } else { return value2; }
```
* **Preferred:**
```typescript theme={"theme":"kanagawa-wave"}
if (condition) { return value1; } return value2;
```
* **Docs:** [Biome: noUselessElse](https://biomejs.dev/linter/rules/no-useless-else/)
* **`useImportType`** (`warn`)
* **Why:** Encourages using `import type` for importing only types. This clearly signals intent and can sometimes help build tools optimize imports.
* **Problematic Example:**
```typescript theme={"theme":"kanagawa-wave"}
import { User, getUser } from "./user";
```
* **Preferred:**
```typescript theme={"theme":"kanagawa-wave"}
import type { User } from "./user";
import { getUser } from "./user";
```
* **Docs:** [Biome: useImportType](https://biomejs.dev/linter/rules/use-import-type/)
* **`useFragmentSyntax`** (`warn`)
* **Why:** Promotes the shorter `<>` syntax for React Fragments over ``.
* **Problematic Example:**
```jsx theme={"theme":"kanagawa-wave"}
... ...
```
* **Docs:** [Biome: useFragmentSyntax](https://biomejs.dev/linter/rules/use-fragment-syntax/)
* **`useDefaultSwitchClause`** (`warn`)
* **Why:** Enforces that `switch` statements have a `default` case, preventing potential errors if an unexpected value is encountered.
* **Problematic Example:**
```typescript theme={"theme":"kanagawa-wave"}
switch (status) { case "PENDING": handlePending(); break; }
```
* **Preferred:**
```typescript theme={"theme":"kanagawa-wave"}
switch (status) {
case "PENDING":
handlePending();
break;
default:
handleDefault(); // Or throw error
break;
}
```
* **Docs:** [Biome: useDefaultSwitchClause](https://biomejs.dev/linter/rules/use-default-switch-clause/)
* **`useAsConstAssertion`** (`warn`)
* **Why:** Suggests using `as const` assertions for object and array literals when you want their properties/elements to be treated as specific literal types rather than general types (e.g., `string` instead of `"ACTIVE"`).
* **Problematic Example:**
```typescript theme={"theme":"kanagawa-wave"}
const config = { theme: 'dark', version: 1 }; // Type: { theme: string, version: number }
```
* **Preferred:**
```typescript theme={"theme":"kanagawa-wave"}
const config = { theme: 'dark', version: 1 } as const; // Type: { readonly theme: 'dark', readonly version: 1 }
```
* **Docs:** [Biome: useAsConstAssertion](https://biomejs.dev/linter/rules/use-as-const-assertion/)
* **`useTemplate`** (`warn`)
* **Why:** Prefers template literals (backticks `` ` ``) over string concatenation (`+`) for readability when embedding expressions.
* **Problematic Example:**
```javascript theme={"theme":"kanagawa-wave"}
const message = 'User ' + userId + ' logged in.';
```
* **Preferred:**
```javascript theme={"theme":"kanagawa-wave"}
const message = `User ${userId} logged in.`;
```
* **Docs:** [Biome: useTemplate](https://biomejs.dev/linter/rules/use-template/)
* **`useNamingConvention`** (`warn`)
* **Why:** Enforces consistent naming conventions (e.g., camelCase for variables, PascalCase for classes/types) across the codebase, improving readability. Configuration might be needed to match team style.
* **Problematic Example (depends on config):**
```typescript theme={"theme":"kanagawa-wave"}
const user_id = 1; class my_class {}
```
* **Preferred (typical JS/TS):**
```typescript theme={"theme":"kanagawa-wave"}
const userId = 1; class MyClass {}
```
* **Docs:** [Biome: useNamingConvention](https://biomejs.dev/linter/rules/use-naming-convention/)
* **`noYodaExpression`** (`warn`)
* **Why:** Prevents "Yoda" conditions where the literal/constant comes before the variable (e.g., `if (5 === count)`). These can be less intuitive to read.
* **Problematic Example:**
```javascript theme={"theme":"kanagawa-wave"}
if (null == value) { /* ... */ }
```
* **Preferred:**
```javascript theme={"theme":"kanagawa-wave"}
if (value == null) { /* ... */ }
```
* **Docs:** [Biome: noYodaExpression](https://biomejs.dev/linter/rules/no-yoda-expression/)
* **`noUnusedTemplateLiteral`** (`warn`)
* **Why:** Flags template literals that don't contain any expressions, as regular string literals are simpler.
* **Problematic Example:**
```javascript theme={"theme":"kanagawa-wave"}
const greeting = `Hello World`;
```
* **Preferred:**
```javascript theme={"theme":"kanagawa-wave"}
const greeting = 'Hello World';
```
* **Docs:** [Biome: noUnusedTemplateLiteral](https://biomejs.dev/linter/rules/no-unused-template-literal/)
* **`noNegationElse`** (`warn`)
* **Why:** Suggests refactoring `if`/`else` statements where the `if` condition is negated, often improving readability by handling the positive case first.
* **Problematic Example:**
```javascript theme={"theme":"kanagawa-wave"}
if (!isActive) { handleInactive(); } else { handleActive(); }
```
* **Preferred:**
```javascript theme={"theme":"kanagawa-wave"}
if (isActive) { handleActive(); } else { handleInactive(); }
```
* **Docs:** [Biome: noNegationElse](https://biomejs.dev/linter/rules/no-negation-else/)
* **`useSelfClosingElements`** (`warn`)
* **Why:** Requires using self-closing tags for JSX elements with no children.
* **Problematic Example:**
```jsx theme={"theme":"kanagawa-wave"}
`).
* **Problematic Example:**
```javascript theme={"theme":"kanagawa-wave"}
switch(val) { case 1: doOne(); case 2: doTwo(); } // Falls through from 1 to 2
```
* **Preferred:**
```javascript theme={"theme":"kanagawa-wave"}
switch(val) { case 1: doOne(); break; case 2: doTwo(); break; }
```
* **Docs:** [Biome: noFallthroughSwitchClause](https://biomejs.dev/linter/rules/no-fallthrough-switch-clause/)
* **`noExplicitAny`** (`warn`)
* **Why:** Discourages the explicit use of `any` as a type, as it effectively disables TypeScript's type checking for that variable. This relates to the `as any` example mentioned earlier.
* **Problematic Example:**
```typescript theme={"theme":"kanagawa-wave"}
let response: any;
```
* **Preferred:** Use specific types, generics, or `unknown` (which requires type checking before use).
* **Docs:** [Biome: noExplicitAny](https://biomejs.dev/linter/rules/no-explicit-any/)
* **`noConsoleLog`** (`warn`)
* **Why:** Flags `console.log` (and other `console` methods) to prevent debug statements from being accidentally committed to production code. Consider using a dedicated logger or removing logs before merging.
* **Problematic Example:**
```javascript theme={"theme":"kanagawa-wave"}
console.log("Debugging value:", data);
```
* **Preferred:** Remove the log or use a proper logging library.
* **Docs:** [Biome: noConsoleLog](https://biomejs.dev/linter/rules/no-console-log/)
### Complexity Rules (Simplifying Code)
* **`noUselessTernary`** (`warn`)
* **Why:** Prevents ternary operators that directly return boolean literals (`true`/`false`) based on a condition, as the condition itself can be used. This relates to the `true ? true : false` example.
* **Problematic Example:**
```javascript theme={"theme":"kanagawa-wave"}
const isEnabled = value > 10 ? true : false;
```
* **Preferred:**
```javascript theme={"theme":"kanagawa-wave"}
const isEnabled = value > 10;
```
* **Docs:** [Biome: noUselessTernary](https://biomejs.dev/linter/rules/no-useless-ternary/)
* **`noUselessTypeConstraint`** (`warn`)
* **Why:** Flags redundant type constraints in generics like `T extends any` or `T extends unknown`, which provide no additional limitation.
* **Problematic Example:**
```typescript theme={"theme":"kanagawa-wave"}
function process(data: T): T { /* ... */ }
```
* **Preferred:**
```typescript theme={"theme":"kanagawa-wave"}
function process(data: T): T { /* ... */ }
```
* **Docs:** [Biome: noUselessTypeConstraint](https://biomejs.dev/linter/rules/no-useless-type-constraint/)
* **`useSimplifiedLogicExpression`** (`warn`)
* **Why:** Encourages simplifying boolean expressions, such as removing double negations (`!!`).
* **Problematic Example:**
```javascript theme={"theme":"kanagawa-wave"}
const hasAccess = !!user.permissions;
```
* **Preferred:**
```javascript theme={"theme":"kanagawa-wave"}
const hasAccess = Boolean(user.permissions); // Or specific check
```
* **Docs:** [Biome: useSimplifiedLogicExpression](https://biomejs.dev/linter/rules/use-simplified-logic-expression/)
* **`noUselessStringConcat`** (`warn`)
* **Why:** Prevents concatenating two string literals, which should just be combined into a single literal.
* **Problematic Example:**
```javascript theme={"theme":"kanagawa-wave"}
const path = '/api' + '/users';
```
* **Preferred:**
```javascript theme={"theme":"kanagawa-wave"}
const path = '/api/users';
```
* **Docs:** [Biome: noUselessStringConcat](https://biomejs.dev/linter/rules/no-useless-string-concat/)
* **`useOptionalChain`** (`warn`)
* **Why:** Promotes using the optional chaining operator (`?.`) instead of longer logical AND (`&&`) chains for accessing nested properties safely.
* **Problematic Example:**
```javascript theme={"theme":"kanagawa-wave"}
const street = user && user.address && user.address.street;
```
* **Preferred:**
```javascript theme={"theme":"kanagawa-wave"}
const street = user?.address?.street;
```
* **Docs:** [Biome: useOptionalChain](https://biomejs.dev/linter/rules/use-optional-chain/)
* **`useDateNow`** (`warn`)
* **Why:** Suggests using `Date.now()` which is slightly more performant and concise than `new Date().getTime()` or `+new Date()` for getting a timestamp.
* **Problematic Example:**
```javascript theme={"theme":"kanagawa-wave"}
const timestamp = new Date().getTime();
```
* **Preferred:**
```javascript theme={"theme":"kanagawa-wave"}
const timestamp = Date.now();
```
* **Docs:** [Biome: useDateNow](https://biomejs.dev/linter/rules/use-date-now/)
* **`noExtraBooleanCast`** (`warn`)
* **Why:** Prevents unnecessary boolean casts (using `Boolean()` or `!!`) in contexts where the value is already treated as a boolean (like `if` statements or logical operators).
* **Problematic Example:**
```javascript theme={"theme":"kanagawa-wave"}
if (!!isValid) { /* ... */ }
```
* **Preferred:**
```javascript theme={"theme":"kanagawa-wave"}
if (isValid) { /* ... */ }
```
* **Docs:** [Biome: noExtraBooleanCast](https://biomejs.dev/linter/rules/no-extra-boolean-cast/)
***
By enabling these rules, we aim to catch more potential errors and enforce stylistic consistency automatically.
# 0013 TLS Certificates for custom domains
Source: https://engineering.unkey.com/architecture/rfcs/0013-custom-domains
Issuing certificates for custom domains using Let's Encrypt's HTTP-01 challenge.
## What is Let's Encrypt?
Let's Encrypt is a free, automated Certificate Authority (CA) that provides SSL certificates to enable HTTPS on websites. Unlike traditional CAs that require manual processes and fees, Let's Encrypt uses an automated protocol called ACME (Automatic Certificate Management Environment) to verify domain ownership and issue certificates programmatically.
## Why Do We Need Domain Validation?
Before issuing an SSL certificate, Let's Encrypt must verify that you actually control the domain you're requesting a certificate for. This prevents malicious actors from getting certificates for domains they don't own. Let's Encrypt offers several challenge types to prove domain control - we use the HTTP-01 challenge.
## How HTTP-01 Challenge Works
The HTTP-01 challenge is simple but effective:
1. **You request a certificate** for `example.com`
2. **Let's Encrypt gives you a unique token** like `abc123`
3. **You must serve a specific response** at `http://example.com/.well-known/acme-challenge/abc123`
4. **Let's Encrypt checks that URL** and verifies you can control what's served there
5. **If verification succeeds**, Let's Encrypt issues your certificate
This proves you control the domain because only someone with access to the web server can serve content at that specific path.
## Our Architecture Approach
We chose HTTP-01 challenges because they provide the best user experience. Customers only need to add a single CNAME record pointing their domain to us. We handle all the certificate complexity behind the scenes.
### System Components
Our SSL certificate management system has several key components:
**Ctrl Service (Control Plane)**
* Communicates with Let's Encrypt using the ACME protocol
* Generates private keys and certificate signing requests
* Manages certificate lifecycle (issuance and renewal)
* Runs a validation server to respond to HTTP-01 challenges
**Frontline (Dataplane)**
* Handle incoming HTTPS traffic using certificates
* Redirect challenge requests to the control plane service validation server
* Completely independent of control plane for serving user traffic
**Database**
* Stores custom domains, certificate request tracking, challenge state, certificates and encrypted private keys for runtime use
**Vault**
* Encrypts all private keys before database storage
* Provides secure key management and rotation
### Why This Architecture?
This separation ensures that customer HTTPS traffic continues working even if our control plane has issues. Only new certificate requests are affected by control plane outages - existing traffic keeps flowing normally.
## Database Design
The database design maintains strict separation between control plane operations and dataplane runtime requirements.
**Database (`unkey`):**
* `custom_domains` - Stores user-owned custom domains
* `frontline_routes` - Maps hostnames to deployments with sticky behavior
* `certificate_requests` - Tracks ACME workflow state and metadata for certificate provisioning and renewal
* `challenges` - Stores ACME challenge responses and metadata
* `certificates` - Production certificates and encrypted private keys for Frontline TLS termination
The control plane service uses the database for certificate request lifecycle management and ACME protocol state tracking, including storing challenge responses. Frontline instances query the same database for certificates during TLS handshakes and cache them in memory.
## Certificate Provisioning
Certificate provisioning involves coordination between multiple services and databases to complete the Let's Encrypt ACME workflow. The process begins with user requests and results in certificates available for Frontline TLS termination.
### Certificate Request Process
When we need to get a certificate for a domain, here's what happens step by step:
#### 1. Initial Setup
* Our control plane service generates a private key and certificate signing request (CSR)
* This happens entirely within our system - no external communication yet
#### 2. Starting the ACME Process
* We send a "new order" request to Let's Encrypt
* Let's Encrypt responds with challenge instructions and URLs we need later to complete the process
#### 3. Challenge Preparation
* We fetch the challenge details (like the token we need to serve)
* We generate and store the challenge response in our control database
* We tell Let's Encrypt we're ready to begin validation
#### 4. Domain Validation
* Let's Encrypt makes a request to `http://yourdomain.com/.well-known/acme-challenge/token`
* Frontline redirects this request to our control plane service
* Our control plane service looks up the challenge response from our control database
* Our control plane service responds with the correct challenge response
* Let's Encrypt verifies the response matches what they expect
#### 5. Certificate Issuance
* Once validation succeeds, we send our CSR to Let's Encrypt
* Let's Encrypt processes the request and generates the certificate
* We download the finished certificate
#### 6. Secure Storage
* We encrypt the private key using Vault
* We store both the certificate and encrypted private key in our dataplane database
* Frontline instances can now use this certificate for HTTPS traffic
",
"type": "API_KEY",
"source": {
"key": {
"keyId": "",
"keySpaceId": "",
"meta": {}
}
}
}
```
Example: a request authenticated with an Unkey API key that has an identity. The identity's external ID becomes the subject, and identity detail appears alongside the key source:
```json theme={"theme":"kanagawa-wave"}
{
"version": "v1",
"subject": "",
"type": "API_KEY",
"identity": {
"externalId": "",
"meta": {}
},
"source": {
"key": {
"keyId": "",
"keySpaceId": "",
"meta": {}
}
}
}
```
For local development without Frontline, developers can set the header manually (`-H 'X-Unkey-Principal: {"version":"v1","subject":"test","type":"API_KEY","source":{"key":{"keyId":"key_test","keySpaceId":"ks_test","meta":{}}}}'`) or omit it entirely for anonymous behavior. This requires no key management or token generation.
## Request Evaluation
Frontline registers a single catch-all route. When a request arrives:
1. Parse the deployment's `frontline.v1.Config` into a `repeated Policy` list.
2. For each policy, in list order:
* Skip if `enabled == false`.
* Evaluate the `repeated MatchExpr` against the request. Skip if any condition doesn't match.
* Execute the policy. It can short-circuit (reject) or continue to the next policy.
3. If all matching policies pass, forward the request to the upstream.
**List order is execution order.** The field numbers in the `oneof config` have no effect on runtime behavior.
The operator has full control over execution order. Authn policies should come before policies that need a Principal. But these are conventions, not constraints — the engine doesn't enforce them.
## Error Responses
When a policy rejects a request, Frontline returns a fixed JSON response using the same RFC 7807 Problem Details format as the Unkey API (see [`svc/api/openapi/spec/error/BaseError.yaml`](https://github.com/unkeyed/unkey/blob/main/svc/api/openapi/spec/error/BaseError.yaml)). The response body is not configurable. Every rejection uses the same structure:
```json theme={"theme":"kanagawa-wave"}
{
"meta": { "requestId": "req_abc123" },
"error": {
"title": "Unauthorized",
"detail": "API key is invalid or expired",
"status": 401,
"type": "https://unkey.com/docs/errors/frontline/unauthorized"
}
}
```
Each implemented policy maps to a standard HTTP status code: KeyAuth → 401 for missing or invalid credentials, 403 for insufficient permissions (`permission_query`), RateLimit → 429, Firewall → 403, and OpenAPI validation → 400. The `detail` field provides a human-readable explanation specific to the rejection reason. The `type` URI is stable per error kind and suitable for programmatic handling.
Custom error responses are not supported in this version. Status codes are what API clients branch on, and the RFC 7807 format is a widely supported standard. If customization becomes necessary, it can be added as a per-status-code template on `Config` without breaking existing behavior.
## Adding a New Policy Type
1. Create a new `.proto` file in [`svc/frontline/proto/frontline/policies/v1/`](https://github.com/unkeyed/unkey/blob/main/svc/frontline/proto/frontline/policies/v1/) with the policy's configuration message.
2. Import it in `policy.proto` and add a field to the `oneof config` block.
3. Implement the policy's execution logic under `svc/frontline/internal/policies/` and dispatch it from the policy engine.
4. If the policy is an authn method, it must produce a Principal. If it depends on authentication, it should read the Principal from context and reject if absent.
No changes to the match system, evaluation engine, or other policies are needed. This is the benefit of the Policy/MatchExpr/Principal separation — new policies compose with the existing system without modification.
## Schema Conventions
* **Durations as int64 milliseconds**: All time fields use `int64` milliseconds (e.g., `window_ms`, `clock_skew_ms`, `jwks_cache_ms`). No `google.protobuf.Duration` — consistent with the rest of the Unkey proto codebase.
* **Policy-internal filtering vs. MatchExpr**: Some policies have their own filtering fields that are not redundant with MatchExpr. MatchExpr controls whether the policy *runs*. Internal fields control the policy's *behavior* once running.
* **Client IP derivation**: Client-IP-dependent behavior, such as RateLimit with `RemoteIpKey`, uses the client IP on the request after Frontline's proxy middleware sanitizes forwarding headers.
## Proto Location
Proto directories: [`svc/frontline/proto/frontline/config/v1/`](https://github.com/unkeyed/unkey/blob/main/svc/frontline/proto/frontline/config/v1/) and [`svc/frontline/proto/frontline/policies/v1/`](https://github.com/unkeyed/unkey/blob/main/svc/frontline/proto/frontline/policies/v1/).
```
config.proto ← Config (ordered policy list)
policy.proto ← Policy (top-level container)
match.proto ← MatchExpr request matchers
keyauth.proto ← individual policy configs...
jwtauth.proto
ratelimit.proto
firewall.proto
openapi.proto
```
The Principal is not defined in proto — it is a hand-written Go struct in
`svc/frontline/internal/policies/principal/principal.go` serialized with `encoding/json`. The
shape is output-only (never crosses a proto wire) and `protojson` does
not produce the JSON contract we want.
Package: `frontline.v1`
Go import: `github.com/unkeyed/unkey/gen/proto/frontline/v1;frontlinev1`
# 0015 Ratelimit Cross-Region Counts
Source: https://engineering.unkey.com/architecture/rfcs/0015-ratelimit-cross-region-counts
Replace the ratelimit_blocklist propagation table with a G-Counter style sharing of actual per-region counts, eliminating the over-block failure modes of the current scheme.
## Summary
Replace the `ratelimit_blocklist`-driven propagation path with a new `ratelimit_window_counts` table that holds per-region observations of each active sliding-window counter. Regions periodically flush their own observed count, and periodically read the sum of other regions' counts to derive the cross-region effective count. The local denial decision becomes "would this request exceed the limit given my own count plus what other regions have reported," which is the question we actually wanted to answer all along.
The blocklist table itself stays for now (rows drain naturally over their sequence-derived expiry); the in-memory machinery that wrote and read it is gone. The table and its sqlc queries are scheduled for deletion in a follow-up PR once we have confidence the new path is healthy in production.
## Motivation
The blocklist propagates denials by writing one row per (workspace, namespace, identifier, duration, sequence) at the moment a region first denies, and inflating the matching local counter to `limit` in every other region on the next sync. That model has two structural failure modes that no amount of filter heuristics can eliminate.
The first is the cold oversized request. A user makes one request whose cost exceeds their entire limit, gets denied locally, and is now pinned at `limit` across every region for the remainder of the window even though they have consumed zero tokens. We currently filter this out by skipping propagation when `currentCount < req.Cost`, which works for the canonical case but leaves the broader pattern (a user denied while having consumed very little of their budget) on the wrong side of the rule.
The second is the prev-window bleed-in denial. A user uses 8 of 10 tokens in window N, then makes a small request at the start of window N+1. The sliding-window math denies on the prev contribution, and propagating that denial pins them at `limit` in window N+1 globally even though they have used nothing in this window. The current code skips propagation when prev is the dominant contributor, but this just trades one heuristic for another.
Both failures share the same root cause: the blocklist communicates a verdict ("blocked") rather than a quantity ("this region saw N requests"). Other regions can act on the verdict only by overriding their local count, which makes a punitive choice on incomplete information. Sharing actual counts removes the choice. The denial decision in each region uses the same sliding-window math it always has, just with a more accurate input.
A secondary motivation is that observed multi-region overlap (\~5% baseline, \~8% peak as of 2026-05-01) is small enough that the heuristic-driven blocklist works most of the time, but spikes into 25% denial bursts are exactly when the failure modes bite hardest, and exactly when over-blocking damages real users. Cleaner semantics during bursts is the user-facing payoff.
## Detailed design
### Data model
A new table replaces the blocklist. Each row records one region's observed count for one sliding-window cell.
```sql theme={"theme":"kanagawa-wave"}
CREATE TABLE `ratelimit_window_counts` (
`pk` bigint unsigned AUTO_INCREMENT NOT NULL,
`workspace_id` varchar(191) NOT NULL,
`namespace` varchar(255) NOT NULL,
`identifier` varchar(255) NOT NULL,
`duration_ms` bigint unsigned NOT NULL,
`sequence` bigint NOT NULL,
`region` varchar(48) NOT NULL,
`count` bigint unsigned NOT NULL,
`expires_at` bigint unsigned NOT NULL,
`updated_at` bigint unsigned NOT NULL,
CONSTRAINT `ratelimit_window_counts_pk` PRIMARY KEY (`pk`),
CONSTRAINT `unique_window_region` UNIQUE (
`workspace_id`, `namespace`, `identifier`, `duration_ms`, `sequence`, `region`
)
);
CREATE INDEX `expires_at_idx` ON `ratelimit_window_counts` (`expires_at`);
CREATE INDEX `lookup_idx` ON `ratelimit_window_counts` (
`workspace_id`, `namespace`, `identifier`, `duration_ms`, `sequence`
);
```
`region` is read from a new `Config.Region` field, populated from the `UNKEY_REGION` environment variable at boot. The column is `varchar(48)` rather than the more generous `varchar(64)` so the unique index (which spans the four key fields plus `sequence` and `region`) stays under MySQL's 3072-byte limit for utf8mb4. Real region tags fit comfortably.
Two regions with the same identifier produce two rows; aggregation is `SUM(count)` across regions. Within a region, multiple instances may write the same row, which is fine because the upsert collapses them via `GREATEST`. `expires_at` is sequence-derived: `(sequence + 2) * duration_ms`. The row is meaningful through window N+1 (where it appears as prev) and useless after, regardless of wall-clock drift between regions.
### Counter entry shape
`counterEntry` gains three atomic fields:
* `globalCount atomic.Int64` — sum of other regions' contributions for this cell, written by the sync goroutine, read by the request path.
* `limit atomic.Int64` — the most recent per-request limit observed on this entry, written by `prepareCheck` on every request. The flush goroutine compares `val` against `limit * floor` to decide whether the entry is worth propagating.
* `lastFlushed atomic.Int64` — the `val` written by the previous successful flush. The flush goroutine skips entries whose `val` has not grown beyond it, so quiet entries don't generate redundant MySQL writes.
The `blocked atomic.Bool`, the `maybePropagateDenial` function, and its gating heuristics are removed. The existing `val` continues to hold this region's own observed count, populated by traffic and the existing replay-from-Redis path.
The sliding-window check becomes:
```go theme={"theme":"kanagawa-wave"}
func (cs *checkState) slidingWindowCount(curCount int64) int64 {
cur := curCount + cs.curGlobal
prev := cs.prev.val.Load() + cs.prevGlobal
return cur + int64(float64(prev)*(1.0-cs.windowElapsed))
}
```
`cs.curGlobal` and `cs.prevGlobal` are snapshots, taken by `prepareCheck` once at the start of each request. The CAS retry loop in `Ratelimit` then re-evaluates `slidingWindowCount` per attempt without re-paying the atomic load on `globalCount` — it only mutates from the sync goroutine on a 10s cadence, so it is effectively constant for the lifetime of any one request. `prev.val` is still loaded fresh because it can move during the CAS loop via concurrent passing requests on the prev counter.
The CAS path keeps incrementing `cur.val` (own count) on each accepted request; nothing about Redis replay changes. `globalCount` is read-only from the request path's perspective.
Naming convention: "global" throughout this package means "across all other regions," not "across all nodes." Nodes within a region already converge through Redis replay; global state excludes own-region rows on read. The deny path reads as `cur.val + cur.globalCount` — local plus global, with the boundary defined by what the sync query filters out (`region != self`).
### Flush path
A periodic goroutine ticks every 10 seconds with 20% jitter and walks the local `counters` sync.Map. Eligible entries are collected into one slice and written to MySQL in a single bulk upsert per tick:
```sql theme={"theme":"kanagawa-wave"}
INSERT INTO ratelimit_window_counts (...)
VALUES (...)
ON DUPLICATE KEY UPDATE
count = GREATEST(count, VALUES(count)),
updated_at = VALUES(updated_at);
```
`GREATEST` makes the write idempotent and monotonic per region. Two instances within the same region writing concurrently always agree on the result, even under arbitrary interleaving. A row whose remote count happens to be ahead of ours stays ahead (which can happen briefly when our flush races a concurrent flush from another instance whose Redis-merged view is fresher).
There is no intermediate buffer. Earlier drafts pushed each row through `pkg/batch` to coalesce writes, but the periodic walk already produces a complete batch in one pass — the buffer just added a 1-second of latency and silent-drop semantics on overflow that are inappropriate for a counter-sharing system. The bulk upsert is wrapped directly in `crossRegionCircuitBreaker` so a sick database fails fast rather than blocking the next tick.
Two filters gate which entries actually flush:
* **Utilization filter** (`val < limit * 0.5`) skips entries where this region has consumed less than half the limit. Such entries cannot meaningfully push another region over its threshold, so propagating their count is wasted MySQL load. This filter runs first because most active windows never cross the floor; checking it before the change filter avoids the second atomic load on the bulk of skipped entries.
* **Change filter** (`val == lastFlushed`) skips entries whose `val` has not moved since the previous successful flush. Most active windows tick once per request and are idle between flushes; without this we'd re-write unchanged rows every cycle.
`lastFlushed` only commits after the bulk upsert succeeds. A transient MySQL failure leaves the entries in a state that re-emits on the next tick. Without that ordering, a dropped batch would silently mark its rows as flushed and not retry until `val` changed again.
The 20% jitter on the tick cadence prevents fleet-wide lockstep — without it, every region's flush goroutine would converge on the same wall-clock multiple of 10s and hammer MySQL in a convoy. Jitter applies fresh on every cycle, anchored to absolute target times so a slow flush does not drift the cadence.
With these filters, the realistic write rate is dominated by the rate at which entries cross the 50% threshold, not by the active window count. Hot identifiers cross once and then write at the cadence of their growth (one or two writes per flush interval until the window rotates); cold identifiers never write at all.
### Sync path
Every 10 seconds (with 20% jitter), each region pulls the per-key sum of every other region's contribution:
```sql theme={"theme":"kanagawa-wave"}
SELECT
workspace_id, namespace, identifier, duration_ms, sequence,
CAST(SUM(count) AS SIGNED) AS imported
FROM ratelimit_window_counts
WHERE expires_at > ?
AND region != ?
GROUP BY workspace_id, namespace, identifier, duration_ms, sequence;
```
Aggregation runs in MySQL because the application only ever uses the sum. Returning per-region rows just to collapse them in Go would waste bandwidth and memory; with `GROUP BY` the receiver gets one row per active window cell instead of one row per (region, cell) pair. `CAST(SUM(count) AS SIGNED)` so sqlc maps the aggregate column to `int64`, matching `atomic.Int64` on the receiver.
No additional filter is needed on the read side because the write-side utilization filter already excludes low-count rows from the table. Every row that appears in the result is, by construction, from a region that has crossed 50% utilization on this entry, which is exactly the population worth syncing.
The receiver writes each row's aggregate directly into the matching `counterEntry.globalCount` via `atomicMax`. Sums are monotonic per cell (each region's contribution only grows within a sequence), so atomicMax is sufficient and idempotent across overlapping ticks. When no local entry exists for a key seen in the result set, one is created on demand via `findOrCreateCounter`. These creations are attributed to `RatelimitGlobalEntriesCreated` rather than the traffic-driven `RatelimitWindowsCreated` so the cardinality signal stays clean.
### What goes away
The `blocked atomic.Bool` on `counterEntry`, the propagation gating heuristics in `maybePropagateDenial`, and `maybePropagateDenial` itself become unnecessary and are removed. Each addressed a symptom of the blocklist's verdict-shaped propagation:
* `blocked` deduped propagation events. With G-Counter writes idempotent under `GREATEST`, no dedup is needed.
* The `currentCount >= req.Cost` and `minPropagationDuration` filters guarded against punitive over-blocking. With actual counts shared, there is no punitive action to guard against.
### What stays
Strict mode (`strictUntils` map, `setStrictUntil`, `loadStrictUntil`, the forced origin fetch in `prepareCheck`) is kept. Strict mode is the in-region convergence mechanism: instances within a region share state through Redis, and the post-denial forced fetch drains any lag between an instance's local view and the region's Redis-backed truth. That role is independent of the cross-region path. The new `globalCount` field handles convergence across regions; strict mode handles convergence between instances of the same region. They coexist cleanly: `effectiveCount = cur.val + cur.globalCount + (prev.val + prev.globalCount) * (1 - elapsed)`, with strict-mode fetches updating `cur.val` and `prev.val` on the request path before the read.
### Cleanup
A `WindowCountsDeleteExpired` query (sqlc) deletes rows where `expires_at < cutoff`. It is intended to be driven by an external Restate cron, mirroring the existing `BlocklistDeleteExpired` arrangement; the ratelimit service itself does not run a cleanup goroutine.
The existing `pkg/mysql/schema/ratelimit_blocklist.sql` and its sqlc queries (`BulkInsertBlocklist`, `BlocklistListActive`, `BlocklistDeleteExpired`) are intentionally untouched in this PR. The in-memory machinery that wrote and read them is gone, so the table is no longer being populated; existing rows drain naturally over their sequence-derived expiry. A follow-up PR removes the table and queries once the new path has been observed in production.
### Configuration changes
`Config` gains a single new field: `Region string` (required), sourced from `UNKEY_REGION` at process start. Used as the row-key partition for own writes and the filter for own-region reads. The constructor returns `ErrRegionRequired` when empty.
Tuning parameters are package-level constants, not Config fields:
* `globalFlushInterval = 10 * time.Second`
* `globalSyncInterval = 10 * time.Second`
* `globalUtilizationFloor = 0.5`
* `globalSyncJitter = 0.2`
* `globalFlushTimeout = 10 * time.Second`
Trading propagation coverage against MySQL write rate is a global property of the system; exposing it as a per-instance knob would only create drift between regions running the same code.
### Metrics
The blocklist metrics (`unkey_ratelimit_blocklist_*`) are removed since the in-memory blocklist machinery is gone. New `unkey_ratelimit_global_*` metrics with the same shape replace them: `writes_total`, `write_errors_total`, `sync_rows_applied_total`, `sync_errors_total`, `entries_created_total`, `rows_last_poll`. The dashboard shape is preserved so the operator experience is continuous after a panel rename.
`RatelimitStrictModeActivations` is kept since strict mode itself is kept.
## Expected MySQL load
The numbers below use observed production traffic as of 2026-05-01: 33 instances across 10 regions, \~24,000 active sliding-window entries fleet-wide at peak, \~5% multi-region overlap, and 2–4% baseline denial fraction (spiking to \~25% during bursts).
**Hot-window cardinality.** Most active windows are quiet — a user makes a handful of cost-1 calls and never approaches the limit. The 50% utilization filter means only entries that consume half their budget are written to MySQL. Empirically (denial fraction + headroom for windows that approach but don't cross limit), the hot subset is on the order of 5–15% of active entries: \~1,500–4,000 windows fleet-wide that are eligible for cross-region flush at any moment.
**Steady-state row count in `ratelimit_window_counts`.** Each hot window has at most one row per region that has crossed the floor on that window. With low overlap (\~5%) most hot windows live in a single region, so the table holds \~1,500–4,500 active rows in steady state, plus expired rows pending cleanup. Bounded; comfortably small for MySQL.
**Write rate.** Each instance flushes every 10 seconds. Per instance, the flush emits one bulk INSERT with \~50–200 rows (its share of hot windows that changed since the last flush). Across 33 instances:
* **\~3.3 INSERT statements/sec fleet-wide** (one per instance per 10s).
* **\~600–6,000 row-writes/sec fleet-wide**, dominated by the bulk size per statement.
Concurrent writes from instances within the same region collapse via `ON DUPLICATE KEY UPDATE count = GREATEST(...)`, so MySQL never sees real contention on the unique key.
**Read rate.** Each instance syncs every 10 seconds. The query has a `GROUP BY` so the result is one row per active hot window, not one per (region, window). Per instance, the result is \~1,500–4,500 rows. Across 33 instances:
* **\~3.3 SELECT statements/sec fleet-wide**.
* **\~5,000–15,000 row-reads/sec fleet-wide**, served from the `lookup_idx` covering index.
**Comparison to the removed blocklist.** Today's (now-removed) blocklist generated \~150 row-reads/sec and well under 10 writes/sec. The new path is roughly:
* Reads: 30–100× higher.
* Writes: 100–600× higher (from a near-zero baseline).
In absolute terms it is still light load — well below the throughput of a single MySQL primary — but it is a meaningful workload shift. Writes go from event-driven (rare, on denial) to periodic (every 10s, every region). Reads go from "the active blocklist" (\~45 rows visible to each node) to "the active hot subset" (\~1,500–4,500 rows visible to each node).
**Scaling.** At 10× traffic — 240k active windows, \~15k–45k hot, similar overlap fraction — read load reaches \~50k–150k row-reads/sec fleet-wide and write load \~5k–60k row-writes/sec. Still well within a single MySQL primary's envelope but no longer trivial. The next bottleneck would be the periodic walk of `s.counters` in each instance, which is O(active\_windows) per flush; at 240k entries the walk is \~milliseconds, fine.
## Drawbacks
The utilization filter creates a cross-region "free zone" below 50% per region. A user spreading traffic evenly across all 10 regions could in principle stay just under 50% in each, totaling just under 5× their advertised limit, without triggering any propagation. This requires the user to actively load-balance across regions, which most clients do not do. Real abuse concentrates in one region (the closest), which crosses the threshold and propagates. The 5× worst-case is a known limitation of any fan-out-style sharing scheme with a per-region threshold; tightening the threshold reduces the fan-out factor at the cost of more writes.
The sync interval bounds cross-region propagation latency. A region that starts seeing a hot identifier takes up to one flush interval (10s) plus one sync interval (10s) before other regions know about it, plus jitter on each. This is broadly the same latency profile as the previous blocklist scheme; users do not see a regression. Tightening latency is independent of choosing between verdict-shaped and count-shaped propagation, and would be its own RFC.
Workload shape on MySQL changes from event-driven to periodic. The previous blocklist wrote almost never and read a small set; the new path writes and reads on every cycle from every instance. Even though absolute load is still light (see "Expected MySQL load" above), the cardinality of active rows is a couple of orders of magnitude higher than the blocklist held, and operator alerting that watched "blocklist row count" needs to be re-tuned for the new baseline.
## Alternatives
Keeping the blocklist with the current heuristics is the do-nothing option. It works for the steady state and the failure modes are bounded by the filters we just landed. The cost of the redesign is real engineering work for a 5%-of-traffic improvement. Rejecting this RFC is a defensible choice if the team has higher priorities.
A global Redis (one shared cluster across all regions, replacing per-region Redis) would let the existing replay path produce a globally consistent count without any new MySQL plumbing. The blocker is cross-region Redis latency: every local decision in a remote region pays a transcontinental round trip for INCR, which puts request p99 in the hundreds of milliseconds. The whole point of per-region Redis is to keep that off the hot path. A global Redis is the right answer if the latency budget ever shifts to allow it, but it is not on offer today.
A pub/sub propagation channel (Redis pub/sub, NATS, Kafka) replacing the blocklist read/write cycle would tighten propagation latency from \~15s to sub-second. The cost is operating a new piece of infrastructure across all regions, with its own availability and authentication story, for a problem that MySQL solves adequately. We already operate cross-region MySQL. Adding a second cross-region system for the same workload is hard to justify until the latency win is needed by a user-visible feature.
A CRDT library (e.g. an existing G-Counter or PN-Counter implementation as a service) would generalize the count-sharing pattern beyond ratelimits. This is interesting if other Unkey subsystems need similar semantics, but premature otherwise. The custom MySQL table is small, debuggable, and operationally identical to other Unkey tables. We can extract a shared abstraction later if a second use case appears.
## Unresolved questions
The 50% utilization threshold is a first guess. Lower (say 25%) shrinks the cross-region free zone but multiplies write rate; higher (say 75%) cuts writes further but lets larger fan-out attacks through. 50% balances the two: a user can use up to roughly 5x their limit by spreading evenly, which is large in absolute terms but requires active load-balancing across regions to achieve. Once production data is available, revisit the floor based on observed cross-region traffic shape.
The flush and sync intervals (both 10s) are first guesses. A faster sync (5s instead of 10s) tightens propagation but doubles read load. If burst-heavy identifiers turn out to dominate the user-facing pain, faster sync may be worth it. Defer tuning until production data is available.
The migration path: this PR removes the in-memory blocklist machinery and adds the window-counts path in one go. The blocklist table and its sqlc queries are intentionally left in place — existing rows drain naturally over their sequence-derived expiry, and a follow-up PR removes the table once we have confidence the new path is healthy. No data migration; the blocklist table will be empty by then.
# 0016 Vault S3 Storage
Source: https://engineering.unkey.com/architecture/rfcs/0016-vault-s3-storage
Move vault object storage from a single R2 bucket to active-passive S3 buckets with cross-region replication and manual failover.
Vault stores encrypted data encryption keys (DEKs) in object storage. These DEKs are required to encrypt and decrypt recoverable customer key material. Vault caches DEKs to reduce object-storage reads, but a cache miss still requires the backing bucket.
Vault-backed routes started failing after cache entries expired. The affected routes were `/v2/apis.listKeys`, `/v2/keys.getKey`, and `/v2/keys.createKey`.
Increasing the cache TTLs to a fresh TTL of `1h`, a stale TTL of `24h` helps warm reads and many `createKey` requests, but it does not help cold reads, evicted entries, first-time workspace encryption, key rotation, or outages longer than the cache window.
## Why
Vault relies on Cloudflare R2 for durable storage and their recent incidents impacted our API's capability to read/create encrypted keys. We do not control R2 or any of its upstream dependencies. This is a single point of failure that we must address.
The goal is to add a regional recovery path while keeping vault's architecture simple. Operators can promote the replica when the primary region is unavailable.
Regional failures are rare, but they happen, so let's prepare for it.
AWS has strong regional isolation, a single regional S3 failure does not affect S3 in another region. Therefore if we used two S3 regions, our services could accept total loss of availability in one region. AWS also provides (async and slow) cross-region replication.
## Design
Vault's code does not change. The design is exactly the same, we only change the durable data source.
Instead of a single R2 bucket, we will create an S3 bucket in region A and one in region B. We will also configure replication from A to B.
All vault instances read from and write to the primary S3 bucket in region A during normal operation. S3 Cross-Region Replication (CRR) copies objects to the replica bucket in region B. Vault does not read from the replica unless operators promote it during an incident.
```diagram theme={"theme":"kanagawa-wave"}
╭────────────────────╮
│ Vault instances │
│ all regions │
╰─────────┬──────────╯
│ read/write
▼
╭────────────────────╮ async CRR ╭────────────────────╮
│ Primary S3 bucket │─────────────────────────────▶│ Replica S3 bucket │
│ region A │ │ region B │
╰────────────────────╯ ╰────────────────────╯
```
During failover, operators switch vault configuration to the replica bucket and roll every vault instance.
After failover, the replica is the active bucket. The old primary is stale until reverse replication or backfill completes and convergence is verified. Mixed fleets are not allowed. Vault instances must not write to both buckets at the same time.
## Consistency and RPO
S3 Replication Time Control is still asynchronous with an SLA of replicating 99.9% of objects within 15 minutes. If the primary region fails immediately after vault writes new DEK material, the replica may not have that object yet. Recently encrypted recoverable material may be unavailable until the primary recovers or the missing object is restored.
This RFC accepts non-zero RPO in favour of keeping the architecture and migration simple.
## Failover
If the primary region becomes unavailable, vault keeps using the unavailable primary bucket and requests fail until operators promote the replica. To promote, we manually change the S3 secrets in AWS Secrets Manager `unkey/vault` to point the S3 URL to the replica region's bucket, then sync all ExternalSecrets and restart vault pods.
```diagram theme={"theme":"kanagawa-wave"}
╭────────────────────╮
│ Vault instances │
│ all regions │
╰─────────┬──────────╯
│ change config, sync ExternalSecrets, restart pods
│ new read/write
└────────────────────────────────────────────────┐
▼
╭────────────────────╮ async CRR ╭────────────────────╮
│ Primary S3 bucket │─────────────────────────────▶│ Replica S3 bucket │
│ region A │ │ region B │
│ failed │ │ promoted primary │
╰────────────────────╯ ╰────────────────────╯
Before the config change, vault still points at the failed primary and requests fail.
```
At this point all reads and writes go to the promoted replica.
After failover, the original primary is stale and the original replica is now the new primary.
We don't necessarily need to switch back, we can simply copy all immutable objects from the new primary to the old primary and reverse the replication.
Either way, we must not switch back until the old primary has caught up.
## Migration
We'll do a classical dual-write migration.
1. Vault will perform dual writes to both the old R2 and new primary S3.
2. We will copy all objects from R2 to S3.
3. Vault switches reads from R2 to S3.
4. We remove the dual write setup.
5. We remove the R2 buckets.
```diagram theme={"theme":"kanagawa-wave"}
Step 1: dual write
╭────────────────────╮
│ Vault │
╰──────┬───────┬─────╯
│ │
▼ ▼
╭──────────╮ ╭────────────────────╮
│ R2 old │ │ Primary S3 bucket │
╰──────────╯ ╰─────────┬──────────╯
│ async CRR
▼
╭────────────────────╮
│ Replica S3 bucket │
╰────────────────────╯
```
## Alternatives considered
Keeping R2 and relying on cache TTLs is not enough because cache only helps warm data.
Active-active S3 buckets are rejected because S3 replication is asynchronous. Routing reads and writes to the nearest bucket can produce stale reads, missing DEKs, and conflicting current-version pointers.
S3 Multi-Region Access Point is not required for the first version. It can simplify endpoint failover, but it does not solve replication lag.
Synchronous dual-write is deferred. It can reduce RPO, but it adds latency, retry complexity, degraded-mode decisions, and application logic.
# 0017 API key plaintext format
Source: https://engineering.unkey.com/architecture/rfcs/0017-api-key-plaintext-format
Define a versioned, fixed-entropy plaintext format for generated API keys
## Summary
This RFC proposes one plaintext format for all Unkey-generated API keys, including customer API keys and Unkey root keys.
Imported and existing keys retain their original plaintext.
This RFC supersedes the proposal in the
[historical key shape RFC](/architecture/rfcs/0003-key-shape).
## Motivation
The currently generated keys are not unkey-branded in a way that would let us write a regular expression to detect them in source control systems.
This only works for root keys right now, as they have a constant `unkey_` prefix, but it does not work for our customer's keys.
## Detailed design
### Format
The general grammar is as follows. `[x]` indicates the number of characters.
```plaintext theme={"theme":"kanagawa-wave"}
{prefix[1-16]}_{random[8]}unkeyv{version[1]}{depending on the version}
```
Version numbering starts at `1`. The version occupies one Base58 character and\
increments in Base58 alphabet order. It must appear before every version-dependent\
field, so a parser can select the payload definition before reading its lengths.
I do not like the fact, that the version is not at the very beginning or very end of the key. That would make parsing and versioning it simpler.\
But the version is - by design - not random and if it's part of the shown data (first 4 chars or 4 last chars) it doesn't help users to see a difference between keys.
```
// bad because you can't visually scan the prefix and figure out what's what
sk_unke...131f
sk_unke...0ann
// bad because you can't visually scan the suffix and figure out what's what
sk_9agx...eyv1
sk_Z0Lk...eyv1
```
I chose to put it after 8 chars of randomness. This works, but also means that this is now fixed for eternity. A new format must also embed its version after the first 8 random chars, otherwise we couldn't parse a key reliably if we ever wanted to. In reality this is probably fine, I do not foresee a reason to parse a key's version other than to figure out if it has a checksum, and since we allow users to import their own keys, we'd never be able to do this for their keys anyways. But I would like to reserve the option...
### Version 1
Version 1 has this exact shape:
```plaintext theme={"theme":"kanagawa-wave"}
{prefix[1-16]}_{random[8]}unkeyv1{random[36]}{checksum[6]}
```
| Field | Length | Contents |
| ------------------ | -----: | ------------------------------ |
| Prefix | 1–16 | User controlled prefix |
| Separator | 1 | `_` |
| Random head | 8 | First part of the random value |
| Marker and version | 7 | Literal `unkeyv1` |
| Random tail | 36 | Rest of the random value |
| Checksum | 6 | Fixed-width Base58 CRC-32C |
The complete key is between 59 and 74 characters.
The random field contains 44 independent, uniformly distributed Base58
characters. It has about 257.75 bits of entropy.
The format splits the field into an 8-character head and a 36-character tail.
The prefix, marker, version, and checksum add no entropy.
### Prefix
Prefixes must match:
```plaintext theme={"theme":"kanagawa-wave"}
^[A-Za-z0-9_]{0,15}[A-Za-z0-9]$
```
They contain one through 16 ASCII letters, numbers, or underscores, and must
end in an alphanumeric character. `prod_sk`, `prod_key`, and `unkey` are valid.
`prod_sk_` and prefixes longer than 16 characters are invalid.
#### Checksum
GitHub [recommends](https://docs.github.com/code-security/secret-scanning/secret-scanning-partnership-program/secret-scanning-partner-program#identify-your-secrets-and-create-regular-expressions) adding a checksum, so we can pre-filter false positives without a database lookup.
We calculate a CRC-32 checksum over the complete key up to, but excluding, the checksum:
```plaintext theme={"theme":"kanagawa-wave"}
unsigned_key = prefix + "_" + random_head + "unkeyv1" + random_tail
checksum_value = CRC-32C(ASCII(unsigned_key))
checksum_text = base58Fixed(checksum_value, 6)
key = unsigned_key + checksum_text
```
### GitHub secret scanning
The version 1 regex for github is:
```plaintext theme={"theme":"kanagawa-wave"}
[A-Za-z0-9_]{0,15}[A-Za-z0-9]_[1-9A-HJ-NP-Za-km-z]{8}unkeyv1[1-9A-HJ-NP-Za-km-z]{42}
```
Legacy root keys keep their existing GitHub pattern. Either we can provide 2 expressions, or we just merge them for one mega-cursed one.
### Database changes
Separately I do want to change how we display keys in our dashboard. I think it would be nice to show not only the first 4 chars, but also the last 4. This just helps to visually discriminate between two keys when looking at them at a glance.
To do that, we'd store the prefix, first four random characters, and final four
characters for newly generated keys:
```plaintext theme={"theme":"kanagawa-wave"}
prefix = "prod_sk"
start = "K7pQ"
end = "X2Ks"
```
And then display them as `prod_sk_K7pQ...X2Ks`.
Existing non-recoverable keys cannot backfill `end`. Legacy rows keep their
existing `start`, and use `""` for empty fields.
Storing the prefix like this, also makes it easier to reroll keys, cause we don't have to parse the key to figure out what the prefix should be.
### Deprecating configurable lengths
We should remove the option to choose a custom length and use a fixed entropy target of at least 256 bits. There's no good reason for us to allow a user to use less and having fewer (mostly irrelevant) config options is always good in my opinion.
## Drawbacks
* New keys are a little longer.
## Alternatives
* **Put `unkeyv1` at the beginning:** easier to scan, but customer keys become
harder to distinguish visually and appear Unkey-branded.
* **Put the version at the end:** keeps the body visually clean, but a parser
cannot know version-dependent payload and checksum lengths before reaching it.
* **Omit the checksum:** shortens keys, but always requires database lookups for secret
scanning false positives.
# Authentication
Source: https://engineering.unkey.com/architecture/services/api/api-design/auth
API authentication concepts, principals, and permission checks
API authentication turns request credentials into a principal, then checks
whether that principal can perform the requested operation. The credential might
come from a root key, a browser session, or another credential source, but the
rest of the API can reason about the same concepts.
This design keeps authentication source details out of business logic. Handlers
read a principal from the request session, then use that principal for
permission checks, workspace scoping, and audit actor metadata. They don't need
to know how the caller proved its identity.
## How it works
The API auth flow has five conceptual steps:
1. Find a credential on the request.
2. Verify that credential with the system that owns it.
3. Normalize the verified caller into a principal.
4. Attach the principal to the request session.
5. Check the principal's permissions before data access or mutation.
The principal is the boundary between authentication and the rest of the API. It
answers three questions:
* Which workspace is this request scoped to?
* Who or what appears as the actor in audit logs?
* Which permissions can this caller use?
That boundary is the main reason the API uses a unified auth flow. It lets the
API add or change credential sources without spreading source-specific logic
through handlers.
## Request lifecycle
Protected API routes authenticate in middleware before the handler runs. The
middleware resolves the credential, stores the resulting principal on the
session, and applies workspace-level API rate limiting with the principal's
workspace scope.
Handlers do not authenticate requests. A handler reads the principal from the
session and fails closed if the protected route was registered without the auth
middleware. After it has a principal, the handler checks the permission required
for the operation and uses the principal's workspace ID for reads, writes,
cache keys, and audit logs.
```mermaid theme={"theme":"kanagawa-wave"}
sequenceDiagram
participant Client
participant Middleware
participant Session
participant Handler
Client->>Middleware: Send protected API request
Middleware->>Middleware: Verify credential
Middleware->>Session: Store principal
Middleware->>Middleware: Apply workspace API rate limit
Middleware->>Handler: Continue request
Handler->>Session: Read principal
Handler->>Handler: Check permissions
Handler->>Handler: Execute scoped operation
```
The session does not store a separate workspace ID. Request metadata that needs
the workspace, such as error logs or request metrics, reads it from the stored
principal. If a request has no principal, the workspace value is empty.
## Credential sources
Credential sources differ in how they prove identity, how long they live, and
who they represent. After verification, they all produce the same principal
concept.
```mermaid theme={"theme":"kanagawa-wave"}
flowchart LR
A["Root key"] --> D["Principal"]
B["Portal session"] --> D
C["JWT"] --> D
D --> E["Workspace"]
D --> F["Actor"]
D --> G["Permissions"]
```
Root keys represent machine-to-machine access. They are long-lived credentials
owned by a workspace and are suitable for public API clients.
Portal sessions represent an end user acting through the customer portal. They
are browser-oriented credentials and grant the permissions attached to that
session.
JWTs represent short-lived bearer authentication from trusted Unkey-owned
callers. The dashboard proxy uses this source to call the API without exposing a
root key to browser code. The important design constraint is that adding a
credential source does not change how handlers authorize or audit requests.
## Dashboard proxy
The dashboard proxy is an internal bridge between browser-authenticated
dashboard sessions and the API's bearer authentication model. Browser code calls
the dashboard, not the API directly. The dashboard verifies the user's dashboard
session, maps the organization to a workspace, mints a short-lived JWT, and
forwards the API request with that JWT as the bearer credential.
```mermaid theme={"theme":"kanagawa-wave"}
sequenceDiagram
participant Browser
participant Dashboard
participant API
Browser->>Dashboard: Call dashboard proxy
Dashboard->>Dashboard: Verify dashboard session
Dashboard->>Dashboard: Resolve workspace and actor
Dashboard->>Dashboard: Mint short-lived JWT
Dashboard->>API: Forward request with bearer JWT
API->>API: Verify issuer, audience, signature, and time claims
API->>API: Normalize JWT claims into principal
API->>API: Apply workspace API rate limit
API->>API: Check handler permission
```
The proxy exists so the dashboard can use the same API surface that external
clients use while keeping browser credentials scoped to the dashboard. The
browser never receives the API signing secret or a root key. The API still owns
authorization because it verifies the JWT, builds the principal, applies the
workspace rate limit, and checks the handler's required permission.
The dashboard signs with one active secret. The API verifies with an ordered
list of secrets so deployments can rotate signing keys safely. Add the new
secret to the API verification list before the dashboard starts signing with it,
then remove the old secret after every token signed with the old secret has
expired.
The proxy rejects caller-supplied authorization headers. It builds the upstream
request headers itself so browser cookies and dashboard session headers do not
cross the service boundary.
## Principal model
A principal is not the raw credential. It is the normalized identity that the API
trusts after verification.
A principal contains:
* A workspace scope for reads, writes, and limits.
* A subject for audit logs.
* A credential source type for debugging and policy decisions.
* A permission set for authorization.
The subject is deliberately separate from the source. For example, a root key
and a portal session are different sources, but both still need a stable actor
for audit logs. Keeping the audit subject inside the principal avoids coupling
authentication to the audit log implementation.
The API principal shape intentionally stays close to the frontline principal
model. Long term, the API can run behind frontline and consume the same concept
instead of maintaining a separate auth model.
The principal owns permission evaluation for API handlers. That keeps the
handler call site direct: read the authenticated principal, ask whether it can
perform the required operation, then execute the operation. The permission
system still owns RBAC query evaluation and error semantics.
## Permission checks
Authentication only proves who the caller is. Authorization decides what that
caller can do.
The API keeps permission checks after authentication so every operation can ask
for the permission that matches the resource and action it is about to perform.
This keeps broad authentication success from becoming broad API access.
Most operations check a concrete resource and action. Some operations need a
broader predicate, such as "does this caller have any permission to verify keys
for API resources?" Those predicates still belong to the permission system
because they answer an authorization question, not an authentication question.
Permission checks must happen before the handler performs data access or
mutation that depends on the requested operation. Authentication middleware
only proves the request's identity and enforces workspace-level request policy.
It does not grant access to every operation in that workspace.
## Workspace API rate limiting
Workspace API rate limiting is a protected-route middleware concern. It runs
after authentication because the rate limit key is the authenticated principal's
workspace ID, and it runs before the handler because the limit applies to every
protected API operation.
The implementation is bundled with authentication middleware instead of a
separate middleware layer. This keeps the ordering invariant local:
authentication must produce a principal before workspace-level request policy
can run. If more post-auth request policies accumulate, the API can split this
into a separate workspace policy middleware.
## Audit logging
Audit logging records the action performed by an authenticated principal. It
does not need to record the act of verifying the root key itself.
This distinction matters because credential verification can happen as a
mechanical step on many requests, while audit logs are meant to capture
security-relevant product actions. The action audit uses the principal's subject
as the actor.
## Tradeoffs
A unified principal adds a small normalization layer, but it removes repeated
credential-specific branching from handlers. That makes it easier to add new
credential sources without changing every API operation.
Putting permission checks on the principal makes handler code read in the same
order as the request lifecycle: get the principal, authorize the operation, and
execute the operation. The tradeoff is that the principal package depends on
RBAC query evaluation. That dependency is acceptable because the principal
already carries the permission set, and the method is a thin fail-closed wrapper
around the permission system.
The model also keeps audit logging separate from authentication. Authentication
produces the subject that audit logging needs, but the audit system owns how
actions are recorded.
# Error handling
Source: https://engineering.unkey.com/architecture/services/api/api-design/errors
Understanding and working with API errors
Error responses use the same top-level envelope but return an `error` object instead of `data`:
```json theme={"theme":"kanagawa-wave"}
{
"meta": {
"requestId": "req_abc123xyz789"
},
"error": {
"title": "Validation Error",
"detail": "You must provide a valid API ID.",
"status": 400,
"type": "https://unkey.com/docs/errors/validation-error",
"errors": [
{
"location": "body.apiId",
"message": "API not found",
"fix": "Provide a valid API ID or create a new API"
}
]
}
}
```
## Error format
All API errors use `application/json` as the response media type. The error
object is inspired by RFC7807 Problem Details, but Unkey doesn't return a
top-level `application/problem+json` document. The Problem Details-style fields
live inside the Unkey response envelope:
* title: short summary
* detail: human-readable explanation
* status: HTTP status code
* type: URI for documentation
* errors: optional validation details
Don't document new v2 errors as `application/problem+json`. Use the same
`application/json` media type as successful responses so clients, SDKs, and
agents can parse every response with the same content-type rule.
## Common error types
| Status | Error type | Description |
| ------ | --------------------- | --------------------------------------------------------------------------------------------------- |
| 400 | validation-error | Request body failed validation |
| 401 | unauthorized | Missing or invalid authorization |
| 403 | forbidden | Valid authorization but insufficient permissions, only when the caller may know the resource exists |
| 404 | not-found | Resource not found, or the caller may not read it |
| 409 | conflict | Conflicts with current state |
| 429 | rate-limited | Rate limit exceeded |
| 500 | internal-server-error | Unexpected server error |
## Authorization failures must not reveal resource existence
A 403 and a 404 carry different information. If an endpoint answers 404 when a
resource does not exist and 403 when it exists but the caller may not read it,
a caller with no permissions can enumerate which resources exist by telling the
two responses apart.
The rule: an endpoint returns 403 only when the principal is allowed to know
the resource exists, meaning it holds a permission that covers reading the
resource, but lacks the permission the operation requires. When the principal
may not read the resource, every operation on it returns the same 404 the
missing resource produces, with an identical error type, title, and detail.
This matches GitHub's documented behavior of returning 404 instead of 403 for
private resources.
For read endpoints this collapses to: a permission rejection returns the
resource's not-found error. Two consequences for handler code:
* The not-found branch needs no permission check at all, since unauthorized
callers receive the identical response either way.
* The masked 404 must be constructed fresh, not by wrapping the authorization
error. The error middleware joins every public message in a fault chain into
the response detail, and authorization rejections name the missing
permissions, including concrete resource IDs, which would leak the existence
the 404 is masking.
The reference implementation is `svc/api/routes/v2_ratelimit_get_override`,
including a test that probes an existing and a missing resource with a
zero-permission key and asserts the responses are indistinguishable.
## Status codes and domain outcomes
HTTP status codes describe whether Unkey could process the HTTP request. They
don't describe normal product decisions.
Use `200` when Unkey successfully processes a request, even if the domain result
is negative. For example, key verification can return `200` with `data.valid:
false`, and rate limiting can return `200` with `data.success: false`.
Use `4xx` when the request can't be processed as submitted, such as invalid JSON,
failed validation, missing authentication, insufficient permissions, or a missing
resource. Use `5xx` when Unkey fails to process a valid request.
Don't use HTTP status codes as business logic for expected decisions. Model the
decision in the response data instead.
## Validation errors
For validation errors, Unkey returns:
* location: where the error occurred
* message: what went wrong
* fix: suggestion for resolution when available
## Error recovery
Error messages are designed to be actionable. Use the `requestId` when reporting issues.
## Error handling best practices
1. Check status codes first.
2. Parse the error object for details.
3. Retry only on 5xx errors when appropriate.
4. Log the full error response for debugging.
Example handling in JavaScript:
```javascript theme={"theme":"kanagawa-wave"}
const response = await fetch("https://api.unkey.com/v2/keys.create", {
method: "POST",
headers: {
Authorization: `Bearer ${rootKey}`,
"Content-Type": "application/json"
},
body: JSON.stringify(keyData)
});
const data = await response.json();
if (!response.ok) {
const { meta, error } = data;
console.error(`Error ${error.status}: ${error.title}`, {
requestId: meta.requestId,
detail: error.detail,
docs: error.type
});
if (error.errors) {
error.errors.forEach(err => {
console.error(`- ${err.location}: ${err.message}`);
});
}
throw new Error(`API Error: ${error.detail}`);
}
```
# API design overview
Source: https://engineering.unkey.com/architecture/services/api/api-design/index
Design philosophy for Unkey APIs
Unkey APIs prioritize developer experience, consistency, and clarity. This document outlines core design decisions and how to work with the API.
## Core principles
* Clear communication: structured responses make success and failure equally informative
* Practical over purist: pragmatic choices over rigid adherence to a single paradigm
* Predictable patterns: consistent endpoint behavior
## Response structure
All responses share a consistent envelope:
```json theme={"theme":"kanagawa-wave"}
{
"meta": {
"requestId": "req_abc123xyz789"
},
"data": {}
}
```
Paginated responses keep the result collection in `data` and include pagination
metadata in a top-level `pagination` object:
```json theme={"theme":"kanagawa-wave"}
{
"meta": {
"requestId": "req_abc123xyz789"
},
"data": [],
"pagination": {
"cursor": "cursor_xyz123",
"hasMore": true
}
}
```
List endpoints must use this shape unless the endpoint returns multiple
independent collections. Requests use `limit` and `cursor` fields in the JSON
body. Responses require `pagination.hasMore`. Include `pagination.cursor` only
when another page exists.
## Working with the API
### Always use the request ID
Every response includes a unique `requestId`. Include it when debugging or requesting support. You can also search for the request ID in logs .
### Handling pagination
1. Make your initial request.
2. Check `pagination.hasMore`.
3. Use `pagination.cursor` for the next request.
```js theme={"theme":"kanagawa-wave"}
const response = await fetch("https://api.unkey.com/v2/keys.listKeys", {
method: "POST",
headers: { Authorization: `Bearer ${rootKey}` },
body: JSON.stringify({ apiId: "api_123" })
});
if (response.pagination?.hasMore) {
await fetch("https://api.unkey.com/v2/keys.listKeys", {
method: "POST",
headers: { Authorization: `Bearer ${rootKey}` },
body: JSON.stringify({
apiId: "api_123",
cursor: response.pagination.cursor
})
});
}
```
## Versioning
APIs use a major version in the URL, for example `/v2/`. Breaking changes increment the major version.
## OpenAPI examples
Every public v2 operation must include realistic OpenAPI examples. Examples are
part of the API contract because docs, SDKs, and agents use them to learn the
correct request and response shape.
Each operation needs:
* At least one request example
* At least one successful response example
* At least one example per status code or different outcome
Use Unkey-shaped values such as `api_1234abcd`, `key_1234abcd`,
`perm_1234abcd`, and `req_1234abcd`. Don't use placeholder names like `foo`,
`bar`, or `example` when a domain-specific value is available.
## Schema strictness
OpenAPI schemas are closed by default. Request bodies must use
`additionalProperties: false` unless the object is intentionally map-like.
Response objects must also be closed when the shape is known.
Use open objects only for explicit key/value maps, such as metadata fields,
analytics query rows, or user-provided attribute bags. Open objects must explain
what keys and values are valid. When possible, document size limits, value type
limits, and performance impact.
Don't use open objects as an escape hatch for incomplete modeling. If an API
field has known variants, model them explicitly in the schema.
## Idempotency and retries
Every operation must declare its idempotency and retry behavior. Retries are a
client-visible contract, especially for SDKs and agents that may retry requests
after connection errors or `5xx` responses.
Classify each operation as one of:
* `idempotent`: retrying the same request body produces the same final state
* `conditionally-idempotent`: retrying is safe only with a stable client value,
such as an idempotency key or caller-provided resource identifier
* `not-idempotent`: retrying may create additional side effects
Be conservative. If the behavior isn't guaranteed by the implementation, mark
the operation as `not-idempotent` until the API provides a stronger contract.
OpenAPI operations must expose this as machine-readable metadata:
```yaml theme={"theme":"kanagawa-wave"}
x-unkey-idempotency: idempotent
```
Non-idempotent create, reroll, migration, and deployment operations must not be
auto-retried by generated SDKs unless they support an explicit idempotency
mechanism.
## Update semantics
Update endpoints use three-state field semantics:
* Omitted field: leave the existing value unchanged
* Field with a value: set or replace the value
* Field with `null`: clear the value, only when the schema explicitly permits
`null`
Empty arrays and empty objects are values. They don't mean "omitted." For
example, `metadata: {}` replaces metadata with an empty object, and
`ratelimits: []` replaces the rate limit collection with an empty collection
when the endpoint defines full replacement behavior.
Document nullable clear behavior on each field that supports it. Don't rely on a
general update endpoint to imply that every field accepts `null`.
Replacement arrays replace the entire collection unless the endpoint is
explicitly named as an incremental action, such as `addPermissions` or
`removeRoles`.
## Resource identifiers
Prefer user-meaningful identifiers for public API inputs when they are stable and
unambiguous. Slugs, names, and external IDs are easier for humans and agents to
use than internal Unkey IDs because callers often know them without making an
extra lookup request.
Internal Unkey IDs are opaque. Don't require callers to parse their prefixes or
derive meaning from their structure. When an endpoint requires an internal ID,
document that the value must be fetched from another API response and include a
realistic example.
Every identifier field must document:
* Whether it accepts a slug, external ID, internal Unkey ID, or multiple forms
* Whether the value is caller-defined or generated by Unkey
* Whether the value is stable over the resource lifetime
* A realistic example value
Use internal IDs when slugs are mutable, ambiguous, unavailable, or unsafe for
the operation. Otherwise, prefer the identifier that users already understand.
## Delete behavior
Resource delete endpoints must be safe by default. Public API callers don't get
dashboard confirmation flows, and agents can delete resources accidentally.
For resources, prefer soft deletion first. Soft deletion immediately removes the
resource from normal use, then schedules hard deletion after 48 hours with a
delayed workflow. During the 48-hour window, users can restore the resource and
cancel the hard deletion.
Use one generic restore endpoint instead of one restore endpoint per resource
type. The caller passes the resource type and resource identifier in the request
body. The restore operation must validate that the resource type supports
restoration and that the resource is still inside its restore window.
Delete endpoints must document:
* Whether deletion is soft, hard, or configurable
* How long the restore window lasts
* How to restore the resource with the generic restore endpoint
* What related resources are affected immediately
* When hard deletion happens
* Whether repeated delete requests are idempotent
* Whether deleted resources remain visible in audit logs
Use immediate hard deletion only when retention is unsafe, illegal, or explicitly
requested by the caller through a clearly named option.
## Bulk operations
Bulk operations are atomic by default. They must either apply every requested
change or apply none of them. This matters for resources such as environment
variables, where partial success can leave deployments or runtime configuration
in an invalid state.
If an endpoint needs partial success behavior, the endpoint must document that
exception explicitly and return per-item results with enough detail to recover.
## Filtering and sorting
List endpoints use explicit, typed request body fields for filters. Don't add a
generic filter language unless the endpoint is inherently query-based, such as an
analytics endpoint.
When sorting is supported, expose sort fields as explicit enum values. Document
the default order and make it deterministic so pagination is stable.
# RPC-style API
Source: https://engineering.unkey.com/architecture/services/api/api-design/rpc
Action-oriented API design
Unkey v2 APIs use an RPC-style design that focuses on actions rather than
resources. New v2 endpoints must follow this pattern unless an explicit design
review accepts an exception.
```plaintext theme={"theme":"kanagawa-wave"}
https://api.unkey.com/v2/{service}.{procedure}
```
Examples:
* `POST /v2/keys.createKey`
* `POST /v2/ratelimit.limit`
## HTTP methods
All v2 operations use `POST`, except liveness and health endpoints. This keeps
request patterns consistent and lets every operation accept a typed JSON body.
Don't add REST-style resource routes to v2. Use `POST /v2/{service}.{procedure}`
for reads, writes, list operations, and command-style actions.
## Request format
* Use POST
* Include `Content-Type: application/json`
* Include `Authorization` header
* Send parameters as JSON in the request body
```bash theme={"theme":"kanagawa-wave"}
curl -X POST "https://api.unkey.com/v2/keys.createKey" \
-H "Authorization: Bearer root_1234567890" \
-H "Content-Type: application/json" \
-d '{
"apiId": "api_1234",
"name": "Production API Key"
}'
```
## Service namespaces
* keys
* apis
* ratelimit
* analytics
* identities
* permissions
## Operation names
Endpoint names use `{service}.{procedure}`. The OpenAPI `operationId` must match
the endpoint name so docs, SDKs, and agents see the same action name.
Use procedure names that describe the action from the caller's perspective:
* `createApi`
* `getKey`
* `listRoles`
* `updateIdentity`
* `deletePermission`
* `addPermissions`
* `removeRoles`
* `setOverride`
Prefer established verbs before adding a new verb. If a new verb is necessary,
document the behavior before adding endpoints that use it.
## Verb meanings
RPC APIs rely on verb consistency. Use these meanings for v2 procedures:
* `create`: create a new resource. If the resource already exists, return a
conflict or document the idempotent behavior explicitly.
* `get`: read one resource.
* `list`: read a collection. Use pagination when the collection can grow.
* `update`: partially update a resource using three-state update semantics.
* `delete`: remove, invalidate, or tombstone a resource. Document soft-delete
and hard-delete behavior explicitly.
* `add`: incrementally add members to a collection. Adding an existing member
must be idempotent.
* `remove`: incrementally remove members from a collection. Removing a missing
member must be idempotent.
* `set`: replace or upsert a singular configuration object or collection.
Document whether omitted members are removed.
Domain verbs such as `verify`, `limit`, `exchange`, `reroll`, and `migrate` are
allowed when the standard verbs don't describe the operation clearly. Domain
verbs must document side effects and idempotency.
## Benefits
* Clear intent in endpoint names
* Natural mapping to code and user intent
* Better support for complex operations
* Flexible request structures
# Configuration
Source: https://engineering.unkey.com/architecture/services/api/configuration
Configuration model and required settings for the api service
## Configuration model
Unkey services read configuration from a TOML file passed at startup. Environment variables can be referenced with `${VAR}` and are expanded before parsing. Defaults and validation run after parsing.
The config schema maps to [`svc/api/config.go`](https://github.com/unkeyed/unkey/blob/main/svc/api/config.go).
Minimal config example:
```toml theme={"theme":"kanagawa-wave"}
instance_id = "${POD_NAME}"
platform = "aws"
http_port = 7070
region = "${UNKEY_REGION}"
redis_url = "${UNKEY_REDIS_URL}"
[database]
primary = "${UNKEY_DATABASE_PRIMARY}"
readonly_replica = "${UNKEY_DATABASE_REPLICA}"
[clickhouse]
url = "${UNKEY_CLICKHOUSE_URL}"
analytics_url = "${UNKEY_CLICKHOUSE_ANALYTICS_URL}"
[control]
url = "${UNKEY_CTRL_URL}"
token = "${UNKEY_CTRL_TOKEN}"
[vault]
url = "${UNKEY_VAULT_URL}"
token = "${UNKEY_VAULT_TOKEN}"
```
Instance identifier for logs and cache invalidation.
Example: `"api-7d9b8c4f5d-2kq7m"`.
Platform label for logs and metrics.
Example: `"aws"`.
Container image identifier logged at startup.
Example: `"ghcr.io/unkeyed/unkey:v2.0.77"`.
HTTP server port.
Example: `7070`.
Region label for logs and analytics.
Example: `"us-east-1"`.
Redis connection string for counters and usage limiting.
Example: `"redis://redis:6379"`.
Enables test-only behaviors. Do not use in production.
Maximum request size in bytes.
Ordered authentication resolver configuration. Each entry registers one auth
mechanism. At least one entry is required: a config without auth entries
would reject every request, including valid root keys, so startup fails
instead.
Auth mechanism. Supported values are `jwt`, `portal_session`, and
`root_key`.
Expected JWT `iss` claim. Required for `type = "jwt"` entries and rejected
as an unknown field on `portal_session` and `root_key` entries.
Optional JWT provider integration. Set `provider = "workos"` to map the
`roles` claim to API permissions. The API does not infer the provider from
the issuer.
Optional expected JWT `aud` claim for `type = "jwt"`. Verification
requires the configured value to appear in the token's `aud` list. Our
WorkOS environments use a JWT template that sets `aud` to
`["api.unkey.com"]`, so set
`audience = "api.unkey.com"` on the WorkOS entry. Omit this field only
for providers that do not emit an audience claim. Setting it rejects
tokens without a matching `aud`, so when introducing the claim, add it
to the provider's token template first, wait for tokens minted without
it to expire, then set this field.
Optional explicit enable flag. `false` is invalid; remove the auth entry
instead.
Ordered JWT verification secrets for `type = "jwt"`. During
rotation, put the active signing secret first and keep retired secrets
later in the list until every token signed with those secrets has expired.
Configure either `secrets` or `jwks_url`, never both.
JWKS endpoint for `type = "jwt"`. Must be an absolute `https` URL so
signing keys cannot be substituted over an unauthenticated channel. The
API fetches the JSON Web Key Set on first use and verifies incoming JWTs
against usable RSA signing keys from the response. When a token fails
verification against every cached key, the key set is refetched (rate
limited to once per minute), so signing-key rotations are picked up
without a restart. Configure either `jwks_url` or `secrets`, never both.
MySQL configuration.
Primary MySQL DSN.
Optional read replica DSN.
ClickHouse configuration.
ClickHouse connection string for shared analytics.
Base URL for workspace-specific analytics connections.
TLS settings for HTTPS.
Disable TLS when true.
Path to TLS certificate.
Path to TLS key.
Vault connection.
Vault base URL.
Bearer token for Vault.
Control plane connection.
Control API URL.
Bearer token for control API.
pprof endpoint configuration.
Basic auth username.
Basic auth password.
Tracing, logging, and metrics configuration.
Trace sampling rate.
Log sampling rate.
Slow log threshold.
Prometheus port for the `/metrics` listener. To disable metrics, omit the `observability.metrics` section.
## Environment variables
The Helm chart provides these variables for the default config template:
Region label for logs and traces.
Redis URL for counters and usage limiting.
MySQL primary DSN.
MySQL read replica DSN.
ClickHouse shared URL.
ClickHouse analytics base URL.
Control API URL.
Control API token.
Vault URL.
Vault bearer token.
pprof username.
pprof password.
## Dashboard proxy configuration
The dashboard proxy forwards the WorkOS access token when a WorkOS session is
available. The API verifies that token through a `type = "jwt"` auth entry
configured with `provider = "workos"`, the WorkOS issuer, and the JWKS URL. The
WorkOS JWT template includes the organization as `org.id`. WorkOS adds the
built-in `roles` claim. The provider integration expands those roles into Unkey
RBAC permissions and ignores the token's `permissions` and singular `role`
claims. The template also sets `aud` to `["api.unkey.com"]`, and the auth entry
pins `audience = "api.unkey.com"`.
Each WorkOS environment (production, canary) must configure this JWT template
in the WorkOS dashboard under Authentication settings:
```json theme={"theme":"kanagawa-wave"}
{
"org": { "id": {{organization.id}} },
"aud": ["api.unkey.com"]
}
```
The `aud` value must be a JSON array. The API parses `aud` as a string list and
rejects tokens that carry it as a bare string, so a string-valued template
claim fails verification with "Invalid bearer token". Without the template, or
with a missing `aud` claim, every forwarded WorkOS access token fails the
audience check the same way.
Local development still uses a dashboard-minted fallback JWT when no WorkOS
access token exists. For that path, the dashboard needs a signing secret and the
API must include the same secret in a `type = "jwt"` auth entry. The local
fallback JWT includes `roles: ["admin"]`. Set `provider = "workos"` on the local
entry to use the same role mapping.
API base URL that dashboard proxy requests are forwarded to.
Local dashboard proxy signing secret. Add the same value to the API JWT auth
entry's `secrets` list so the API can verify dashboard-minted fallback JWTs.
## Example configuration
```toml theme={"theme":"kanagawa-wave"}
instance_id = "${POD_NAME}"
platform = "aws"
http_port = 7070
region = "${UNKEY_REGION}"
redis_url = "${UNKEY_REDIS_URL}"
[[auth]]
type = "jwt"
issuer = "https://api.workos.com"
audience = "api.unkey.com"
jwks_url = "${UNKEY_JWT_JWKS_URL}"
provider = "workos"
[[auth]]
type = "portal_session"
[[auth]]
type = "root_key"
enabled = true
[observability.tracing]
sample_rate = 0.1
[observability.logging]
sample_rate = 0.01
slow_threshold = "1s"
[observability.metrics]
prometheus_port = 2112
[database]
primary = "${UNKEY_DATABASE_PRIMARY}"
readonly_replica = "${UNKEY_DATABASE_REPLICA}"
[clickhouse]
url = "${UNKEY_CLICKHOUSE_URL}"
analytics_url = "${UNKEY_CLICKHOUSE_ANALYTICS_URL}"
[control]
url = "${UNKEY_CTRL_URL}"
token = "${UNKEY_CTRL_TOKEN}"
[vault]
url = "${UNKEY_VAULT_URL}"
token = "${UNKEY_VAULT_TOKEN}"
[pprof]
username = "${UNKEY_PPROF_USERNAME}"
password = "${UNKEY_PPROF_PASSWORD}"
```
## Related docs
* [Overview](/architecture/services/api/overview)
# Architecture
Source: https://engineering.unkey.com/architecture/services/api/overview
API service components, request flow, and dependencies
The API service is the primary way users interact with Unkey. It exposes an
authenticated RPC-style HTTP API for CRUD operations.
## Request handling pipeline
Most endpoints share a standard middleware stack:
* Panic recovery
* Tracing
* ClickHouse request metrics
* Structured logging with request ID
* Error translation using fault codes and OpenAPI error schemas
* One-minute timeout
* Request validation
Routes that serve internal tooling such as pprof use a reduced stack. The
liveness and reference endpoints disable ClickHouse logging to avoid analytics
noise.
## Core services
The API service composes domain services into handlers during startup.
* Authentication service for normalizing request credentials into principals.
Protected handlers read the principal from the session and use it for
permission checks.
* Key service for root key verification, key authorization, and key mutations.
* [Rate limiting](/architecture/ratelimiting/overview) for standalone
limits, key verification limits, and workspace API throttling.
* Usage limiter backed by Redis counters and MySQL for credit tracking.
* Audit log service for write actions.
* Caches for key, API, and ratelimit namespace lookups.
* Analytics connection manager for per-workspace ClickHouse access.
## Data and storage
* MySQL stores control plane data such as keys, APIs, identities, and
permissions.
* Redis stores regional counters and rate limiting state.
* ClickHouse stores verification and analytics events.
ClickHouse is optional. When it is not configured, analytics writes become
no-ops.
## Cache invalidation
Each node maintains local caches with fresh/stale TTLs. Entries expire on their
own schedule; there is no distributed invalidation.
## Control plane and Vault integration
The API service uses Connect RPC clients to interact with:
* Control plane deployment APIs for deployment operations.
* Vault for analytics credentials and secret handling.
Clients inject `Authorization: Bearer ` headers on every request.
## Reference and schema
The OpenAPI specification is bundled into the API service and served at
`/openapi.yaml`. The `/reference` route serves the Scalar API reference UI built
from the same spec.
# Architecture
Source: https://engineering.unkey.com/architecture/services/control-plane/api/architecture
Runtime composition and request flow for the control plane API
The control plane API is the control surface for deployment intent and orchestration in Unkey. It is the system of record for control-plane state, and it triggers asynchronous workflows for operations that cannot complete on the request path.
## Place in the stack
The control plane API sits between the user-facing control surfaces and the execution plane.
* Accepts configuration and deployment intent from internal automation and the dashboard
* Persists that intent in MySQL as the system of record
* Invokes Restate workflows that perform long running work
* Provides control-plane data to services that reconcile desired state
## Responsibilities
The service owns these responsibilities.
* Persist deployment intent, runtime settings, and Git metadata
* Expose control-plane RPCs for reads, writes, and streaming state
* Trigger workflows for deployments, certificates, routing, and custom domains
* Serve ACME challenge state and custom domain configuration
* Coordinate cluster state and status reporting
## Control-plane data model
The API is the write path for these entities.
* Deployments and their runtime settings
* Custom domains and ACME challenges
* Cluster and routing state
* OpenAPI specs tied to deployments
## Connect RPC surface
The API exposes Connect services over HTTP/2 for control-plane reads, writes, and streaming state used by other services.
* `CtrlService` for core control plane operations
* `DeployService` for deployment creation and workflow triggers
* `OpenApiService` for OpenAPI storage and retrieval
* `AcmeService` for ACME challenges and certificate state
* `ClusterService` for cluster coordination and status
* `CustomDomainService` for domain provisioning and routing
## GitHub webhook flow
When `github.webhook_secret` is set, the API exposes `POST /webhooks/github` and verifies the signature before handling events. Push events create a deployment record and start a deploy workflow.
1. Parse the push payload, and ignore forked repositories or non-branch refs.
2. Look up GitHub repo connections for the installation and repository IDs.
3. Map the branch to `production` or `preview` based on the project's default branch.
4. Load environment settings and variables, and encode a secrets config blob.
5. Insert a deployment record with Git metadata and runtime settings.
6. Invoke the Restate deploy workflow with a Git source payload.
## Certificate bootstrap
When `default_domain` or `regional_domain` is configured, the API seeds wildcard domain records on startup. This sets up platform owned domains so the certificate workflow can issue and renew them.
* Create a `custom_domains` record under the `unkey_internal` workspace
* Insert an ACME challenge with status `waiting` so renewal jobs can pick it up
* Bootstrap `*.{default_domain}` and `*.{region}.{regional_domain}` entries
## Networking and protocols
The API serves Connect RPC over HTTP/2 using h2c. It keeps streaming RPCs open for control-plane watches, and it exposes health endpoints for orchestration.
# Configuration
Source: https://engineering.unkey.com/architecture/services/control-plane/api/configuration
Configuration model and required settings for the control plane API
Unkey services read configuration from a TOML file passed at startup. Environment variables can be referenced with `${VAR}` and are expanded before parsing. Defaults and validation run after parsing.
The config schema maps to [`svc/ctrl/api/config.go`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/api/config.go).
The control plane API is configured via a TOML file: `control-api --config=unkey.toml`.
Instance identifier for logs and tracing.
Region label for routing and observability.
HTTP server port.
Prometheus metrics port. Set to 0 to disable.
Bearer token for control API clients.
Known consumers:
* API service
* Krane service
Rotation is manual today. There is no built-in rotation mechanism.
TODO: Replace with JWT-based auth once `auth.unkey.cloud` is in place.
Base domain for wildcard certificates.
Base domain for regional routing.
Base domain for custom CNAME targets.
MySQL DSN. The control plane API uses one read-write connection for all
queries.
Restate integration.
Restate ingress URL.
Restate admin URL.
Restate API key.
GitHub webhook configuration.
Webhook signature secret.
Tracing configuration. Logging settings are parsed but not applied by the control API runtime.
Trace sampling rate.
Log sampling rate.
Slow log threshold.
## Example configuration
Control API:
```toml theme={"theme":"kanagawa-wave"}
http_port = 8080
prometheus_port = 9090
region = "${UNKEY_REGION}"
instance_id = "${POD_NAME}"
auth_token = "${UNKEY_AUTH_TOKEN}"
default_domain = "${UNKEY_DEFAULT_DOMAIN}"
regional_domain = "${UNKEY_REGIONAL_DOMAIN}"
cname_domain = "${UNKEY_CNAME_DOMAIN}"
database = "${UNKEY_DATABASE_PRIMARY}"
[restate]
url = "${UNKEY_RESTATE_URL}"
admin_url = "${UNKEY_RESTATE_ADMIN_URL}"
api_key = "${UNKEY_RESTATE_API_KEY}"
[github]
webhook_secret = "${UNKEY_GITHUB_APP_WEBHOOK_SECRET}"
[observability.tracing]
sample_rate = 0.1
[observability.logging]
sample_rate = 0.01
slow_threshold = "2s"
```
# Overview
Source: https://engineering.unkey.com/architecture/services/control-plane/api/overview
Control plane API for deployment intent and orchestration
The control plane is the deployment and orchestration plane for Unkey. It consists of a control API and a worker service that runs workflows for deployments, certificates, routing, and background jobs.
The control API is the system of record for deployment intent and configuration. It accepts changes from the dashboard and internal automation, persists them in MySQL, and kicks off worker workflows when asynchronous work is required.
The worker runs the long-lived workflows that create deployments, manage certificates, assign routes, and coordinate background maintenance. Together they provide a consistent control surface for everything that is not handled on the request path.
# Configuration
Source: https://engineering.unkey.com/architecture/services/control-plane/worker/configuration
Configuration model and required settings for the control plane worker
Unkey services read configuration from a TOML file passed at startup. Environment variables can be referenced with `${VAR}` and are expanded before parsing. Defaults and validation run after parsing.
The config schema maps to [`svc/ctrl/worker/config.go`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/worker/config.go).
The control plane worker is configured via a TOML file: `control-worker --config=unkey.toml`.
## Configuration model
The control plane worker loads configuration from a TOML file using `config.Load`. Defaults and validation are applied after parsing.
Runtime-only values (for example `Clock`) cannot be set in the file.
## Required settings
These fields must be set for production deployments.
| Field | Type | Notes |
| -------------- | ------ | ------------------------------------------------------ |
| `cname_domain` | string | Base domain for custom domain CNAME targets. Required. |
| `database` | string | Single read-write MySQL DSN. Required. |
| `vault` | object | Vault connection settings. Required. |
| `restate` | object | Restate admin URL and worker HTTP port. Required. |
## Optional settings
| Field | Type | Default | Notes |
| ---------------- | ------ | ------------- | ------------------------------------------------------------------------------------ |
| `instance_id` | string | - | Instance identifier for logs and tracing. |
| `region` | string | - | Region label for logs and tracing. |
| `observability` | object | - | Observability config (logging, metrics, tracing). |
| `default_domain` | string | `unkey.app` | Fallback domain for system operations, including wildcard certificate bootstrapping. |
| `build_platform` | string | `linux/amd64` | Build platform, format `linux/{arch}`. |
| `acme` | object | - | ACME config for cert issuance. |
| `depot` | object | - | Depot.dev config for builds. |
| `registry` | object | - | Registry credentials for builds. |
| `clickhouse` | object | - | ClickHouse connection settings. |
| `github` | object | - | GitHub App config for deploys. |
| `heartbeat` | object | - | Checkly heartbeat URLs. |
| `slack` | object | - | Slack webhook config for quota alerts. |
## ACME configuration
ACME settings live under `acme`. Enable Route53 DNS-01 challenges with `acme.route53`.
| Field | Type | Default | Notes |
| -------------------------------- | ------- | ----------- | ------------------------------------- |
| `acme.enabled` | boolean | `false` | Enables ACME certificate issuance. |
| `acme.email_domain` | string | `unkey.com` | Used for ACME account email. |
| `acme.route53.enabled` | boolean | `false` | Enables Route53 DNS-01. |
| `acme.route53.access_key_id` | string | - | Required when Route53 is enabled. |
| `acme.route53.secret_access_key` | string | - | Required when Route53 is enabled. |
| `acme.route53.region` | string | `us-east-1` | Route53 region. |
| `acme.route53.hosted_zone_id` | string | - | Optional override for zone discovery. |
## Restate configuration
| Field | Type | Default | Notes |
| --------------------- | ------ | --------------------- | -------------------------------- |
| `restate.admin_url` | string | `http://restate:9070` | Admin API endpoint. |
| `restate.api_key` | string | - | Optional Restate admin auth key. |
| `restate.http_port` | int | `9080` | Worker Restate ingress port. |
| `restate.register_as` | string | - | Optional self-registration URL. |
## Build and registry configuration
`build.backend` selects how builds run. The default `depot` backend runs builds on Depot.dev and is what production uses. Builds are enabled when `registry.password` is set. In that case, `registry.repository`, `registry.username`, `build.depot.api_url`, and `build.depot.project_region` must also be set.
The `kubernetes` backend runs each build as a one-off BuildKit Job in the cluster the worker runs in. It needs no Depot account, but the build pods run privileged without further isolation, so it is intended for local development only. It requires `registry.repository` and in-cluster credentials with permission to manage Jobs and read Pods in the configured namespace.
The worker accepts the legacy top-level `[depot]` table during migration. New configurations should use `[build.depot]`.
| Field | Type | Default | Notes |
| ---------------------------- | ------- | ----------------------- | ------------------------------------------------------------------ |
| `build.backend` | string | `depot` | `depot` or `kubernetes`. |
| `build.depot.api_url` | string | - | Depot API endpoint. |
| `build.depot.project_region` | string | `us-east-1` | Depot storage region. |
| `build.kubernetes.namespace` | string | `unkey` | Namespace for build Jobs. |
| `build.kubernetes.image` | string | `moby/buildkit:v0.26.3` | BuildKit daemon image. |
| `registry.repository` | string | - | Registry repository path, for example `registry.depot.dev/abc123`. |
| `registry.username` | string | - | Registry username. |
| `registry.password` | string | - | Registry password or token. |
| `registry.insecure` | boolean | `false` | Allow plain-HTTP pushes. Local registries only. |
## ClickHouse configuration
| Field | Type | Notes |
| ---------------------- | ------ | ----------------------------------------- |
| `clickhouse.url` | string | ClickHouse connection string. |
| `clickhouse.admin_url` | string | Enables ClickHouse user service when set. |
## GitHub configuration
GitHub configuration is optional and can be omitted for local development.
| Field | Type | Notes |
| ------------------------------------------ | ------- | ------------------------------------ |
| `github.app_id` | int | GitHub App ID. |
| `github.private_key_pem` | string | GitHub App private key. |
| `github.allow_unauthenticated_deployments` | boolean | Only set true for local development. |
## Heartbeat and Slack
| Field | Type | Notes |
| ------------------------------- | ------ | ------------------------------------ |
| `heartbeat.cert_renewal_url` | string | Checkly heartbeat for cert renewals. |
| `heartbeat.quota_check_url` | string | Checkly heartbeat for quota checks. |
| `heartbeat.key_refill_url` | string | Checkly heartbeat for key refills. |
| `slack.quota_check_webhook_url` | string | Slack webhook for quota alerts. |
## Example
```toml theme={"theme":"kanagawa-wave"}
[observability.tracing]
sample_rate = 0.1
[observability.logging]
sample_rate = 0.01
slow_threshold = "2s"
[observability.metrics]
prometheus_port = 9090
region = "${UNKEY_REGION}"
instance_id = "${POD_NAME}"
default_domain = "${UNKEY_DEFAULT_DOMAIN}"
build_platform = "linux/amd64"
cname_domain = "${UNKEY_CNAME_DOMAIN}"
database = "${UNKEY_DATABASE_PRIMARY}"
[vault]
url = "${UNKEY_VAULT_URL}"
token = "${UNKEY_VAULT_TOKEN}"
[acme]
enabled = true
email_domain = "unkey.com"
[acme.route53]
enabled = true
access_key_id = "${UNKEY_ACME_ROUTE53_ACCESS_KEY_ID}"
secret_access_key = "${UNKEY_ACME_ROUTE53_SECRET_ACCESS_KEY}"
region = "${UNKEY_ACME_ROUTE53_REGION}"
[restate]
admin_url = "${UNKEY_RESTATE_ADMIN_URL}"
http_port = 9080
register_as = "${UNKEY_RESTATE_REGISTER_AS}"
[build]
backend = "depot"
[build.depot]
api_url = "https://api.depot.dev"
project_region = "us-east-1"
[registry]
repository = "${UNKEY_REGISTRY_REPOSITORY}"
username = "${UNKEY_REGISTRY_USERNAME}"
password = "${UNKEY_REGISTRY_PASSWORD}"
[clickhouse]
url = "${UNKEY_CLICKHOUSE_URL}"
admin_url = "${UNKEY_CLICKHOUSE_ADMIN_URL}"
[github]
app_id = ${UNKEY_GITHUB_APP_ID}
private_key_pem = "${UNKEY_GITHUB_PRIVATE_KEY_PEM}"
[heartbeat]
cert_renewal_url = "${UNKEY_CERT_RENEWAL_HEARTBEAT_URL}"
quota_check_url = "${UNKEY_QUOTA_CHECK_HEARTBEAT_URL}"
key_refill_url = "${UNKEY_KEY_REFILL_HEARTBEAT_URL}"
[slack]
quota_check_webhook_url = "${UNKEY_QUOTA_CHECK_SLACK_WEBHOOK_URL}"
```
# Deployment sync
Source: https://engineering.unkey.com/architecture/services/control-plane/worker/deployment-sync
How the control plane streams state changes to krane agents
The deployment sync system delivers desired state from the control plane to krane agents running in each Kubernetes cluster. When a deploy workflow creates or updates deployment topologies, agents must learn about those changes so they can converge their cluster to match.
Key components:
* The `deployment_changes` MySQL table ([`pkg/mysql/schema/deployment_changes.sql`](https://github.com/unkeyed/unkey/blob/main/pkg/mysql/schema/deployment_changes.sql)), a change notification log.
* The `WatchDeploymentChanges` RPC ([`svc/ctrl/services/cluster/rpc_watch_deployment_changes.go`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/services/cluster/rpc_watch_deployment_changes.go)), a streaming endpoint that delivers changes to agents.
* The krane watcher ([`svc/krane/internal/watcher`](https://github.com/unkeyed/unkey/blob/main/svc/krane/internal/watcher)), which consumes the stream and dispatches events to controllers.
## Why this exists
The control plane and krane agents are separate processes in separate clusters. When a deploy workflow writes new state to MySQL, agents need to discover that change and apply it to Kubernetes. The deployment sync system bridges this gap with a pull-based streaming model similar to Kubernetes LIST+WATCH.
Previously, a Restate virtual object generated monotonic version numbers that were stamped onto state table rows. This coupled version generation to Restate and required a cross-system round-trip on every write. The `deployment_changes` table replaces this with a pure MySQL solution: writes and notifications happen in a single transaction, and the system can be tested and operated without Restate.
## How it works
### The `deployment_changes` table
Every mutation to deployment state (topology inserts, desired status changes) writes a row to `deployment_changes` in the same MySQL transaction as the state change itself. The row contains no state, just a pointer:
| Column | Purpose |
| --------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `pk` | Auto-increment primary key. Acts as the streaming cursor. |
| `resource_type` | Enum value `deployment_topology`. The schema also defines `cilium_network_policy` and `sentinel`, both legacy values that are no longer produced or dispatched. Per-deployment Cilium policies are now installed by krane during deployment apply. |
| `resource_id` | The ID of the changed resource in its state table. |
| `region_id` | The region this change applies to. |
| `created_at` | Timestamp for TTL-based cleanup. |
A composite index on `(region_id, resource_type, pk)` makes polling efficient.
### The unified stream
Krane agents open a single `WatchDeploymentChanges` stream per region. The stream operates in two modes:
**Full sync (version\_last\_seen = 0).** On first connection or periodic resync, the server:
1. Reads `MAX(pk)` from `deployment_changes` to establish the cursor.
2. Paginates through all rows in `deployment_topology` for the region.
3. Streams every resource as a `DeploymentChangeEvent` with the version set to the max cursor.
This ensures agents see all current state regardless of how old the `deployment_changes` entries are.
**Incremental (version\_last\_seen > 0).** The server polls `deployment_changes` for rows with `pk > version_last_seen`, does a point lookup for each row to load current state from the relevant table, wraps it in a `DeploymentChangeEvent`, and streams it. Polling happens every second when idle.
```mermaid theme={"theme":"kanagawa-wave"}
sequenceDiagram
participant Krane as Krane Agent
participant Ctrl as Control Plane
participant DB as MySQL
Krane->>Ctrl: WatchDeploymentChanges(version=0)
Ctrl->>DB: SELECT MAX(pk) FROM deployment_changes
Ctrl->>DB: Paginate deployment_topology
Ctrl-->>Krane: Stream all state (version = max_pk)
loop Every second
Ctrl->>DB: SELECT FROM deployment_changes WHERE pk > cursor
Ctrl->>DB: Point lookup per changed resource
Ctrl-->>Krane: Stream changed resources
end
```
### Event dispatch in krane
The krane watcher receives `DeploymentChangeEvent` messages carrying deployment state and dispatches each to the deployment controller:
* `DeploymentState` → `deployment.Controller.ApplyDeployment` or `DeleteDeployment`
Unrecognized or nil events are treated as errors to prevent silently skipping changes. The cursor only advances past successfully dispatched events.
### Periodic full resync
The watcher runs a full sync every 10 minutes. This acts as a consistency safety
net: if a change was missed (for example, a `deployment_changes` row was cleaned
up before the agent processed it), the periodic full sync will reconcile the
drift.
## Writing changes
Every code path that mutates deployment state must insert a `deployment_changes` row in the same transaction. The current write sites are:
* [`deploy_handler.go`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/worker/deploy/deploy_handler.go), `createTopologies` (bulk insert + deployment\_changes per region).
* [`deployment_state.go`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/worker/deployment/deployment_state.go), `ChangeDesiredState` (topology status update + deployment\_changes per region).
If you add a new mutation to any of these tables, you must also insert a `deployment_changes` row or the change will be invisible to krane until the next periodic full sync.
## Cleanup
Old `deployment_changes` rows can be cleaned up with TTL-based deletion since full syncs read directly from state tables. The cleanup query deletes rows older than a threshold in batches of 10,000 to avoid long-running transactions.
## Related docs
* [Deployment workflows](/architecture/services/control-plane/worker/workflows/deployments)
* [Worker overview](/architecture/services/control-plane/worker/overview)
# Overview
Source: https://engineering.unkey.com/architecture/services/control-plane/worker/overview
Control plane worker for workflow execution
## Purpose
The control plane worker is Unkey's asynchronous control plane execution layer. It owns long-running, stateful workflows that coordinate changes across the control plane and downstream systems. The worker sits between the control API and infrastructure services, taking durable tasks off the request path and ensuring they complete exactly once.
## Source
[`svc/ctrl/worker`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/worker)
## Place in the stack
The control plane worker is not an API surface. It is a workflow host that acts on state changes initiated by the control API and scheduled jobs.
1. The control API validates requests and persists intent to MySQL.
2. The control API triggers a Restate workflow on the worker.
3. The worker coordinates downstream systems, writes new state to MySQL, and emits side effects such as builds, certificate issuance, and routing updates.
4. Edge components (Frontline and Krane) consume the updated state to apply changes at the data plane.
This separation keeps API handlers short, deterministic, and retry-safe. The worker assumes responsibility for orchestration, retries, and durable state transitions.
## Interfaces
* Restate workflow handlers served by the worker.
* Health endpoints: `/health/live`, `/health/ready`, and `/health/startup`.
* Optional Prometheus metrics server.
## Service boundaries
The worker groups multiple Restate services into one process. Each service uses a virtual object key that defines concurrency boundaries and protects against conflicting state mutations.
| Service | Virtual object key | Responsibility |
| --------------------------- | -------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| DeployService | `project_id` | Build, deploy, promote, and rollback orchestration for a project. |
| DeploymentService | `deployment_id` | Serializes desired state changes with nonce-based last-writer-wins. |
| RoutingService | `project_id` | Atomic reassignment of frontline routes to a deployment. |
| CustomDomainService | `domain` | Domain ownership verification and post-verify actions. |
| CertificateService | `domain` | Certificate issuance and renewal with ACME and Vault. |
| ClickhouseUserService | `workspace_id` | ClickHouse user provisioning and quota updates when enabled. |
| CronService | `varies per handler` | Unified entry point for all scheduled tasks (quota check, key refill, key last-used sync, audit log export, audit log outbox cleanup, ratelimit global counters cleanup). Per-handler VO keys: billing period, date, or a fixed per-task slug. |
| KeyLastUsedPartitionService | `partition_index` | Per-partition fan-out target for `CronService.RunKeyLastUsedSync`. |
## System responsibilities
The worker centralizes orchestration for operations that touch multiple systems or must span minutes:
* Deployment orchestration across regions, including builds, rollout, and routing updates.
* Domain ownership verification, certificate issuance, and renewal.
* Background maintenance, such as key refills and quota checks.
* Optional ClickHouse user provisioning for analytics access.
## Durability model
The worker relies on Restate to make workflows durable and idempotent. Each workflow step is journaled so Restate can replay completed steps and resume from the last successful checkpoint.
* Durable steps isolate side effects and provide exactly-once semantics.
* Virtual object keys serialize conflicting operations per domain, project, deployment, workspace, or region.
* Long-running operations use Restate retries and durable sleep for external rate limits.
* Background jobs persist progress in Restate state for safe resumption.
### Determinism: never read the wall clock directly
Handler bodies replay on every retry, so any non-deterministic value read outside a journaled step diverges between executions and Restate aborts with a diverging-paths error. Reading `time.Now()` (or a `clock.Clock`) directly in handler code is the common offender: the first run and each replay observe different timestamps.
Read the current time through `restateutil.Now(ctx)`, which wraps the read in a `restate.Run` step so the value is journaled on the first execution and reused verbatim on every replay. Unit conversions on the result (`UnixMilli`, `Add`, ...) are deterministic and safe outside the step.
```go theme={"theme":"kanagawa-wave"}
now, err := restateutil.Now(ctx)
if err != nil {
return nil, fmt.Errorf("get now: %w", err)
}
cutoff := now.Add(-retention).UnixMilli()
```
The same rule applies to any other non-deterministic source (randomness, external reads): produce it inside a `restate.Run` step, never in the bare handler body.
## Dependencies
* MySQL for control plane state.
* Restate admin and ingress endpoints.
* Vault for encryption operations.
* ClickHouse for analytics and build telemetry (optional).
* GitHub App credentials for git-based deployments.
* Route53 credentials for ACME DNS challenges.
* Depot and registry credentials for builds.
## Related docs
* [Configuration](/architecture/services/control-plane/worker/configuration)
* [Deployment workflow](/architecture/services/control-plane/worker/workflows/deployments)
* [Routing workflow](/architecture/services/control-plane/worker/workflows/routing)
* [Certificate workflow](/architecture/services/control-plane/worker/workflows/certificates)
* [Deployment sync](/architecture/services/control-plane/worker/deployment-sync)
# Certificates
Source: https://engineering.unkey.com/architecture/services/control-plane/worker/workflows/certificates
ACME challenge and certificate issuance
Certificate issuance is handled by the control worker certificate service. Workflows are keyed by domain name to avoid duplicate issuance.
Key components:
* Certificate service ([`svc/ctrl/worker/certificate`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/worker/certificate)).
* ACME providers ([`svc/ctrl/services/acme`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/services/acme)).
* Vault for encrypting private keys.
* Restate virtual object keyed by domain.
## Flow: issue or renew certificate
```mermaid theme={"theme":"kanagawa-wave"}
sequenceDiagram
participant Worker as Control Worker
participant ACME as ACME Provider
participant Vault as Vault
participant DB as MySQL
Worker->>DB: Claim ACME challenge
Worker->>ACME: Request certificate (HTTP-01 or DNS-01)
Worker->>Vault: Encrypt private key
Worker->>DB: Persist certificate
Worker->>DB: Mark challenge verified
```
## Challenge types
* Wildcard domains use DNS-01.
* Regular domains use HTTP-01.
## Renewal workflow
Certificates are renewed through a Restate handler that scans `acme_challenges` for challenges that are waiting or expiring within 30 days. It triggers `ProcessChallenge` per domain. The renewal handler is intended to be invoked on a schedule via GitHub Actions.
```mermaid theme={"theme":"kanagawa-wave"}
sequenceDiagram
participant Scheduler
participant Worker as Control Worker
participant DB as MySQL
participant Restate as Restate
Scheduler->>Worker: RenewExpiringCertificates
Worker->>DB: ListExecutableChallenges
loop per domain
Worker->>Restate: ProcessChallenge (domain key)
end
```
## Notes
`ProcessChallenge` uses Restate durable sleep when Let's Encrypt returns a rate-limit retry-after value.
TODO: Document challenge routing, HTTP-01 provider details, and renewal scheduling intervals.
# Custom domains
Source: https://engineering.unkey.com/architecture/services/control-plane/worker/workflows/custom-domains
Custom domain verification and lifecycle
Custom domains are registered through the control API and verified through Restate workflows. Each domain is keyed by its hostname to prevent duplicate workflows.
Key components:
* Custom domain service ([`svc/ctrl/services/customdomain`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/services/customdomain)).
* Restate verification workflow (`hydrav1.CustomDomainService`).
* Database records for domain state.
## Flow: add custom domain
```mermaid theme={"theme":"kanagawa-wave"}
sequenceDiagram
actor Client
participant CtrlAPI as Control API
participant DB as MySQL
participant Restate as Restate
Client->>CtrlAPI: AddCustomDomain(domain)
CtrlAPI->>DB: Insert custom domain (status=pending, token, target CNAME)
CtrlAPI->>Restate: VerifyDomain (domain key)
Restate-->>CtrlAPI: invocation_id
CtrlAPI->>DB: Store invocation_id
```
## Flow: retry verification
```mermaid theme={"theme":"kanagawa-wave"}
sequenceDiagram
actor Client
participant CtrlAPI as Control API
participant DB as MySQL
participant Restate as Restate
Client->>CtrlAPI: RetryVerification(domain)
CtrlAPI->>Restate: Cancel existing invocation
CtrlAPI->>Restate: VerifyDomain (domain key)
CtrlAPI->>DB: Reset verification status + invocation_id
```
## Flow: delete domain
```mermaid theme={"theme":"kanagawa-wave"}
sequenceDiagram
actor Client
participant CtrlAPI as Control API
participant DB as MySQL
participant Restate as Restate
Client->>CtrlAPI: DeleteCustomDomain(domain)
CtrlAPI->>Restate: Cancel invocation (if active)
CtrlAPI->>DB: Delete frontline route
CtrlAPI->>DB: Delete ACME challenge
CtrlAPI->>DB: Delete custom domain
```
## Notes
Verification checks run every minute for up to 24 hours. Both checks must pass before the domain is marked verified:
* TXT ownership record at `_unkey.` with value `unkey-domain-verify=`.
* CNAME record pointing to the stored target CNAME.
Once verified, the workflow creates an ACME challenge (HTTP-01) and a frontline route. It triggers certificate issuance asynchronously.
# Deploy Billing
Source: https://engineering.unkey.com/architecture/services/control-plane/worker/workflows/deploy-billing
How Deploy usage flows from ClickHouse to Stripe: the hourly month-to-date push and the month-end invoice close.
## Why this exists
Customers running Deploy workloads are billed for CPU, memory, disk, and egress. The raw usage lives in ClickHouse as per-pod counter checkpoints written by [heimdall](/infra/metering/heimdall), but Stripe is the system that turns usage into an invoice. The Deploy billing push is the hourly job that bridges the two: it computes each workspace's running month-to-date total and reports it to Stripe so the monthly invoice reflects actual consumption.
The push reports the absolute period-to-date total every tick rather than per-tick deltas. That single decision removes the failure modes a delta pipeline normally has: there are no deltas to deduplicate and no exactly-once delivery requirement, because a re-send of the same or a newer total is harmless. What it does not remove is coverage of the period boundary: the last value Stripe receives before the invoice finalizes is the one it bills, so the hourly push alone leaves the final partial hour of the month unbilled. A separate close step pushes the final total for the just-closed period before the invoice finalizes; that is where end-of-month coverage is handled. See [Month-end close](#month-end-close).
## How it works
A cronjob runs every hour and calls `CronService.RunDeployBillingPush` through the Restate ingress. The invocation is keyed by billing period (`YYYY-MM`), so ticks for the same month serialize on one virtual object while different months stay independent.
Each tick does four things:
1. Reads the running month-to-date usage for every workspace from ClickHouse, windowed from the first of the month to now.
2. Aggregates the per-resource rows into per-workspace meter totals, converting each meter into the unit its Stripe meter expects.
3. Resolves each workspace's Stripe customer ID from MySQL and drops only workspaces with no customer. Disabled workspaces are still billed: usage already incurred is owed regardless of current state.
4. Pushes each remaining workspace's totals to Stripe as billing meter events, fanning the pushes out in bounded batches.
```mermaid theme={"theme":"kanagawa-wave"}
sequenceDiagram
participant Cron as CronJob (every 1h)
participant Restate
participant Handler as RunDeployBillingPush
participant CH as ClickHouse
participant MySQL
participant Stripe
Cron->>Restate: POST .../YYYY-MM/RunDeployBillingPush/send
Restate->>Handler: Handle(period=YYYY-MM)
Handler->>CH: GetInstanceMeterUsage(monthStart..now)
Handler->>Handler: aggregate per workspace
Handler->>MySQL: ListWorkspacesForDeployBillingByIDs
loop batches of 16 workspaces
Handler->>Stripe: POST meter events (period-to-date total)
end
Handler-->>Restate: empty response (counts are logged, not returned)
```
### The meter contract
Stripe billing meters are configured with `default_aggregation.formula = "last"`, so each meter keeps the last value it received during the period. The worker sends the period-to-date running total, identifies the customer with the `stripe_customer_id` payload key, and carries the total in the `value` payload key. At period close, Stripe multiplies the metered price by the last value to produce the usage line on the invoice.
The worker references Stripe only by stable meter event names, never by generated price or meter IDs:
| Meter | Event name | Unit |
| ------ | -------------------- | ----------------- |
| CPU | `cpu_seconds` | CPU-seconds |
| Memory | `memory_gib_seconds` | GiB-seconds |
| Disk | `disk_gib_seconds` | GiB-seconds |
| Egress | `egress_public_gib` | binary GiB (2^30) |
These names are the contract between the worker and the Stripe catalog managed in the infra repo. The meter definitions and prices live there, not in this service. See [Stripe catalog setup](#stripe-catalog-setup-infra-repo).
### Why it's safe to re-run
The push is idempotent because the meter aggregates with `last` and the worker always sends the absolute total:
* A missed tick self-corrects on the next send, which carries an even larger month-to-date total.
* A duplicate or overlapping tick sends the same or a newer total, and `last` keeps whichever has the newest event timestamp.
* A Restate replay or manual re-trigger re-sends the current total; `last` keeps the newest, so the billed quantity is unchanged or advances, never doubles.
The events carry no idempotency identifier on purpose. `last` aggregation already makes correctness depend only on the most recent value, so dedup is unnecessary. A stable identifier would actively hurt: Stripe rejects a duplicate identifier with a hard 400, so a re-run within the same window would fail instead of being a harmless no-op. Workspaces are pushed forward only in the sense that the billed quantity tracks the latest observed total; there is no per-event accounting that a retry could double-count.
### Fan-out
Each workspace push runs as its own `DeployBillingPushService` invocation, keyed by workspace id, so a customer's pushes serialize and a broken workspace retries and fails in isolation. The hourly orchestrator dispatches all tasks, awaits each child response, and withholds the Checkly heartbeat when any push fails so monitoring surfaces partial fleet failures.
## Month-end close
The hourly push leaves the final partial hour of the month unbilled: whatever total Stripe last received before the renewal invoice finalizes is the one it bills. The close covers that boundary, and it runs in ctrl-api, not the worker.
When Stripe creates a Deploy workspace's renewal invoice at the period roll it emits `invoice.created`. ctrl-api handles it at `POST /webhooks/stripe` with a narrow relevance gate:
* `billing_reason` must be `subscription_cycle` (manual, custom, and proration invoices are ignored; Stripe keeps its own schedule).
* The customer must resolve to a workspace with a Deploy plan (`deploy_plan IS NOT NULL`).
* The invoice's `subscription` must match that workspace's `stripe_subscription_id` (a second subscription on the same customer is left alone).
For invoices that pass the gate, ctrl-api claims the draft (`auto_advance=false`) and dispatches `CronService.CloseDeployBillingWorkspace` keyed by workspace id. That handler pushes final usage for **this workspace only** and finalizes **this invoice id**. The idempotency key is `deploy-billing-close--`, so each renewal gets its own durable close even when Stripe creates invoices minutes apart.
A 00:30 UTC backup cron runs `RunDeployBillingClose` as a **fleet sweep** keyed by the closed period (`YYYY-MM`). It re-pushes every billable workspace and finalizes any renewal drafts still open. Its idempotency key carries the run timestamp, so it is a fresh retry rather than deduping against webhook closes. A workspace whose final push fails is deliberately **left in draft** rather than finalized: finalizing would freeze an under-billed `last` value onto the invoice with no way to correct it. The sweep runs at 00:30 so it lands before Stripe's \~1h auto-finalization of any invoice the webhook never claimed. A full ctrl outage degrades to Stripe's own one-hour auto-finalization.
### Billing period key
The webhook derives the closed period from the invoice's `period_start` (`YYYY-MM` in UTC). That timestamp always lies inside the billed month, unlike `period_end - 1s`, which drifts when the subscription anchor is not exactly midnight UTC. The fleet sweep still keys off the calendar month; both paths compare draft `period_end` against `pkg/billingperiod` boundaries when selecting invoices to finalize.
### Subscription anchor
Deploy subscriptions are anchored at `00:00:00 UTC` on the 1st. The dashboard pins that anchor at checkout (`subscribeDeploy` / `createSubscription`); there are no pre-existing Deploy subscriptions without it. Stripe may not land on the exact second, which is why the webhook keys off `period_start` rather than `period_end`.
### ctrl-api configuration
The webhook verifies signatures and the close finalizes invoices through the Stripe API, so ctrl-api needs both a webhook secret and an API key in its TOML config:
```toml theme={"theme":"kanagawa-wave"}
[stripe]
webhook_secret = "${STRIPE_WEBHOOK_SECRET}"
secret_key = "${STRIPE_SECRET_KEY}"
```
An empty `webhook_secret` leaves `/webhooks/stripe` unregistered. Both come from the `stripe-credentials` secret (`dev/.env.stripe` in local dev).
### Testing the close locally
1. Configure ctrl-api's Stripe as above, and forward Stripe events to ctrl-api (separate from any dashboard forwarding) so the webhook fires:
```bash theme={"theme":"kanagawa-wave"}
stripe listen --forward-to https://ctrl-api.unkey.local/webhooks/stripe
```
Paste the printed `whsec_...` into `dev/.env.stripe` as `STRIPE_WEBHOOK_SECRET`.
2. Put a workspace on a Deploy plan under a Stripe test clock (the dashboard checkout creates the clocked customer) with Deploy usage in ClickHouse for the period.
3. Advance the clock past the period end so Stripe finalizes the cycle and emits `invoice.created`:
```bash theme={"theme":"kanagawa-wave"}
mise run unkey -- dev stripe clock advance --customer cus_...
```
4. The webhook claims the draft and dispatches `CloseDeployBillingWorkspace` for that workspace. Confirm in the Stripe test dashboard that the renewal invoice carries the final period total and is finalized rather than left as a draft.
## Code layout
The work is split across three packages so the cron handler stays focused on orchestration:
| Package | Responsibility |
| ------------------------------------ | -------------------------------------------------------------------------------------------- |
| `svc/ctrl/worker/cron/deploybilling` | The cron handler: reads usage, aggregates totals, resolves customers, and fans out pushes. |
| `svc/ctrl/internal/billingmeter` | The billing provider client: the `Pusher` interface, the Stripe implementation, and a no-op. |
| `pkg/billingperiod` | Parses the `YYYY-MM` period key into a typed `Period`. |
The push is disabled unless `stripe_secret_key` is configured. When it is empty, the worker wires `billingmeter.NewNoop()` and the cron still runs end to end (reading and aggregating usage) without reporting anything. This keeps the cron binding and schedule uniform across environments that do not bill.
## Configuration
The worker reads its Stripe secret key from its TOML config. Never inline the
key: the config loader expands `${VAR}` from the environment, so reference an
env var and keep the secret out of the file and out of version control.
```toml theme={"theme":"kanagawa-wave"}
[billing]
stripe_secret_key = "${STRIPE_SECRET_KEY}"
```
Use a test-mode key (`sk_test_...`) outside production. When `STRIPE_SECRET_KEY` is unset the value expands to empty and the push is a no-op. An optional Checkly heartbeat URL (`deploy_billing_push_url`) is pinged after a successful run.
## Stripe catalog (infra repo)
The worker only sends meter events by event name. The Stripe objects those events map to (the Deploy product, the usage meters, the metered prices, and the plan-fee prices) are **managed as code in the infra repo, not here** — this service never creates or mutates Stripe objects. The catalog design, meter unit prices, plan fees, per-environment setup, and the apply workflow all live there.
For setup, see the infra guide: Stripe Billing .
## Testing
### Unit tests
The aggregation, period parsing, and meter event building are pure functions with table tests:
```bash theme={"theme":"kanagawa-wave"}
mise exec -- rask ./pkg/billingperiod
mise exec -- rask ./svc/ctrl/internal/billingmeter
mise exec -- rask ./svc/ctrl/worker/cron/deploybilling
```
These cover unit conversion, the `YYYY-MM` parser, and the decimal formatting of meter values without touching Stripe.
### End to end with a Stripe sandbox
To exercise the full path against a real Stripe test account:
1. The usage meters are managed in the infra repo and are already applied to the shared sandbox, so there's nothing to apply from here. (To stand up a fresh sandbox, follow the [infra Stripe guide](https://github.com/unkeyed/infra/blob/main/docs/services/stripe-billing.md).)
2. Give the worker a test-mode key. In local dev (`mise run dev`), copy `dev/.env.stripe.example` to `dev/.env.stripe` and set a `sk_test_...` key from the shared sandbox:
```bash theme={"theme":"kanagawa-wave"}
cp dev/.env.stripe.example dev/.env.stripe
# edit dev/.env.stripe: STRIPE_SECRET_KEY=sk_test_...
```
Tilt loads it into the `stripe-credentials` secret, which the worker reads as `STRIPE_SECRET_KEY` (the config's `stripe_secret_key = "${STRIPE_SECRET_KEY}"` expands to it). Without the file the push stays a no-op and just logs the numbers it would send.
3. Make sure a workspace has a `stripe_customer_id` set, is enabled, and has Deploy usage checkpoints in ClickHouse for the current month.
4. Trigger the push manually through the Restate ingress, keyed by the current billing period:
```bash theme={"theme":"kanagawa-wave"}
curl -X POST \
"http://localhost:8080/hydra.v1.CronService/$(date -u +%Y-%m)/RunDeployBillingPush/send"
```
5. Verify the result. The worker logs `workspaces_pushed` and `meters_pushed` on completion. In the Stripe test dashboard, open the customer's billing meters and confirm the meter values match the month-to-date totals. Run the push again and confirm the values converge on the latest total rather than doubling, which demonstrates the `last` aggregation.
Because the push sends absolute totals, you can re-run it as many times as you like during testing without inflating the billed quantity.
# Deploy Spend Cap
Source: https://engineering.unkey.com/architecture/services/control-plane/worker/workflows/deploy-spend-cap
How Compute spend budgets are checked, alerted, and enforced.
## Why this exists
Workspace admins can set a monthly Compute spend budget in the dashboard. The spend cap emails them at 50%, 75%, and 100% of that budget (measured as gross total metered spend, credits included), and optionally stops all running Compute workloads when the budget is reached.
The check prices usage from ClickHouse with the same catalog rates as the hourly billing push, so enforcement matches what customers see on the billing page.
## How it works
A cronjob invokes `CronService.RunDeploySpendCheck`, keyed by billing period (`YYYY-MM`). The orchestrator:
1. Lists workspaces with a configured budget, plus any workspace that is currently spend-cap suspended (so it can resume after a budget raise, period roll, or budget removal).
2. Reads month-to-date Deploy usage for every workspace in one ClickHouse scan (instance meters + active keys).
3. Prices gross month-to-date usage locally (credits included; the cap is on total metered spend).
4. Fans out to `DeploySpendCheckService.CheckWorkspaceSpend` for workspaces at or above the 50% alert threshold, or any suspended workspace.
Each per-workspace check owns threshold emails, the `deploy_spend_suspended` column, and suspend/resume via `DeployTeardownService`.
```mermaid theme={"theme":"kanagawa-wave"}
sequenceDiagram
participant Cron as CronJob
participant Orch as RunDeploySpendCheck
participant CH as ClickHouse
participant Check as CheckWorkspaceSpend
participant TD as DeployTeardownService
Cron->>Orch: keyed YYYY-MM
Orch->>CH: fleet usage scan
Orch->>Check: fan-out (≥50% or suspended)
alt New threshold crossed
Check->>Check: Resend alert email
end
alt stop=true and gross ≥ budget
Check->>TD: Teardown(SUSPEND)
Check->>Check: deploy_spend_suspended=true
end
alt suspended and (stop=false or gross < budget)
Check->>TD: Resume
Check->>Check: deploy_spend_suspended=false
end
```
## Cap on gross spend
Spend budgets apply to gross month-to-date metered spend, credits included: a $100 budget stops at $100 of usage, not \$100 on top of the plan's included credit. The check subtracts no credit and reads no Stripe balance; it prices the same gross total the hourly billing push reports and compares it directly against the budget.
## Enforcement gates
* **New deployments** are blocked while `deploy_spend_suspended=true`.
* **Wake deployment** (preview environments) is also blocked while suspended.
* **Cancel Deploy** clears `deploy_spend_suspended` along with the plan entitlement.
## Cadence
The local dev cron runs **every 3 minutes** so alerts and enforcement are observable without a long wait. Production runs every 15 minutes (configured in the infra repo). Worst-case detection latency equals the cron cadence plus teardown drain time.
## Configuration
The worker needs ClickHouse (usage reader), Resend (`RESEND_API_KEY`), and WorkOS (`WORKOS_API_KEY`) for alert emails. Without Resend or WorkOS the check still runs and can suspend compute, but no email is sent.
Resend templates `compute-budget-alert` and `compute-budget-stopped` must be published (`web/internal/resend/scripts/sync-templates.tsx --publish`).
See [local development](/contributing/local/development) for `dev/.env.resend` and `dev/.env.workos`.
## Code layout
| Package | Responsibility |
| ----------------------------------------------------------- | --------------------------------------------------------------- |
| `svc/ctrl/worker/cron/deployspendcheck` | Orchestrator, per-workspace check, threshold math, alert emails |
| `svc/ctrl/worker/deployteardown` | Suspend (stop workloads, record snapshot) and resume |
| `web/apps/dashboard/.../spend-budget.tsx` | Budget UI and spend meter |
| `web/apps/dashboard/lib/trpc/routers/billing/deploy-budget` | Get/set budget preferences |
## Testing
```bash theme={"theme":"kanagawa-wave"}
mise exec -- bazel test //svc/ctrl/worker/cron/deployspendcheck:deployspendcheck_test
mise exec -- bazel test //svc/ctrl/integration:integration_test --test_filter=DeploySpendCheck
```
Integration tests cover suspend/resume, budget removal while suspended, and resuming when stopping is turned off.
# Deployments
Source: https://engineering.unkey.com/architecture/services/control-plane/worker/workflows/deployments
Deploy, promote, rollback, cancel, and build-queue workflows
The control plane deployment service creates deployment records and delegates execution to Restate workflows.
## Virtual object keying
Each Restate VO uses the narrowest key that gives the serialization it needs:
* **`DeployService`** is keyed by `deployment_id`. Each deployment runs as its own isolated workflow, so multiple deployments in the same environment can build in parallel.
* **`RoutingService`** is keyed by `env_id`. All routing changes for an environment (frontline route assignment + the live-deployment swap) serialize here, so concurrent deploys/rollbacks/promotes for the same env can never race on `apps.current_deployment_id`.
* **`BuildSlotService`** is keyed by `workspace_id`. Caps how many deployments build at once across the workspace and prioritises production over preview waiters.
* **`DeploymentService`** (delayed desired-state transitions) is keyed by `deployment_id`.
Rollback and Promote are themselves workflows but they don't need their own env-keyed gate — the actual mutation goes through `RoutingService.SwapLiveDeployment`, which is per-env serialized.
Key components:
* Control API deployment service — [`svc/ctrl/services/deployment`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/services/deployment)
* `DeployService` Restate workflow — [`svc/ctrl/worker/deploy`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/worker/deploy)
* `RoutingService` (route assignment + live swap) — [`svc/ctrl/worker/routing`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/worker/routing)
* `DeploymentService` VO for delayed desired-state transitions — [`svc/ctrl/worker/deployment`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/worker/deployment)
* `BuildSlotService` VO for per-workspace build concurrency — [`svc/ctrl/worker/buildslot`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/worker/buildslot)
* Dedup helper for cancelling superseded queued siblings — [`svc/ctrl/dedup`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/dedup)
## Flow: create deployment
```mermaid theme={"theme":"kanagawa-wave"}
sequenceDiagram
actor Client
participant CtrlAPI as Control API
participant DB as MySQL
participant Restate as Restate
participant BuildSlot as BuildSlotService
Instance identifier for logs and tracing.
Plain-HTTP listener port. Serves ACME HTTP-01 challenges and 308-redirects everything else to https\://.
HTTPS listener port. Terminates TLS, runs the policy engine, and forwards customer traffic to a deployment instance (or to a peer frontline in another region).
Cloud provider identifier (for example, `aws`, `gcp`, or `local`).
Region label for routing.
Apex domain for regional routing.
Maximum number of cross-region routing hops.
A 64-character hexadecimal Ed25519 seed. Frontline uses the seed to sign
PASETO v4.public metadata. Set the same seed in every region in one
environment.
Generate the seed with OpenSSL:
```bash theme={"theme":"kanagawa-wave"}
openssl rand -hex 32
```
Control API connection settings.
Control API address.
Preshared bearer token for authenticated Control API requests.
Prometheus metrics port. Set to 0 to disable.
Maximum duration for a proxied request before the context is cancelled and a `504` is returned.
TLS settings for HTTPS.
Disable TLS when true.
Path to TLS certificate.
Path to TLS key.
MySQL configuration.
Primary DSN.
Optional read replica DSN.
ClickHouse analytics storage for request-level events. When the URL is empty, a no-op backend is used and no request telemetry is recorded.
ClickHouse connection string.
Maximum number of items to collect before flushing a buffer to ClickHouse. Applies to all event buffers (frontline requests, key verifications).
Capacity of the channel buffer holding incoming items. When full, new items are dropped.
Number of goroutines that drain each buffer.
Redis connection for distributed rate limiting and usage limiting in the policy engine. When the URL is empty, an in-memory counter is used as a fallback and rate limits are not shared across replicas.
Redis connection string.
Vault connection.
Vault URL.
Vault token.
Tracing and logging configuration.
Trace sampling rate.
Log sampling rate.
Slow log threshold.
Go pprof profiling endpoints on a loopback-only listener. Disabled when omitted or when credentials are empty.
Basic Auth username for the pprof endpoints.
Basic Auth password for the pprof endpoints.
TCP port for the loopback-only (127.0.0.1) pprof server.
## Environment variables
The Helm chart provides these variables for the default config template:
Region label.
Apex domain for routing.
Control API address.
Preshared bearer token for authenticated Control API requests.
A 64-character hexadecimal Ed25519 seed that signs PASETO v4.public metadata
between Frontline regions.
Vault URL.
Vault token.
MySQL primary DSN.
MySQL read replica DSN.
ClickHouse connection string for request telemetry. Optional.
Redis connection string for distributed rate limiting. Optional.
## Example configuration
```toml theme={"theme":"kanagawa-wave"}
http_port = 7070
https_port = 7443
platform = "aws"
region = "${UNKEY_REGION}.aws"
instance_id = "${POD_NAME}"
apex_domain = "${UNKEY_APEX_DOMAIN}"
max_hops = 10
frontline_meta_signing_key = "${UNKEY_FRONTLINE_META_SIGNING_KEY}"
prometheus_port = 9090
request_timeout = "15m"
[control]
url = "${UNKEY_CONTROL_URL}"
token = "${UNKEY_CONTROL_TOKEN}"
[database]
primary = "${UNKEY_DATABASE_PRIMARY}"
readonly_replica = "${UNKEY_DATABASE_REPLICA}"
[clickhouse]
url = "${UNKEY_CLICKHOUSE_URL}"
[redis]
url = "${UNKEY_REDIS_URL}"
[vault]
url = "${UNKEY_VAULT_URL}"
token = "${UNKEY_VAULT_TOKEN}"
[observability.tracing]
sample_rate = 0.1
[observability.logging]
sample_rate = 0.01
slow_threshold = "2s"
```
## Related docs
* [Overview](/architecture/services/frontline/overview)
# Failure modes
Source: https://engineering.unkey.com/architecture/services/frontline/failure-modes
Failure scenarios, responses, and diagnosis
Use this page to look up an error code or symptom you're seeing and find out what caused it and what to do.
## Error code reference
Find the error code from the response body, logs, or traces, then look it up here.
### `Frontline.Routing.ConfigNotFound` (404)
The request hostname does not resolve to a deployment. No `frontline_route` row matches the fully qualified domain name.
**What to check:**
1. Verify the hostname has a route in MySQL (custom domain verified, or a live deployment for the apex domain).
2. The `frontline_route` cache is fresh for 5 seconds and stale for up to 5 minutes. A recently created route may not appear immediately.
### `Frontline.Routing.DeploymentNotFound` (404)
The resolved deployment does not exist, or it did not match the expected environment.
Frontline returns 404 (not 403) for both cases to avoid leaking whether a deployment exists.
**What to check:**
1. Verify the deployment resolved from the hostname exists in MySQL.
2. If the deployment was recently moved or recreated, the cache may be stale (fresh: 30s, stale: up to 5min). Wait for the entry to expire.
### `Frontline.Routing.NoRunningInstances` (503)
All instances for the deployment in this region are down, scaling to zero, or not yet ready, and no peer region has a healthy instance either.
**What to check:**
1. Frontline logs include the total instance count versus running count for this deployment.
2. Check Krane logs for the deployment controller to see instance status.
3. The instance cache has a 10s fresh TTL. A recently started instance may not appear for up to 10 seconds.
### `Frontline.Routing.DeploymentSelectionFailed` (500)
Frontline resolved the deployment but failed to select a target instance, for example because the routing data was malformed or a backing query failed.
**What to check:**
1. Check Frontline logs for the underlying error and deployment ID.
2. Verify the deployment's instance records in MySQL.
### `Frontline.Proxy.ServiceUnavailable` (503)
The selected instance is not accepting connections (`ECONNREFUSED`), the host is unreachable (`EHOSTUNREACH`), or DNS resolution failed.
**What to check:**
1. Check the target instance pod logs (the instance address is included in the error context).
2. Verify the instance container is running and listening on its port.
3. For DNS failures, check cluster DNS health (`kube-dns` or `coredns` pods).
### `Frontline.Proxy.BadGateway` (502)
The instance accepted the connection but the request failed. Common causes: connection reset mid-request (`ECONNRESET`), application crash, OOM kill, or pod replacement during a rollout.
**What to check:**
1. Check the instance pod logs and events for OOM kills or restarts.
2. If this correlates with a deployment rollout, it may be transient.
### `Frontline.Proxy.GatewayTimeout` (504)
The instance did not respond before the request deadline, or the transport dial timed out.
**What to check:**
1. Check instance latency in ClickHouse or Prometheus.
2. Check whether the instance is under heavy load or blocked on a downstream dependency.
Frontline enforces a request timeout through the `WithTimeout` middleware, set from the `request_timeout` config (default 15m). The server read and write timeouts are disabled (-1) so that streaming responses and long-lived upgrades are not cut off by the HTTP server itself; the request timeout is the ceiling instead.
### `Frontline.Proxy.ProxyForwardFailed` (502)
Generic proxy failure that did not match a more specific error category.
**What to check:**
1. Check Frontline logs for the full error message and instance address.
2. Check cluster networking (CNI, CiliumNetworkPolicies).
### `Frontline.Internal.InvalidConfiguration` (422)
The deployment's `sentinel_config` column contains malformed JSON or invalid protobuf, or a KeyAuth policy has an unparseable permission query. This is the config author's fault rather than a Frontline fault, so it is a 422 in the config domain.
**What to check:**
1. Query the deployment record in MySQL and inspect `sentinel_config`.
2. Validate it parses as a `frontline.v1.Config` protobuf.
### `Frontline.Internal.ConfigLoadFailed` (500)
Frontline failed to load the deployment's configuration, for example because a backing query failed.
**What to check:**
1. Check Frontline logs for the underlying error.
2. Check MySQL primary and replica health.
### `Frontline.Internal.InternalServerError` (500)
Unexpected error in Frontline. Check Frontline logs for the full stack trace.
### `Frontline.Auth.MissingCredentials` (401)
No API key found in the request. The KeyAuth policy checked all configured extraction locations (Bearer token, header, query param) and found nothing.
**What to check:**
1. Verify the client is sending the key in the expected location.
2. Check the KeyAuth policy's `locations` config on the deployment.
### `Frontline.Auth.InvalidKey` (401)
The API key was found but failed verification. Covers: key not in database, key disabled, key expired, workspace disabled, or key not in any of the configured `key_space_ids`.
**What to check:**
1. Verify the key exists and is enabled in the Unkey dashboard or database.
2. Check the key's keyspace matches one of the policy's `key_space_ids`.
### `Frontline.Auth.InsufficientPermissions` (403)
The key is valid but does not satisfy the policy's `permission_query`.
**What to check:**
1. Check the key's assigned permissions against the RBAC query in the KeyAuth policy.
### `Frontline.Auth.RateLimited` (429)
The key exceeded its rate limit or usage limit. Rate limit headers are included in the response regardless of success or failure.
**What to check:**
1. Read `X-RateLimit-Remaining` and `X-RateLimit-Reset` from the response.
2. Check if the limit is per-key or per-deployment in the policy config.
### `Frontline.Firewall.Denied` (403)
A Firewall policy with `action=DENY` matched the request. The request is rejected before reaching the instance.
**What to check:**
1. Review the deployment's Firewall policies and their match expressions.
### `Frontline.OpenApi.InvalidRequest` (400)
The request does not conform to the deployment's OpenAPI specification (unknown operation, or a path, query, header, or body that fails schema validation).
**What to check:**
1. Compare the request against the deployment's OpenAPI spec.
2. Confirm the spec scraped from the running deployment is current.
### `User.BadRequest.ClientClosedRequest` (499)
The client disconnected before Frontline finished proxying the response. This is a client-side issue (categorized as `user` error type in metrics), not a platform problem.
## Startup and degraded modes
### Redis unavailable
Redis is optional. When `redis.url` is empty or unset, the rate limit counter falls back to in-memory. In that mode rate limits are enforced per replica rather than globally across the fleet; distributed enforcement requires Redis.
Authentication, firewall, and OpenAPI policies do not depend on Redis and are still enforced. A missing Redis degrades distributed rate limiting only, it does not disable policy evaluation.
**What to check:**
1. Verify the `redis.url` in the Frontline config if you expect distributed rate limiting.
2. Check Redis pod health in the cluster.
## Cache behavior
When routing or policy changes are not taking effect, the cause is usually cache staleness. Each Frontline node maintains its own caches and refreshes them on their own TTLs.
| Cache | Fresh | Stale | Max entries |
| ------------------------- | ---------- | ---------- | ----------- |
| `frontline_route` | 5 seconds | 5 minutes | 10,000 |
| `policies` | 30 seconds | 5 minutes | 10,000 |
| `instances_by_deployment` | 10 seconds | 60 seconds | 10,000 |
| `tls_certificate` | 1 hour | 12 hours | 10,000 |
During the stale window, Frontline serves old data while refreshing in the background. A routing or policy change can take up to its stale TTL to propagate to every node, because each node refreshes on its own schedule. When a node's working set exceeds 10,000 entries for a cache, eviction increases miss rates and database load.
## Where to look
| Signal | Where to find it |
| ----------------------- | ----------------------------------------------------------------------------------------- |
| Error code | Response body JSON (`error.code`), trace span attributes |
| Error type | Prometheus `error_type` label (`none`, `user`, `customer`, `platform`) |
| Request details | ClickHouse `FrontlineRequest` table (30-day TTL, Authorization header redacted) |
| Latency breakdown | `Server-Timing` response header, ClickHouse `GatewayLatency` / `InstanceLatency` fields |
| Request rate and errors | Prometheus `unkey_frontline_requests_total` (labels include status\_code and error\_type) |
| Distributed trace | Span `frontline.proxy` with request\_id, status\_code, error\_type attributes |
| Instance-level failures | Error context includes instance address, check pod logs for that address |
| Frontline pod health | Kubernetes pod status for the Frontline deployment |
# Frontline ingress
Source: https://engineering.unkey.com/architecture/services/frontline/ingress
How Frontline terminates TLS, resolves hostnames, and proxies to deployment instances
Frontline is the shared ingress tier for Unkey. It is the first Unkey-owned hop
for inbound traffic. Its job is to convert a public hostname into a concrete
deployment, evaluate that deployment's policies, and proxy the request to a
running instance with minimal latency.
## Role in the stack
Frontline runs as a regional, multi-tenant edge service. A single fleet per
region serves every workspace and environment. It terminates TLS for custom
domains, looks up the target deployment in the control plane database, runs the
deployment's policies inline, and proxies directly to a running instance in the
same region. When the local region has no healthy instance, it forwards to
another region's Frontline, which redoes the full hostname to instance chain
while preserving TLS termination and routing consistency.
There is no separate per-environment proxy. Frontline owns the request path end
to end.
## Responsibilities
* Terminate TLS using SNI and custom domain certificates.
* Resolve hostnames to deployments and their running instances.
* Evaluate the deployment's policies (KeyAuth, RateLimit, Firewall, OpenAPI) before proxying.
* Proxy to a local instance, or forward to a peer region when none is healthy locally.
* Enforce hop limits to prevent routing loops.
* Render HTML error pages when clients prefer HTML.
* Serve ACME HTTP-01 challenges for certificate issuance.
## Traffic flow
```mermaid theme={"theme":"kanagawa-wave"}
sequenceDiagram
actor Client
participant Frontline
participant Router
participant Engine as Policy Engine
participant Instance
participant RemoteFrontline
Client->>Frontline: HTTPS request
Frontline->>Router: Route(hostname)
Router-->>Frontline: RouteDecision (instances + policies)
alt local instance available
Frontline->>Engine: Evaluate policies
Engine-->>Frontline: Principal or rejection
Frontline->>Instance: Proxy request (HTTP)
else no local instance
Frontline->>RemoteFrontline: Forward with signed metadata (HTTPS)
RemoteFrontline->>RemoteFrontline: Verify metadata and read hop history
end
```
## Routing model
Frontline reads routing data from MySQL and caches it with stale-while-revalidate
semantics. The cache stores hostname to route mappings, the deployment's parsed
policies, and the deployment's running instances. Each node maintains its own
local cache; entries refresh on their fresh/stale schedule.
Instance selection is based on health and region proximity. If the deployment
has a running instance in the current region, Frontline proxies to it directly,
trying candidates in shuffled order and advancing on dial failures. Otherwise
Frontline forwards to the nearest region that has a running instance. If no
region has one, Frontline returns a service unavailable error.
## Proxying model
Frontline proxies directly to the deployment instance for local routes and uses
a shared HTTP transport for cross-region forwarding to a peer Frontline. Requests
carry routing and trace headers that the receiving instance or peer Frontline
uses to identify the deployment and to track the forwarding chain.
Key headers:
* `X-Unkey-Frontline-Id`, `X-Unkey-Region`, and `X-Unkey-Request-Id`
* `X-Unkey-Frontline-Meta` on peer requests only
* `X-Deployment-Id`
* `X-Forwarded-Proto`
* `X-Unkey-Timing`
Frontline uses a PASETO v4.public token in `X-Unkey-Frontline-Meta`. The claims
contain an expiry time and an ordered hop history. Each hop contains the region,
request ID, Frontline ID, and forward time as Unix milliseconds. HTTPS protects
the request in transit. The PASETO signature lets a peer authenticate the
metadata and detect changes.
Each Frontline removes incoming metadata before routing. It verifies metadata
when the header contains one non-empty value. A request without valid metadata
starts with an empty hop history. Invalid metadata never blocks a request.
Before a cross-region forward, Frontline appends the current hop and sets the
expiry to 1 minute in the future. It uses the history length as the hop count.
It rejects a forward that reaches the configured hop limit. Frontline removes
the metadata before it forwards the request to a deployment instance.
## TLS and certificate selection
Frontline supports three TLS modes:
* Dynamic certificates from Vault via the certificate manager.
* Static certificates from files for development.
* TLS disabled explicitly in configuration.
The certificate manager looks up certificates by exact hostname and then by the
immediate wildcard (for example `*.example.com`). Certificates are stored in
MySQL with encrypted private keys that are decrypted using Vault and cached for
reuse.
## ACME HTTP-01 challenges
Frontline runs a separate HTTP server on the challenge port for
`/.well-known/acme-challenge/*` requests. It validates the hostname, forwards
the token to the control plane ACME service, and returns the authorization
response to the ACME client.
## Observability and error handling
Every request is wrapped in a middleware that emits tracing spans, Prometheus
metrics, and structured logs. Frontline also captures errors from policy
evaluation and proxying and maps them to typed error codes. When a client
prefers HTML, Frontline renders a styled error page; otherwise it returns JSON
error payloads. Errors from the upstream instance are categorized and mapped to
Frontline gateway errors for consistent observability.
# Overview
Source: https://engineering.unkey.com/architecture/services/frontline/overview
Multi-tenant ingress and gateway: TLS termination, policy enforcement, and routing to deployment instances
Frontline is Unkey's multi-tenant ingress and gateway. It is the first Unkey-owned hop for inbound traffic and the boundary where every request is authenticated, rate limited, and routed before it reaches a customer's deployment.
Frontline runs as a regional, multi-tenant edge service. A single fleet serves all workspaces and environments in a region. There is no per-environment proxy: Frontline owns the request path end to end, from TLS termination to the deployment instance.
## Responsibilities
* Terminate TLS for apex and custom domains using SNI, and redirect plain HTTP to HTTPS.
* Resolve the request hostname to a deployment using control-plane data in MySQL, with short-lived routing and certificate caches to avoid a round trip on every request.
* Evaluate the deployment's policies (KeyAuth, rate limiting, firewall, OpenAPI validation) before proxying. A request that fails a policy receives a structured error and never reaches the instance.
* Select a healthy instance of the deployment in the same region and proxy the request directly, streaming the response back. When no local instance exists, forward to a peer Frontline in another region.
* Strip any client-supplied `X-Unkey-Principal` header and set the verified principal after authentication, so downstream code can trust it.
* Record request telemetry to ClickHouse and expose Prometheus metrics.
## Architecture position
```
Client (HTTPS)
│
▼
Frontline (regional, multi-tenant)
TLS termination, hostname routing,
policy evaluation, instance selection
│
▼
Instance (customer workload)
```
Routing decisions come from control-plane data stored in MySQL. When the local region has no healthy instance for the target deployment, Frontline forwards the request to a peer Frontline in a region that does, preserving TLS termination and routing consistency.
## Runtime subsystems
Frontline embeds the subsystems policy execution needs. Rate limit policies use Unkey's [rate limiting](/architecture/ratelimiting/overview) service rather than a Frontline-owned counter implementation.
## Related pages
* [Ingress](/architecture/services/frontline/ingress) for TLS termination, hostname resolution, and cross-region routing
* [Routing](/architecture/services/frontline/routing) for instance selection and failover
* [Request flow](/architecture/services/frontline/request-flow) for the full lifecycle of a proxied request
* [Policies](/architecture/services/frontline/policies/index) for the policy engine
* [Configuration](/architecture/services/frontline/configuration) for config fields and defaults
# Firewall
Source: https://engineering.unkey.com/architecture/services/frontline/policies/firewall
Policy that denies matched requests
Firewall denies any request that matches the policy's [match expressions](/architecture/services/frontline/policies/match-expressions) (path, method, header, or query parameter). The MVP has a single action and no other configuration — when a match hits, Frontline rejects the request with HTTP 403 and a fixed `Forbidden` body. The action enum exists so additional outcomes (allow, log, challenge) can be added later without restructuring the message.
## Fields
The outcome to apply when the policy's match expressions all succeed. Only `ACTION_DENY` is defined today.
## Actions
| Action | Behavior |
| ------------- | --------------------------------------------------------------------------------------------------------------------------- |
| `ACTION_DENY` | Rejects the request with HTTP 403 and body `Forbidden`. Short-circuits the whole policy chain — no downstream policies run. |
## Examples
Block everything below `/admin`:
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "block-admin",
"name": "Block /admin",
"enabled": true,
"match": [
{ "path": { "path": { "prefix": "/admin" } } }
],
"firewall": { "action": "ACTION_DENY" }
}
]
}
```
## Observability
Every Firewall match increments `frontline_firewall_matches_total{policy_id, action}`. Denied requests do not currently produce a ClickHouse request log row — they never reach an instance, and the existing request-log pipeline gates on instance presence. Dedicated observability for firewall matches is deferred.
# Policies
Source: https://engineering.unkey.com/architecture/services/frontline/policies/index
Policy engine and evaluation model
Frontline evaluates middleware policies before proxying traffic to deployment instances. Policies are stored in the deployment's `sentinel_config` column as a JSON-serialized `frontline.v1.Config` protobuf.
## Evaluation flow
```mermaid theme={"theme":"kanagawa-wave"}
sequenceDiagram
actor Client
participant Frontline
participant Router
participant Engine
participant Instance
Client->>Frontline: Request
Frontline->>Router: GetDeployment + SelectInstance
Frontline->>Engine: Parse and evaluate policies
Engine-->>Frontline: Principal (optional)
Frontline->>Instance: Proxy request with X-Unkey-Principal
```
## Evaluation model
Policies are evaluated in declaration order. For each policy:
1. Skip if `enabled` is false.
2. Evaluate all match expressions. All expressions must match (AND semantics). If any expression does not match, the policy is skipped.
3. Execute the policy. If it rejects the request (invalid key, rate limited, insufficient permissions), evaluation stops and Frontline returns a structured error response.
4. If the policy is an auth policy and succeeds, it sets the `Principal` for the request. Subsequent auth policies are skipped.
Unknown policy types are silently skipped for forward compatibility. This means a Frontline instance running an older version can load a config that references a new policy type without breaking.
## Composability
Policies are composable building blocks. The policy list is not a set of independent checks but an ordered pipeline where earlier policies can establish context that later policies consume.
### Auth sets context for downstream policies
Auth policies produce a [Principal](/architecture/services/frontline/policies/principal) containing a subject and a method-specific `source` object. Today only KeyAuth is implemented; JWTAuth is defined in the protobuf schema but not yet executed by the engine (see [implementation status](#implementation-status)). Downstream policies reference this principal. For example, a RateLimit policy can use the authenticated subject as its bucket key, giving each user their own rate limit window instead of a shared one.
```
Policy 1: KeyAuth → authenticates the request, sets Principal
Policy 2: RateLimit → rate limits per Principal.subject (not yet implemented)
```
The examples in this section illustrate the composability model. Only KeyAuth executes today; other policy types are shown to demonstrate the design.
### Same policy type, different match expressions
The same policy type can appear multiple times with different [match expressions](/architecture/services/frontline/policies/match-expressions). This enables path-specific or method-specific rules without requiring a single policy to handle every case.
For example, applying different rate limits to different paths:
```
Policy 1: RateLimit match: /v1/expensive/* → 10 req/min
Policy 2: RateLimit match: /v1/* → 1000 req/min
```
Policies are evaluated in order and all matching policies execute until one rejects. A request to `/v1/expensive/foo` matches both policies, so both rate limits apply independently. If the stricter limit rejects, evaluation stops and the broader policy never runs. Place more specific match expressions before broader ones so a rejection skips unnecessary work.
### Layering multiple concerns
A typical production configuration layers firewall, auth, rate limiting, and validation:
```
Policy 1: Firewall match: path prefix /admin → DENY admin routes from public traffic
Policy 2: KeyAuth match: all → authenticate, set Principal
Policy 3: RateLimit match: /v1/* → 1000 req/min per subject
Policy 4: RateLimit match: /v1/expensive/* → 10 req/min per subject
Policy 5: OpenAPI match: all → validate request shape
```
Each policy can reject the request independently. A blocked path never reaches auth. An invalid key never reaches rate limiting. A rate-limited request never reaches schema validation. Order matters.
## Full config example
The `sentinel_config` column on a deployment stores a JSON-serialized `frontline.v1.Config` protobuf. Here is a complete example that blocks `/admin`, authenticates with KeyAuth, and applies two rate limit tiers:
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "block-admin",
"name": "Block /admin",
"enabled": true,
"match": [
{ "path": { "path": { "prefix": "/admin" } } }
],
"firewall": { "action": "ACTION_DENY" }
},
{
"id": "api-auth",
"name": "Authenticate API keys",
"enabled": true,
"match": [],
"keyauth": {
"key_space_ids": ["ks_abc123"],
"locations": [
{ "bearer": {} }
],
"permission_query": "api.read"
}
},
{
"id": "search-ratelimit",
"name": "Strict limit on search",
"enabled": true,
"match": [
{ "path": { "path": { "prefix": "/v1/search" } } },
{ "method": { "methods": ["GET"] } }
],
"ratelimit": {
"limit": 10,
"window_ms": 60000,
"key": { "authenticated_subject": {} }
}
},
{
"id": "global-ratelimit",
"name": "Default rate limit",
"enabled": true,
"match": [
{ "path": { "path": { "prefix": "/v1/" } } }
],
"ratelimit": {
"limit": 1000,
"window_ms": 60000,
"key": { "authenticated_subject": {} }
}
}
]
}
```
What happens for `GET /v1/search?q=test` with a valid API key:
1. `block-admin` match does not apply (path is `/v1/search`, not `/admin`), so the policy is skipped.
2. `api-auth` runs. Extracts the Bearer token, verifies it against keyspace `ks_abc123`, checks the `api.read` permission, and sets the Principal.
3. `search-ratelimit` runs. Both match expressions pass (path starts with `/v1/search` AND method is GET). Rate limits the request at 10/min using the authenticated subject as the bucket key.
4. `global-ratelimit` match also passes (path starts with `/v1/`), but the request was already rate-limited by policy 3. Both policies evaluate independently.
What happens for `POST /v1/keys` with an invalid key:
1. `block-admin` is skipped (path does not match).
2. `api-auth` rejects the request with 401. Evaluation stops. Policies 3 and 4 never run.
## Implementation status
The engine executes only KeyAuth. Other policy types are defined in the protobuf schema and can be configured, but the engine skips them at evaluation time.
| Policy | Implemented |
| ----------------------------------------------------------------------- | ----------- |
| [KeyAuth](/architecture/services/frontline/policies/keyauth) | Yes |
| [Firewall](/architecture/services/frontline/policies/firewall) | Yes |
| [JWTAuth](/architecture/services/frontline/policies/jwtauth) | Schema only |
| [RateLimit](/architecture/services/frontline/policies/ratelimit) | Schema only |
| [OpenAPI validation](/architecture/services/frontline/policies/openapi) | Schema only |
## Shared types
* [Policy schema](/architecture/services/frontline/policies/policy) for the `frontline.v1.Policy` message structure
* [Match expressions](/architecture/services/frontline/policies/match-expressions) for request matching rules
* [Principal](/architecture/services/frontline/policies/principal) for the authenticated identity shape
## Config parsing
The engine handles these edge cases when parsing `sentinel_config`:
| Input | Behavior |
| ------------------------ | --------------------------------------------------------- |
| `nil` or empty bytes | Pass-through (no policies) |
| `{}` (empty JSON object) | Pass-through (legacy compatibility) |
| Valid JSON with policies | Parse and evaluate |
| Invalid JSON | Error with code `Frontline.Internal.InvalidConfiguration` |
## Adding a new policy type
1. Define the policy proto in `svc/frontline/proto/frontline/policies/v1/`.
2. Add the new type to the `config` oneof in `policy.proto`.
3. Add a case in the evaluation switch in `svc/frontline/internal/policies/engine.go`.
4. Implement the executor (follow `keyauth.go` as a reference).
5. Run `mise run generate` to regenerate protobuf code.
Old Frontline versions that do not recognize the new policy type skip it silently, so rollouts are safe.
# JWTAuth
Source: https://engineering.unkey.com/architecture/services/frontline/policies/jwtauth
JWT authentication policy (schema only)
JWTAuth validates Bearer JSON Web Tokens using JWKS, an OIDC issuer, or a static public key, and produces a principal on success.
JWTAuth is defined in the policy schema but is not executed by the policy engine yet.
## Fields
URL of the JWKS endpoint for token verification.
OIDC issuer URL. Frontline discovers the JWKS URI from the issuer's `.well-known/openid-configuration`.
PEM-encoded public key for token verification. Use this for static key pairs.
Required `iss` claim value.
Allowed `aud` claim values.
Allowed signing algorithms.
Claim used as the principal subject. Defaults to `sub`.
When true, requests without a token are allowed through without setting a principal.
Tolerance for time-based claim validation (`exp`, `nbf`, `iat`), in milliseconds.
How long to cache the JWKS response, in milliseconds.
## Examples
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "jwt-auth",
"name": "Validate JWTs via OIDC",
"enabled": true,
"match": [],
"jwtauth": {
"oidc_issuer": "https://auth.example.com",
"audiences": ["api.example.com"],
"algorithms": ["RS256"]
}
}
]
}
```
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "jwt-auth",
"name": "Validate JWTs via JWKS",
"enabled": true,
"match": [],
"jwtauth": {
"jwks_uri": "https://auth.example.com/.well-known/jwks.json",
"issuer": "https://auth.example.com",
"audiences": ["api.example.com"],
"algorithms": ["RS256"],
"jwks_cache_ms": 3600000
}
}
]
}
```
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "jwt-auth-optional",
"name": "Optional JWT auth",
"enabled": true,
"match": [],
"jwtauth": {
"oidc_issuer": "https://auth.example.com",
"audiences": ["api.example.com"],
"allow_anonymous": true
}
}
]
}
```
Requests without a token pass through without a Principal. Requests with an invalid token are rejected.
# KeyAuth
Source: https://engineering.unkey.com/architecture/services/frontline/policies/keyauth
API key authentication policy
KeyAuth authenticates requests using Unkey API keys. It is the only authentication policy the engine executes today; JWTAuth is defined in the schema but not yet active.
## Fields
List of keyspace IDs the key must belong to. If the key belongs to a keyspace not in this list, authentication fails with `Frontline.Auth.InvalidKey`.
Ordered list of locations to extract the API key from. Frontline tries each location in order and uses the first non-empty key. If omitted, defaults to extracting a Bearer token from the `Authorization` header.
Optional RBAC query evaluated against the key's permissions. If the key does not satisfy the query, authentication fails with `Frontline.Auth.InsufficientPermissions`.
Optional list of rate limits to enforce on the verified key, mirroring the `ratelimits` field of the verifyKey API. Each entry references a rate limit by `name`. This is in addition to any auto-applied limits on the key or its identity, which are always enforced. Each entry may optionally override the `limit`, `duration` (milliseconds), and `cost`. Supplying both `limit` and `duration` defines an inline limit that does not need to exist on the key. If a named limit does not exist and no inline `limit`/`duration` is provided, the request is rejected.
Optional override for how many usage credits a matching request deducts from the verified key, mirroring the `credits.cost` behavior of the verifyKey API. Defaults to `1` when unset. Set to `0` to verify the key (and evaluate permissions and rate limits) without spending credits, for example on read-only routes or gateways that only prove the key is valid before proxying. Keys with unlimited remaining usage are unaffected. Must be non-negative; when the cost exceeds the key's remaining credits, the request fails with `Frontline.Auth.UsageExceeded`.
## Examples
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "api-auth",
"name": "Authenticate API keys",
"enabled": true,
"match": [],
"keyauth": {
"key_space_ids": ["ks_abc123"],
"locations": [
{ "bearer": {} }
]
}
}
]
}
```
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "api-auth",
"name": "Authenticate via X-API-Key header",
"enabled": true,
"match": [],
"keyauth": {
"key_space_ids": ["ks_abc123"],
"locations": [
{ "header": { "name": "X-API-Key" } }
]
}
}
]
}
```
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "api-auth",
"name": "Authenticate with prefix stripping",
"enabled": true,
"match": [],
"keyauth": {
"key_space_ids": ["ks_abc123"],
"locations": [
{ "header": { "name": "Authorization", "strip_prefix": "ApiKey " } }
]
}
}
]
}
```
Placing API keys in query parameters is dangerous and should only be used as a last resort for trusted-internal services. Query strings are logged by proxies, CDNs, and web servers, cached by browsers and intermediaries, saved in browser history, and leaked to third parties via the `Referer` header. Prefer the `Authorization` header (`bearer` or `header` location) whenever possible.
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "api-auth",
"name": "Authenticate via query param",
"enabled": true,
"match": [],
"keyauth": {
"key_space_ids": ["ks_abc123"],
"locations": [
{ "query_param": { "name": "api_key" } }
]
}
}
]
}
```
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "api-auth",
"name": "Authenticate with RBAC",
"enabled": true,
"match": [],
"keyauth": {
"key_space_ids": ["ks_abc123"],
"locations": [{ "bearer": {} }],
"permission_query": "(api.keys.read OR api.keys.list) AND billing.read"
}
}
]
}
```
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "api-auth",
"name": "Authenticate and enforce key rate limits",
"enabled": true,
"match": [{ "path": { "exact": "/api/v2/getAll" } }],
"keyauth": {
"key_space_ids": ["ks_abc123"],
"locations": [{ "bearer": {} }],
"ratelimits": [
{ "name": "expensive" },
{ "name": "burst", "limit": 10, "duration": 1000 }
]
}
}
]
}
```
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "api-auth",
"name": "Verify the key without spending credits",
"enabled": true,
"match": [{ "path": { "prefix": "/readonly/" } }],
"keyauth": {
"key_space_ids": ["ks_abc123"],
"locations": [{ "bearer": {} }],
"credits": 0
}
}
]
}
```
## Key extraction
Frontline supports three key extraction locations:
| Location | Behavior |
| --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Bearer | Reads the `Authorization` header and strips the `Bearer ` prefix (case-insensitive) |
| Header | Reads a named header. Optionally strips a prefix (case-insensitive) |
| Query parameter | Reads a named query parameter. **Security risk:** query strings are recorded in server logs, proxy logs, CDN logs, browser history, and `Referer` headers. Use only as a last resort for trusted-internal traffic; prefer `bearer` or `header` locations. |
When multiple locations are configured, Frontline tries each in order and uses the first non-empty result.
## Verification flow
1. Extract the key from the request using configured locations.
2. Hash the key using SHA-256.
3. Look up the hash in the key cache (fresh: 10s, stale: 10min, max: 100k entries).
4. Validate key status. Keys that are not found, disabled, expired, or belong to a disabled workspace are rejected.
5. Verify the key belongs to one of the configured `key_space_ids`.
6. Parse the permission query (if configured) and build verify options, including any configured key `ratelimits`.
7. Call `verifier.Verify()`, deducting the policy's `credits` cost (default `1`, `0` skips credit spending) per request. Auto-applied key/identity limits and any policy-configured `ratelimits` are enforced here, using the same path as the verifyKey API.
8. Write rate limit headers (regardless of success or failure).
9. Check post-verification status (rate limit, usage exceeded, permissions).
10. Build and return the principal on success.
## Response headers
KeyAuth writes rate limit headers on every response, including rejected requests:
| Header | Value |
| ----------------------- | ------------------------------------------------------------- |
| `X-RateLimit-Limit` | Rate limit ceiling |
| `X-RateLimit-Remaining` | Remaining requests in the window |
| `X-RateLimit-Reset` | Unix timestamp when the window resets |
| `Retry-After` | Seconds until retry (only on 429 responses, minimum 1 second) |
## Error responses
| Scenario | Status | Code |
| ------------------------------------------- | ------ | ----------------------------------------- |
| No key found in request | 401 | `Frontline.Auth.MissingCredentials` |
| Key not found in database | 401 | `Frontline.Auth.InvalidKey` |
| Key disabled | 401 | `Frontline.Auth.InvalidKey` |
| Key expired | 401 | `Frontline.Auth.InvalidKey` |
| Key not in allowed keyspace | 401 | `Frontline.Auth.InvalidKey` |
| Workspace disabled | 401 | `Frontline.Auth.InvalidKey` |
| Permission query not satisfied | 403 | `Frontline.Auth.InsufficientPermissions` |
| Rate limit exceeded | 429 | `Frontline.Auth.RateLimited` |
| Usage limit exceeded | 429 | `Frontline.Auth.UsageExceeded` |
| Configured `ratelimits` name missing on key | 500 | `Frontline.Internal.InternalServerError` |
| Invalid permission query syntax | 500 | `Frontline.Internal.InvalidConfiguration` |
# Match expressions
Source: https://engineering.unkey.com/architecture/services/frontline/policies/match-expressions
Request matching rules for policies
Match expressions define which requests a policy applies to. A policy carries a list of `MatchExpr` entries. All entries must match for the policy to execute (AND semantics). An empty list matches all requests.
## AND vs OR
Within a single policy, match expressions are combined with AND. A policy with a path match and a method match only runs when both conditions are true.
There is no built-in OR operator. To express OR, create multiple policies with the same config and different match lists. For example, to apply different rate limits to different parts of an API:
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"match": [
{ "path": { "path": { "prefix": "/v1/search" } } }
],
"ratelimit": { "limit": 10, "window_ms": 60000 }
},
{
"match": [
{ "path": { "path": { "prefix": "/v1/keys" } } }
],
"ratelimit": { "limit": 1000, "window_ms": 60000 }
}
]
}
```
Each policy is evaluated independently in order. A request to `/v1/search` matches policy 1 and gets the stricter limit. A request to `/v1/keys` skips policy 1 (path does not match) and hits policy 2.
This approach is simpler to reason about than a recursive expression tree and covers the vast majority of routing needs. The proto schema is designed so that combinators (And/Or/Not) can be added later as new oneof branches without breaking the wire format.
## Matcher types
### Path
Matches against `request.URL.Path` using a string match. The path is compared without the query string. Patterns must include the leading slash.
| Comparison | Behavior |
| ---------- | --------------------------------------------------------------------------------------------------------------- |
| Exact | Case-sensitive by default. Set `ignore_case` to match case-insensitively. |
| Prefix | Case-sensitive by default. Set `ignore_case` to match case-insensitively. |
| Regex | Uses Go's RE2 engine. Patterns are compiled once and cached. Set `ignore_case` to wrap the pattern with `(?i)`. |
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"match": [
{ "path": { "path": { "prefix": "/v1/" } } }
]
// ...omitted
}
]
}
```
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"match": [
{ "path": { "path": { "regex": "^/v[0-9]+/keys/[^/]+$" } } }
]
// ...omitted
}
]
}
```
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"match": [
{ "path": { "path": { "exact": "/healthcheck", "ignore_case": true } } }
]
// ...omitted
}
]
}
```
### Method
Matches against the HTTP method. Comparison is always case-insensitive per the HTTP specification.
Multiple methods can be listed, and the request matches if it uses any of them (OR semantics). An empty method list matches all methods.
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"match": [
{ "method": { "methods": ["POST", "PUT", "DELETE"] } }
]
// ...omitted
}
]
}
```
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"match": [
{ "path": { "path": { "prefix": "/v1/" } } },
{ "method": { "methods": ["POST", "PUT", "DELETE"] } }
]
// ...omitted
}
]
}
```
### Header
Matches against request headers. Header names are matched case-insensitively per HTTP specification. When a header has multiple values, the match succeeds if any value matches (OR semantics).
Header name. Matched case-insensitively.
Either `present` (bool, checks header existence) or `value` (string match against header values).
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"match": [
{ "header": { "name": "Authorization", "present": true } }
]
// ...omitted
}
]
}
```
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"match": [
{ "header": { "name": "X-API-Version", "value": { "exact": "2024-01-01" } } }
]
// ...omitted
}
]
}
```
### Query parameter
Matches against URL query parameters. Parameter names are matched case-sensitively. When a parameter has multiple values, the match succeeds if any value matches (OR semantics).
Parameter name. Matched case-sensitively.
Either `present` (bool, checks parameter existence) or `value` (string match against parameter
values).
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"match": [
{ "query_param": { "name": "debug", "present": true } }
]
// ...omitted
}
]
}
```
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"match": [
{ "query_param": { "name": "version", "value": { "exact": "beta" } } }
]
// ...omitted
}
]
}
```
## String match types
All string comparisons support three modes:
| Mode | Behavior |
| -------- | ----------------------------------------------------------------------------------------- |
| `exact` | Full string equality. Supports `ignore_case`. |
| `prefix` | Matches if the string starts with the given prefix. Supports `ignore_case`. |
| `regex` | Regular expression match using Go's RE2 syntax. Supports `ignore_case` (prepends `(?i)`). |
## Regex caching
Compiled regular expressions are cached in a thread-safe map keyed by the pattern string. The first evaluation of a regex pattern compiles and caches it. Subsequent evaluations reuse the compiled pattern.
# OpenAPI validation
Source: https://engineering.unkey.com/architecture/services/frontline/policies/openapi
OpenAPI request validation
OpenAPI validation rejects requests that do not conform to an OpenAPI 3.0 or 3.1 specification before they reach the upstream instance. The policy engine compiles the spec into a validator and validates each matched request against it.
## Fields
The OpenAPI specification as raw YAML bytes. Frontline parses the spec at evaluation time and validates the request path, method, parameters, and request body against it.
## Example
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "schema-validation",
"name": "Validate requests against OpenAPI spec",
"enabled": true,
"match": [
{ "path": { "path": { "prefix": "/v1/" } } }
],
"openapi": {
"spec_yaml": "openapi: '3.0.3'\ninfo:\n title: My API\n version: '1.0'\npaths:\n /v1/keys:\n post:\n requestBody:\n required: true\n content:\n application/json:\n schema:\n type: object\n required: [name]\n properties:\n name:\n type: string\n"
}
}
]
}
```
# Policy schema
Source: https://engineering.unkey.com/architecture/services/frontline/policies/policy
The frontline.v1.Policy message structure
The `frontline.v1.Policy` message is the unit of middleware configuration. Each policy combines match expressions with exactly one policy configuration.
## Fields
Stable identifier used in logs, metrics, and troubleshooting. Must be unique within a deployment's policy list.
Human-readable label for display in the dashboard and logs.
When false, Frontline skips this policy during evaluation. Defaults to true if not set. This lets you disable a misbehaving policy during an incident without removing it from the config.
List of match expressions. All entries must match for the policy to execute (AND semantics). An empty list matches all requests. See [match expressions](/architecture/services/frontline/policies/match-expressions).
Exactly one policy configuration. Options: `keyauth`, `jwtauth`, `ratelimit`, `firewall`, `openapi`.
## Example
A minimal policy that authenticates all requests:
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "api-auth",
"name": "Authenticate all requests",
"enabled": true,
"match": [],
"keyauth": {
"key_space_ids": ["ks_abc123"],
"locations": [{ "bearer": {} }]
}
}
]
}
```
## Evaluation behavior
* Policies are evaluated in declaration order.
* If a policy rejects the request, evaluation stops immediately.
* Unknown `config` types are skipped (forward compatibility).
* Disabled policies are skipped without evaluating match expressions.
# Principal
Source: https://engineering.unkey.com/architecture/services/frontline/policies/principal
Authenticated identity from auth policies
`engine.Principal` is the shared identity shape produced by authentication policies. It decouples the authentication mechanism from downstream authorization decisions and from upstream applications, which receive it as a JSON payload on the `X-Unkey-Principal` header.
The struct is hand-written with `encoding/json` rather than generated from protobuf. The Principal is output-only (Frontline produces it, apps consume the JSON), so proto-as-IDL buys nothing while fighting the JSON contract we want to expose.
The wire format is authoritative. For the full public reference, see the product docs .
## Fields
Schema version of the Principal payload. Currently `"v1"`. Bumped only on breaking changes to the JSON shape.
The primary identifier of the authenticated entity. For KeyAuth, this is the identity's external ID when the key is linked to an identity, otherwise the key ID. For JWTAuth, this is the configured subject claim (default `sub`).
Which authentication method produced this Principal. `API_KEY` or `JWT`. Always matches the populated variant of `source`.
The Unkey identity linked to the credential, when present. Absent from the JSON entirely when no identity is linked — never `null` and never an empty object.
Discriminated union over method-specific detail. Contains exactly one populated variant matching `type`: `source.key` for API keys, `source.jwt` for JWT. The variant name is the lowercase of the type (minus the underscore).
## What a principal looks like
```json theme={"theme":"kanagawa-wave"}
{
"version": "v1",
"subject": "user_abc123",
"type": "API_KEY",
"identity": {
"externalId": "user_abc123",
"meta": { "plan": "pro" }
},
"source": {
"key": {
"keyId": "key_xyz",
"keySpaceId": "ks_abc123",
"name": "ACME Production",
"expiresAt": 1717200000000,
"credits": 4900,
"meta": {},
"roles": ["admin"],
"permissions": ["api.read", "api.write"]
}
}
}
```
```json theme={"theme":"kanagawa-wave"}
{
"version": "v1",
"subject": "auth0|abc123",
"type": "JWT",
"source": {
"jwt": {
"header": { "alg": "RS256", "typ": "JWT", "kid": "key-1" },
"payload": {
"iss": "https://acme.com",
"sub": "auth0|abc123",
"aud": "api.acme.com",
"exp": 1711310400,
"email": "user@acme.com",
"org_id": "org_456"
},
"signature": "dBjftJeZ4CVP-mB92K27uhbUJU1p1r_wW..."
}
}
}
```
## KeyAuth source fields
When produced by a KeyAuth policy, `source.key` carries the verified key detail. Fields marked optional are omitted from the JSON when unset.
| Field | Optional | Description |
| ------------- | -------- | --------------------------------------------------------------------------------- |
| `keyId` | | The ID of the verified key. |
| `keySpaceId` | | The keyspace the key belongs to. |
| `name` | yes | Human-readable key name, when set. |
| `expiresAt` | yes | Unix timestamp in milliseconds (JSON number). Omitted when the key has no expiry. |
| `credits` | yes | Credits remaining after this request. Omitted for unlimited keys; `0` is emitted. |
| `meta` | | Custom key metadata. Always emitted, `{}` when empty. |
| `roles` | yes | Raw RBAC role names. Omitted when empty. |
| `permissions` | yes | Raw RBAC permissions. Omitted when empty. |
## Header propagation
Frontline serializes the principal to JSON and sets it on the `X-Unkey-Principal` header before forwarding to the instance. The instance reads this header to make authorization decisions without re-verifying the credential.
The proxy handler strips any incoming `X-Unkey-Principal` header before policy evaluation to prevent clients from spoofing an authenticated identity.
# RateLimit
Source: https://engineering.unkey.com/architecture/services/frontline/policies/ratelimit
Gateway rate limiting policy
RateLimit defines gateway-level rate limiting with configurable identifiers. Frontline executes RateLimit policies and delegates counter state to [rate limiting](/architecture/ratelimiting/overview), so policy execution uses the same distributed counters as API and Frontline rate-limit checks.
## Fields
Maximum number of requests allowed in the time window.
Time window in milliseconds. For example, `limit: 100` with `window_ms: 60000` means 100 requests per minute.
Ordered list of 1 to 5 sources that form the rate limit key. Each unique
combination of resolved values gets its own counter with the same limit and
window.
Deprecated single-source form. Frontline still reads it from policies stored
before compound identifiers existed; the API normalizes all writes to
`identifiers`.
## Examples
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "global-ratelimit",
"name": "Rate limit by IP",
"enabled": true,
"match": [],
"ratelimit": {
"limit": 1000,
"window_ms": 60000,
"identifiers": [{ "remote_ip": {} }]
}
}
]
}
```
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "user-ratelimit",
"name": "Rate limit per user",
"enabled": true,
"match": [],
"ratelimit": {
"limit": 500,
"window_ms": 60000,
"identifiers": [{ "authenticated_subject": {} }]
}
}
]
}
```
Requires a KeyAuth or JWTAuth policy earlier in the list to set the Principal.
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "tenant-ratelimit",
"name": "Rate limit per tenant",
"enabled": true,
"match": [],
"ratelimit": {
"limit": 5000,
"window_ms": 60000,
"identifiers": [{ "header": { "name": "X-Tenant-Id" } }]
}
}
]
}
```
Use only when the header is set by a trusted upstream proxy.
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "org-ratelimit",
"name": "Rate limit per organization",
"enabled": true,
"match": [],
"ratelimit": {
"limit": 10000,
"window_ms": 60000,
"identifiers": [{ "principal_field": { "path": "source.key.meta.org_id" } }]
}
}
]
}
```
Creates a shared bucket for all keys that resolve to the same `org_id` meta value. The path is a dotted route into the Principal JSON -- for JWT-authenticated traffic you might use `source.jwt.payload.org_id` instead.
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "path-ratelimit",
"name": "Rate limit per endpoint",
"enabled": true,
"match": [
{ "path": { "path": { "prefix": "/v1/" } } }
],
"ratelimit": {
"limit": 100,
"window_ms": 60000,
"identifiers": [{ "path": {} }]
}
}
]
}
```
Creates a separate bucket per URL path, protecting expensive endpoints without a separate policy for each.
```json theme={"theme":"kanagawa-wave"}
{
"policies": [
{
"id": "subject-path-ratelimit",
"name": "Rate limit per user per endpoint",
"enabled": true,
"match": [],
"ratelimit": {
"limit": 100,
"window_ms": 60000,
"identifiers": [
{ "authenticated_subject": {} },
{ "path": {} }
]
}
}
]
}
```
A compound key: each subject gets a separate counter on each path. Parts join with `:` after escaping, so tuple boundaries stay unambiguous.
## Identifier sources
| Source | Description |
| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------ |
| `remote_ip` | Client IP address. Effective for anonymous traffic, but can over-limit behind shared NATs. |
| `header` | Value of a named request header. Only use behind trusted proxies that set the header. |
| `authenticated_subject` | Principal subject from an upstream auth policy. Most accurate for authenticated APIs. |
| `path` | Request URL path. Creates a separate bucket per endpoint. |
| `principal_field` | Value resolved from a dotted path into the Principal JSON (for example, `source.key.meta.org_id` for per-organization limits). |
# Request flow
Source: https://engineering.unkey.com/architecture/services/frontline/request-flow
Request lifecycle through Frontline, from TLS termination to the proxied response
This page traces a request from the moment Frontline accepts the TLS connection to the response being streamed back to the client. Frontline owns the full path: it resolves the hostname, evaluates the deployment's policies inline, and proxies directly to a running instance.
```mermaid theme={"theme":"kanagawa-wave"}
sequenceDiagram
participant C as Client
participant MW as Middleware Chain
participant H as Proxy Handler
participant R as Router
participant E as Policy Engine
participant I as Instance
participant RF as Remote Frontline
C->>MW: HTTPS request
MW->>MW: Panic recovery, strip reserved headers, logging, ClickHouse, observability
MW->>H: Request
H->>H: Verify and remove peer metadata
H->>R: Route(hostname)
R-->>H: RouteDecision (instances + parsed policies)
alt Remote Frontline selected
H->>RF: Forward with signed metadata (HTTPS)
RF->>RF: Verify metadata and read hop history
RF-->>H: Response
else Local instance selected
opt Deployment has policies and engine is configured
H->>E: Evaluate policies
alt Policy rejects
E-->>H: Error (401/403/429)
H-->>MW: Error response
MW-->>C: JSON error + X-Unkey-Error-Source
else Policy succeeds
E-->>H: Principal
H->>H: Set X-Unkey-Principal header
end
end
H->>I: Forward request with X-Forwarded-* headers
I-->>H: Response
end
H-->>MW: Response + timing headers
MW-->>C: Final response (metrics, tracing, ClickHouse write)
```
## Middleware chain
Every request passes through a middleware chain before reaching the proxy handler. The chain executes top-to-bottom on the request path and bottom-to-top on the response path. The order is set in [`svc/frontline/routes/register.go`](https://github.com/unkeyed/unkey/blob/main/svc/frontline/routes/register.go):
1. **PanicRecovery.** Catches panics in downstream handlers so a single request cannot crash the process.
2. **Reserved header strip.** Removes reserved headers, including `X-Unkey-Principal`, so a client cannot forge an identity. It removes every `X-Unkey-*` trailer. It preserves the `X-Unkey-Frontline-Meta` request header for verification in the proxy handler.
3. **Logging.** Structured request logging. Skips internal paths (`/_unkey/internal/`).
4. **ClickHouseLogging.** Creates a tracking context with a start timestamp and, on completion, writes the full request and response to ClickHouse. It wraps observability so it reads the final status code after observability has written the response.
5. **Observability.** Starts an OpenTelemetry span (`frontline.proxy`), records Prometheus metrics (`unkey_frontline_requests_total`), maps fault codes to HTTP status codes, and renders an HTML error page when the client prefers HTML.
6. **Timeout.** Enforces the configured request timeout.
## Proxy handler
After the middleware chain, the proxy handler runs. Its inputs are the router service, the proxy service, and the policy engine.
### 1. Verify peer metadata
The handler checks for `X-Unkey-Frontline-Meta`. If the header is present, the
handler removes it and verifies the PASETO v4.public token. A request without
valid metadata starts with an empty hop history. Frontline treats empty,
duplicate, invalid, expired, or oversized metadata as absent. Invalid metadata
never blocks a request.
### 2. Route the hostname
The handler calls `Route(hostname)` on the router. The router resolves the hostname to a `frontline_route` (deployment ID, `sentinel_config`, upstream protocol), parses the `sentinel_config` bytes into a policy list, and selects a destination. For a local destination the decision carries the running instances in shuffled order plus the parsed policies. If the hostname has no configured route, or the deployment has no running instance in any reachable region, the router returns a `Frontline.Routing` error (for example `NoRunningInstances`, surfaced as 503).
### 3. Evaluate policies
When the deployment has policies and the engine is configured, the engine evaluates each policy in order against the request. A policy that rejects the request (invalid key, rate limited, insufficient permissions) produces an error response before any byte is proxied. On success, the first authentication policy yields a `Principal`, which the handler serializes to the `X-Unkey-Principal` header. When the deployment has no policies, the request is forwarded without policy evaluation.
### 4. Forward the request
The handler proxies to a running instance of the deployment in the same region, attempting the shuffled candidates in order and advancing on dial failures. When every local instance fails and a peer region has a healthy instance, the request falls through to a peer Frontline. The following headers are set on the proxied request:
| Header | Value |
| ------------------- | --------------------------------------------- |
| `X-Forwarded-For` | Client IP |
| `X-Forwarded-Host` | Original request host |
| `X-Forwarded-Proto` | `http` (TLS is terminated at Frontline) |
| `X-Unkey-Principal` | JSON-serialized principal (if auth succeeded) |
Before a cross-region forward, Frontline sets `exp` to 1 minute in the future
and appends a signed hop entry. The entry contains the region, request ID,
Frontline ID, and forward time as Unix milliseconds. The hop history length is
the hop count. Frontline rejects the forward when the length reaches `max_hops`.
### 5. Stream and record the response
The handler streams the instance response back to the client. The ClickHouse logging middleware records the request and response (status, headers, and a size-capped body) and the timing breakdown for analytics.
## Headers reference
### Set on cross-region forwards
| Header | Purpose |
| ------------------------ | ------------------------------------- |
| `X-Deployment-Id` | Identifies the request deployment |
| `X-Unkey-Frontline-Meta` | Carries signed expiry and hop history |
`X-Unkey-Frontline-Meta` contains a PASETO v4.public token. Frontline accepts the
metadata only when the request has one non-empty header value, the signature is
valid, and the token has not expired. Each forwarding Frontline appends one hop,
replaces the metadata, and sets `exp` to 1 minute in the future. Each hop records
the region, request ID, Frontline ID, and forward time. Frontline removes the
metadata before it forwards a request to a deployment instance. The complete
header value cannot exceed 4,096 bytes.
Frontline removes invalid metadata and treats it as absent for all requests.
This behavior prevents a client-controlled header from blocking service.
### Forwarded to the instance
| Header | Purpose |
| ------------------- | -------------------------------------------- |
| `X-Forwarded-For` | Client IP address |
| `X-Forwarded-Host` | Original request hostname |
| `X-Forwarded-Proto` | Always `http` (TLS terminated at Frontline) |
| `X-Unkey-Principal` | JSON principal from a successful auth policy |
| `X-Unkey-Region` | Region of the serving Frontline instance |
### Returned to the client
| Header | Purpose |
| ----------------------- | -------------------------------------------- |
| `X-Unkey-Latency` | Latency breakdown (Frontline and instance) |
| `X-Unkey-Error-Source` | Identifies Frontline as the source on errors |
| `X-RateLimit-Limit` | Rate limit ceiling from policies |
| `X-RateLimit-Remaining` | Remaining requests in window |
| `X-RateLimit-Reset` | Unix timestamp when the window resets |
| `Retry-After` | Seconds until retry (only on 429 responses) |
# Routing and failover
Source: https://engineering.unkey.com/architecture/services/frontline/routing
Frontline routing decisions and cross-region forwarding
Frontline routes a request by looking up its hostname, selecting a healthy instance of the target deployment, and proxying locally or forwarding to another region.
Key components:
* Router ([`svc/frontline/internal/router`](https://github.com/unkeyed/unkey/blob/main/svc/frontline/internal/router)).
* Proxy handler ([`svc/frontline/routes/proxy`](https://github.com/unkeyed/unkey/blob/main/svc/frontline/routes/proxy)).
## Flow: route request
```mermaid theme={"theme":"kanagawa-wave"}
sequenceDiagram
actor Client
participant Frontline as Frontline
participant Router as Router
participant Instance as Instance
participant Remote as Remote Frontline
Client->>Frontline: HTTPS request
Frontline->>Router: Route(hostname)
Router-->>Frontline: RouteDecision
alt local instance available
Frontline->>Instance: Evaluate policies, then proxy (HTTP)
else no local instance
Frontline->>Remote: Forward with signed metadata (HTTPS)
end
```
## Routing decisions
Frontline uses deployment state and regional proximity to select a destination.
* Frontline looks up the route by FQDN in the database and parses the deployment's policies.
* If the deployment has a running instance in the current region, it proxies locally, trying instances in shuffled order.
* If not, it selects the nearest region with a running instance using the region proximity list.
## Cross-region forwarding
When forwarding to another region, Frontline targets:
```plaintext theme={"theme":"kanagawa-wave"}
https://frontline..
```
The original hostname is preserved so the remote Frontline can perform TLS termination, policy evaluation, and instance selection.
Frontline sends `X-Unkey-Frontline-Meta` to the peer. The header contains an
PASETO v4.public token with an expiry time and an ordered hop history. Each hop
contains the region, request ID, Frontline ID, and forward time as Unix
milliseconds. The complete header value cannot exceed 4,096 bytes.
The following flow shows how Frontline handles the metadata.
```mermaid theme={"theme":"kanagawa-wave"}
flowchart TD
request["Receive request"] --> header{"Metadata present?"}
header -- No --> zero["Use empty hop history"]
header -- Yes --> remove["Remove metadata header"]
remove --> verify{"One value with valid signature, expiry, and size?"}
verify -- No --> zero
verify -- Yes --> read["Read signed hop history"]
zero --> route["Route request"]
read --> route
route --> destination{"Destination?"}
destination -- Instance --> strip["Forward without metadata"]
destination -- Peer --> limit{"History length at hop limit?"}
limit -- Yes --> reject["Return routing limit error"]
limit -- No --> sign["Append current hop
Instance identifier for logs and tracing.
Identity of the infrastructure cell managed by this Krane agent.
Cell identifier, such as `cell001`.
Infrastructure provider, such as `aws`.
Geographic region, such as `us-east-1`.
RPC server port.
Registry credentials. The krane runtime does not read this config today.
Registry URL.
Registry username.
Registry password.
Vault connection for the secrets service.
Vault URL.
Vault token.
Control plane connection.
Control API URL.
Control API token.
Tracing, logging, and metrics configuration.
Trace sampling rate.
Log sampling rate.
Slow log threshold.
Prometheus port. Set to 0 to disable.
## Example configuration
```toml theme={"theme":"kanagawa-wave"}
instance_id = "${POD_NAME}"
rpc_port = 8080
[cluster]
cell_id = "${UNKEY_CELL_ID}"
platform = "aws"
region = "${UNKEY_REGION}"
[control]
url = "${UNKEY_CTRL_URL}"
token = "${UNKEY_CTRL_TOKEN}"
[vault]
url = "${UNKEY_VAULT_URL}"
token = "${UNKEY_VAULT_TOKEN}"
[registry]
url = "${UNKEY_REGISTRY_URL}"
username = "${UNKEY_REGISTRY_USERNAME}"
password = "${UNKEY_REGISTRY_PASSWORD}"
[observability.tracing]
sample_rate = 0.1
[observability.logging]
sample_rate = 0.01
slow_threshold = "2s"
[observability.metrics]
prometheus_port = 9090
```
# Deployment
Source: https://engineering.unkey.com/architecture/services/krane/deployment
Deployment model and failover expectations for the krane service
Krane runs as a single control agent for one Kubernetes cluster and one
control-plane cluster key. The cluster key is the `cell_id`, `platform`, and
`region` tuple that Krane sends to the control plane.
## Deployment model
Run one active Krane instance for each `(cell_id, platform, region)` tuple.
That instance owns reconciliation for the cluster. It watches desired state from
the control plane, applies Kubernetes resources, reports workload status, and
sends heartbeats.
Krane doesn't serve user traffic. Frontline serves traffic to user workloads, and
the control plane remains the source of desired deployment state.
## Replacement and downtime
Short Krane downtime is acceptable. During downtime, existing workloads keep
running, but new desired-state changes and status reports wait until Krane
reconnects.
Brief overlap during replacement is acceptable, ideally on the order of seconds.
Reconciliation is idempotent, and Krane applies Kubernetes resources from
control-plane desired state. After a restart, Krane reconnects to the change
stream and runs a full desired-state sync, so missed changes converge back to the
control-plane state.
## Production example
For an example of this model, see the Krane chart in Infra
and the Krane runtime config .
# Overview
Source: https://engineering.unkey.com/architecture/services/krane/overview
Kubernetes control agent for deployments and secrets
Krane is Unkey's in-cluster Kubernetes control agent. It reconciles control plane intent into Kubernetes resources, reports actual cluster state upstream, and brokers secrets decryption when Vault is configured.
Krane does not serve user traffic or make product decisions. It keeps Kubernetes state aligned with upstream intent.
## Place in the stack
Krane runs in each Kubernetes cluster. It is the only service in this stack with direct Kubernetes API credentials, which keeps cluster access isolated to Krane.
## Service boundaries
Krane only talks to three systems.
* Upstream: control plane streams desired state and receives status updates
* Downstream: Kubernetes API server for creating, updating, and watching resources
* Sidecar dependency: Vault for secrets decryption when enabled
Krane does not perform scheduling decisions, tenancy policy, or routing logic. Those live in the control plane and Frontline. Krane only reconciles Kubernetes resources and reports state.
## Core responsibilities
Krane is built around these core responsibilities.
* Reconcile user workloads as Kubernetes ReplicaSets
* Install a per-deployment Cilium network policy that admits Frontline ingress
* Report actual state for workloads upstream
* Decrypt workload secrets using Vault when enabled
## Control plane interface
Krane connects upstream with a Connect RPC client that keeps long-running streams open. It injects the `Authorization: Bearer ` header on every request and stamps a `ClusterKey` (cell ID, platform, and region) on every request proto message. h2c is supported for non-TLS URLs.
## Reconciliation model
```mermaid theme={"theme":"kanagawa-wave"}
flowchart TD
Ctrl[Control plane] -->|WatchDeploymentChanges| Watcher[Watcher]
Watcher --> DeployCtrl[Deployment controller]
DeployCtrl -->|Apply desired state| K8SDeploy[ReplicaSets]
DeployCtrl -->|Install per-deployment policy| K8SCilium[CiliumNetworkPolicy]
K8SDeploy -->|ReportDeploymentStatus| Ctrl
```
## Control loops
Krane consumes a single `WatchDeploymentChanges` stream from the control plane and dispatches each event to the deployment controller. The stream reconnects with jittered backoff between one and five seconds. A version cursor advances only after a state is applied successfully, which makes stream replay safe.
### Deployment controller
The deployment controller manages user workloads as Kubernetes ReplicaSets. It runs three loops.
* Desired state apply loop consumes deployment events from `WatchDeploymentChanges` and applies or deletes ReplicaSets
* Actual state report loop watches ReplicaSet events and reports status to the control plane
* Resync loop runs every minute and corrects drift by re-reading desired state
Applying a deployment also installs its [Cilium network policy](#cilium-network-policies) in the same step, so the policy is created and garbage-collected alongside the ReplicaSet.
## Kubernetes resource model
Krane uses server-side apply for all Kubernetes resources and labels everything it manages. Labels include `app.kubernetes.io/managed-by=krane` and a component label for selection.
### Deployments
User workloads are represented as ReplicaSets with the following characteristics.
* Namespaces are created on demand
* Pods run with `RuntimeClassName: gvisor` for isolation
* Pods select and tolerate `node-class=untrusted` nodes
* Topology spread keeps replicas balanced across zones
* Env vars include `PORT`, `UNKEY_DEPLOYMENT_ID`, `UNKEY_ENVIRONMENT_SLUG`, `UNKEY_REGION`, `UNKEY_INSTANCE_ID`, and the `UNKEY_GIT_*` set (commit SHA, branch, repo, commit message)
* A `command` override from the deployment spec replaces the image entrypoint when set; otherwise the image's `ENTRYPOINT`/`CMD` runs
* Decrypted environment variables are mounted from a per-deployment K8s Secret via `envFrom.secretRef`
* Healthchecks map to HTTP probes, and POST uses an exec probe with `wget`
* Optional preStop hook sends non-SIGTERM shutdown signals
### Cilium network policies
When the deployment controller applies a deployment, it also installs a `-frontline-ingress` CiliumNetworkPolicy in the deployment's namespace. Cilium default-deny applies to any endpoint a policy selects, so this policy is what admits ingress: it permits Frontline pods to reach the deployment's pods on the container port, and nothing else. Krane builds the policy from the deployment spec and applies it with the dynamic client using server-side apply. The policy is owned by the ReplicaSet, so Kubernetes garbage-collects it when the deployment is deleted.
## Consistency guarantees
Krane uses streaming desired state, Kubernetes watches, and a periodic resync to ensure eventual consistency. The resync loop lists all Krane-managed resources, queries the control plane for desired state, and applies or deletes resources when drift is detected.
# Secrets service
Source: https://engineering.unkey.com/architecture/services/krane/secrets
Secrets decryption RPC and authentication
Krane exposes a SecretsService for workloads. Injected pods call this service to decrypt their secrets blob using Vault.
Service definition: [`svc/krane/proto/krane/v1/secrets.proto`](https://github.com/unkeyed/unkey/blob/main/svc/krane/proto/krane/v1/secrets.proto).
## Authentication
Requests are authenticated with a Kubernetes service account token. Krane validates the token with the TokenReview API, resolves the pod from the token details, and verifies the pod labels match the deployment and environment IDs.
## DecryptSecretsBlob
JSON-encoded `ctrl.v1.SecretsConfig` bytes containing encrypted values.
Environment ID used as the Vault keyring.
Service account token for validation.
Deployment ID for token validation.
Decrypted environment variables.
## Decryption flow
The secrets blob contains a map of key names to Vault encrypted values. Krane decrypts each value separately using the environment ID as the keyring, then returns a map of plaintext environment variables.
# Overview
Source: https://engineering.unkey.com/architecture/services/logdrain/overview
How the logdrain service streams audit logs to customer destinations
The logdrain service (`svc/logdrain`) streams audit logs from ClickHouse to
customer-owned destinations: generic HTTP endpoints and Axiom.
Under one valid lease, it sends events in cursor order. It provides
at-least-once delivery. MySQL stores configuration and delivery progress. The
service does not use an external queue or workflow engine.
## How the service works
Each process runs a lease service and a delivery engine. The lease service
assigns drains to the process and keeps those assignments valid. The delivery
engine polls for assigned drains that are due, then gives them to a worker
pool.
A worker first confirms its assignment in MySQL. It then reads the drain's
configuration and cursor, queries the next audit log batch from ClickHouse,
and sends that batch to the customer destination. After the destination
acknowledges the batch, the worker advances the cursor in MySQL. A failed
delivery leaves the cursor unchanged.
MySQL stores the destination configuration as a serialized
`logdrain.v1.Config` protobuf. Its provider `oneof` keeps HTTP and Axiom fields
separate. Secret fields contain Vault ciphertext, so MySQL does not store
plaintext credentials. The Vault ciphertext includes the key ID that the
service needs for decryption. HTTP header names stay in plaintext. Each HTTP
header value has separate Vault ciphertext.
The `stream` oneof identifies the stream and contains its typed configuration,
independently of the destination. `audit_logs` selects `AuditLogStreamConfig`.
Its `event_types` list selects exact audit event names. An empty list means all
audit events, including event types added later. Existing protobuf configs with
no stream retain the legacy audit-log behavior without filters. The MySQL
`stream` column remains the scheduling index and must agree with the config.
Deploy workers that understand the stream config before enabling the dashboard
selector; older workers ignore it.
ClickHouse applies the filter before ordering and limiting each batch. Windows
with no matching events still advance the cursor. Editing the filter expires
the lease and resumes from the committed cursor with a new fencing token. It
does not replay events skipped by the previous filter. A request already in
flight can still reach the destination, but its old fence cannot commit.
The worker does not hold a MySQL transaction while it calls the customer
destination. A fencing token protects each state read and write instead.
```plaintext theme={"theme":"kanagawa-wave"}
svc/logdrain process
├── lease service
│ └── acquires and refreshes leases in MySQL
└── delivery engine
├── polls due leases from MySQL
└── workers
├── read config and cursor from MySQL
├── read audit logs from ClickHouse
├── send batches to the customer destination
└── write delivery state to MySQL and telemetry to ClickHouse
```
## Delivery cycle
On each poll, the delivery engine finds running drains assigned to its
lease ID that have reached `next_attempt_at`. It passes the drain ID and
fencing token to a worker. The worker reads the drain only if that token still
identifies a valid lease. If the lease expired or another process acquired it,
the worker stops.
The engine owns leases, delivery, and durable progress. A `batchReader` owns
source queries and adaptive windows. It returns a page containing events, a
proposed next cursor, and a `caughtUp` flag. Reading a page does not commit its
cursor. The engine commits only after the destination acknowledges all events
in that page. Empty pages need no destination request.
After each commit, the engine immediately reads another page unless
`caughtUp` is true. A partial or empty page can exhaust one window without
reaching the watermark. It does not cause a poll delay while a backlog remains.
```mermaid theme={"theme":"kanagawa-wave"}
stateDiagram-v2
state "Queue drain and fix watermark" as Queued
state "Read configuration and committed cursor with lease fence" as CheckLease
state "Read page through batchReader" as ReadPage
state "Send events to destination" as Deliver
state "Commit proposed cursor with lease fence" as Commit
state "Wait PollInterval" as Wait
state "Record retry or pause without advancing cursor" as Failure
state "Stop this cycle" as Stop
[*] --> Queued
Queued --> CheckLease: worker starts with a 1-minute window
CheckLease --> ReadPage: valid, due, cursor before watermark
CheckLease --> Stop: not eligible, read error, or already caught up
ReadPage --> Deliver: page contains events
ReadPage --> Commit: empty page
ReadPage --> Failure: source error
Deliver --> Commit: acknowledged
Deliver --> Failure: rejected or failed
Commit --> CheckLease: committed and not caught up / no delay
Commit --> Wait: committed and caught up
Commit --> Stop: write error or fence rejects update
Failure --> Stop
Wait --> [*]
Stop --> [*]
note right of Commit
No backlog: next_attempt_at = MySQL now + PollInterval.
Backlog remains: next_attempt_at = MySQL now.
An acknowledged but uncommitted batch can be delivered again.
end note
```
Cancellation stops the cycle. It does not mark a source cancellation as a
retryable failure. If the failure-state write fails or loses its fence, the
worker also stops. These exits never authorize a cursor advance.
### Adaptive read windows
Each cycle starts with a 1-minute window and one fixed watermark. The reader
sets `windowEnd = min(cursor.Time + windowSize, watermark)`. Its query reads
after the composite cursor and strictly before `windowEnd`, with at most
`BatchSize` events.
```mermaid theme={"theme":"kanagawa-wave"}
stateDiagram-v2
state "Start from committed cursor" as Start
state "Read through windowEnd, excluding the boundary" as Query
state "Full batch" as Full
state "Partial batch" as Partial
state "Empty batch" as Empty
state "Return proposed page to engine" as Page
state "Return empty, caught-up page" as CaughtUp
state "Return source error" as Error
[*] --> Start
Start --> Query: cursor before watermark
Start --> CaughtUp: cursor at or beyond watermark
Query --> Full: count equals BatchSize
Query --> Partial: count above zero and below BatchSize
Query --> Empty: count is zero
Query --> Error: query fails
Full --> Page: last-event cursor / not caught up / reset size to 1 minute
Partial --> Page: window-end cursor / keep size
Empty --> Page: window-end cursor / double size up to 1 hour
Page --> [*]
CaughtUp --> [*]
Error --> [*]
note right of Start
Creation initializes the cursor to creation time.
end note
note right of Page
A window-end cursor has an empty event ID.
Partial and empty pages are caught up only at the watermark.
The engine must deliver and commit before reading another page.
end note
```
Empty windows grow through 1, 2, 4, 8, 16, 32, and 60 minutes. Further empty
windows stay at 60 minutes. A full batch resets the next query to 1 minute
from the last event's cursor, including its event ID. A partial batch keeps
the window size. Every query is capped at the fixed watermark, even when the
chosen window is larger than the remaining interval.
Window size lives only in the reader for that cycle. It is not stored in
MySQL. A restart, retry, or later poll starts at 1 minute again, using the
committed cursor. A larger window can still be expensive when an empty period
is followed by a burst. Adaptation limits the time range, not query duration
or rows scanned.
## Cursor and watermark
Each drain subscribes to one stream. The supported stream is `audit_logs`,
backed by the ClickHouse table `audit_logs_raw_v1`. The cursor is the pair
`(inserted_at, event_id)`. MySQL stores it as
`committed_offset_inserted_at` and `committed_offset_event_id`.
Creation sets the cursor to the drain's creation time with an empty event ID.
Historical backfill before creation is not supported. Pauses, failures, and
outages resume from the committed cursor and still catch up normally.
A zero cursor follows the same adaptive window path from the Unix epoch.
It eventually catches up without a separate history lookup, but can require
many empty-window queries.
MySQL and ClickHouse must order `event_id` values the same way. ClickHouse
compares `String` values byte by byte, so the MySQL
`committed_offset_event_id` column uses the binary `utf8mb4_0900_bin`
collation. If the two databases used different ordering, they could disagree
about which event follows the committed cursor and skip or repeat events.
The event ID is necessary because multiple events can have the same
`inserted_at` value. For example, assume a batch size of two and these events:
| `inserted_at` | `event_id` |
| ------------- | ---------- |
| 1,000 | `evt_a` |
| 1,000 | `evt_b` |
| 1,000 | `evt_c` |
| 1,001 | `evt_d` |
The first query returns `evt_a` and `evt_b`, then commits the cursor
`(1000, evt_b)`. The next query starts after that pair, so it returns `evt_c`
before `evt_d`. A cursor that stored only `1000` would start the next query
after that millisecond and skip `evt_c`.
A full batch advances the cursor to its last event. A short or empty batch
advances the cursor to that window's exclusive end with an empty event ID.
The empty event ID keeps events at that exact boundary eligible for the next
read, which can happen in the same cycle.
The watermark is the poll's enqueue timestamp minus `WatermarkLag`. It stays
fixed while the worker catches up, rather than moving forward with every
batch. This settling period gives late ClickHouse inserts time to arrive
before the cursor passes their timestamp.
## Leasing and fencing
Leases divide drains between service processes. Each process creates one
startup-unique `lease_id`. The lease service acquires expired leases for that
ID and refreshes them before they expire. Database time controls acquisition,
refresh, and expiry.
Each acquisition also creates a new `fencing_token`. The token identifies one
specific ownership period, even if the same process loses and later reacquires
the drain. The poller includes this token with each work item. Every delivery
state read and write requires the same token and a lease that is still valid.
As a result, stale workers cannot change the cursor, retry time, or failure
state.
Fencing protects durable state. It cannot cancel a customer request that
started before the lease expired.
## Disablement, retries, and pauses
The `logdrains.status` field is `running`, `paused_by_user`, or
`paused_by_failure`. The engine processes a drain only when it is running.
A destination rejection, such as HTTP 400 or HTTP 500, leaves the cursor
unchanged. An unexpected delivery error, such as a timeout or DNS failure,
also leaves the cursor unchanged. The engine retries both cases.
The engine calculates an exponential delay from `consecutive_failures`. The
first retry waits 1 minute. Each later failure doubles the delay through 128
minutes. Later retries wait 4 hours. With the default failure threshold of 50,
the 49 retry waits span 7 days and 15 minutes.
A destination can request a longer delay. The HTTP sink reads the standard
`Retry-After` response header. The Axiom sink reads `Retry-After` first. If that
header is not valid, it reads `X-RateLimit-Reset` as a UTC Unix timestamp in
seconds. The engine uses the longer of the local delay and the destination
delay. It limits a destination delay to 24 hours.
The failure update query increments `consecutive_failures` and writes an
absolute retry time to `next_attempt_at`. It calculates that timestamp from
MySQL time, not process time. The due-drain queries return the drain only after
`next_attempt_at` has passed. A successful cursor update resets the failure
count. A page remains due immediately while a backlog remains. Only a page
that reaches the watermark schedules the next attempt after `PollInterval`.
The engine pauses the drain only after it reaches the configured failure
threshold. Re-enabling a drain or changing its destination makes it running. The
change resets its failure count and retry time. It preserves the cursor.
Delivery resumes from that cursor, subject to the 90-day ClickHouse retention
period for audit logs.
## Delivery guarantees
The cursor, lease, and fencing token define the delivery semantics:
* Deliver-then-commit gives at-least-once delivery. If the process crashes
after the destination acknowledges a batch but before the cursor update,
the next cycle sends that batch again.
* A failure never advances the cursor. The drain retries the same events until
the destination accepts them or the drain pauses.
* Fencing prevents stale workers from changing durable state after lease
expiry, configuration changes, deletion, or reacquisition. It cannot cancel
an external request after a worker completes its fenced read.
Customers must handle duplicate events. They can use the audit log event `id`
as a deduplication key.
## Health telemetry
Each destination attempt writes one row to the ClickHouse table
`logdrain_deliveries_raw_v1`. The row contains the outcome (`success` or
`error`), event count, `webhook_duration_ms`, `request_body_bytes`, and a
truncated error message. `request_body_bytes` contains the uncompressed encoded
body size. It does not include request headers. A rejected HTTP response also
includes the response status and up to 4 KiB of the response body. The duration
measures only the destination request. The dashboard reads this table for the
drain health charts and delivery errors.
### Prometheus metrics
The Prometheus endpoint reports scheduling, delivery, and failure health for
the service.
| Metric name | Type | Labels | Meaning |
| ------------------------------------------ | --------- | --------------------------- | ----------------------------------------------------- |
| `unkey_logdrain_drains` | Gauge | `status`, `stream` | Configured drains by status and stream. |
| `unkey_logdrain_work_queue_depth` | Gauge | None | Items waiting in the work queue. |
| `unkey_logdrain_work_queue_capacity` | Gauge | None | Work queue capacity. |
| `unkey_logdrain_inflight_drains` | Gauge | None | Drains queued or processing. |
| `unkey_logdrain_polls_total` | Counter | `result` | Poll outcomes. |
| `unkey_logdrain_deliveries_total` | Counter | `kind`, `stream`, `outcome` | Delivery attempt outcomes. |
| `unkey_logdrain_events_delivered_total` | Counter | `kind`, `stream` | Events in committed deliveries. |
| `unkey_logdrain_delivery_duration_seconds` | Histogram | `kind`, `stream`, `outcome` | Delivery attempt latency. |
| `unkey_logdrain_drain_failures_total` | Counter | `stream` | Failures recorded by the engine. |
| `unkey_logdrain_drains_paused_total` | Counter | `stream` | Drains paused after they reach the failure threshold. |
# Authentication
Source: https://engineering.unkey.com/architecture/services/vault/auth
RPC authentication and bearer token handling
## Bearer authentication
Vault requires an `Authorization: Bearer ` header on Encrypt, Decrypt, and ReEncrypt RPCs. The token must match `bearer_token` from the service config. Missing or invalid tokens return Unauthenticated. Liveness does not require authentication.
The token is compared using constant-time equality to avoid timing leaks.
## Token rotation
Vault does not manage token rotation. You must update the token in AWS Secrets Manager and roll the deployment.
Runtime callers that embed the bearer token:
* API service ([`svc/api/run.go`](https://github.com/unkeyed/unkey/blob/main/svc/api/run.go))
* Frontline service and certificate manager ([`svc/frontline/run.go`](https://github.com/unkeyed/unkey/blob/main/svc/frontline/run.go), [`svc/frontline/services/certmanager/service.go`](https://github.com/unkeyed/unkey/blob/main/svc/frontline/services/certmanager/service.go))
* Krane service and secrets service ([`svc/krane/run.go`](https://github.com/unkeyed/unkey/blob/main/svc/krane/run.go), [`svc/krane/secrets/service.go`](https://github.com/unkeyed/unkey/blob/main/svc/krane/secrets/service.go))
* Control plane worker and workflows ([`svc/ctrl/worker/run.go`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/worker/run.go), [`svc/ctrl/worker/deploy/service.go`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/worker/deploy/service.go), [`svc/ctrl/worker/certificate/service.go`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/worker/certificate/service.go), [`svc/ctrl/worker/clickhouseuser/service.go`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/worker/clickhouseuser/service.go))
* Control ACME user service ([`svc/ctrl/services/acme/user.go`](https://github.com/unkeyed/unkey/blob/main/svc/ctrl/services/acme/user.go))
* Analytics connection manager ([`internal/services/analytics/service.go`](https://github.com/unkeyed/unkey/blob/main/internal/services/analytics/service.go))
# Configuration
Source: https://engineering.unkey.com/architecture/services/vault/configuration
Configuration model and required settings for the vault service
Unkey services read configuration from a TOML file passed at startup. Environment variables can be referenced with `${VAR}` and are expanded before parsing. Defaults and validation run after parsing.
## Configuration model
The config schema maps to [`svc/vault/config.go`](https://github.com/unkeyed/unkey/blob/main/svc/vault/config.go).
Vault loads configuration via `config.Load`, which expands `${VAR}` environment variables before parsing.
Minimal config example:
```toml theme={"theme":"kanagawa-wave"}
http_port = 8060
region = "${UNKEY_REGION}"
instance_id = "${POD_NAME}"
bearer_token = "${UNKEY_VAULT_TOKEN}"
[encryption]
master_key = "${UNKEY_ENCRYPTION_MASTER_KEY}"
[storage.s3]
url = "${UNKEY_S3_URL}"
bucket = "${UNKEY_S3_BUCKET}"
access_key_id = "${UNKEY_S3_ACCESS_KEY_ID}"
access_key_secret = "${UNKEY_S3_ACCESS_KEY_SECRET}"
```
Used for tracing attributes. Set to pod name in Kubernetes. Example: `vault-7d9b8c4f5d-2kq7m`.
Port for HTTP and RPC traffic. Example: `8060`.
Included in logs and traces. Example: `us-east-1`.
Used for RPC auth. Must be non-empty. Example: `"s3cr3t-token"`.
## Encryption
Encryption key configuration.
Base64-encoded KeyEncryptionKey protobuf. Used for new writes. Example:
`"CiV2YXVsdC1rZXktMSIsIk9TQjRvYjBqWnU9"`.
Optional base64 key used to decrypt existing data during rotation. Example:
`"CiV2YXVsdC1rZXktMCIsIk9TQjRvYjBqWnU9"`.
Vault expects `master_key` values to be a base64 encoding of the serialized `KeyEncryptionKey` protobuf. This format is produced by the vault key generation code.
Example:
```toml theme={"theme":"kanagawa-wave"}
[encryption]
master_key = "${UNKEY_ENCRYPTION_MASTER_KEY}"
previous_master_key = "${UNKEY_ENCRYPTION_PREVIOUS_MASTER_KEY}"
```
## Storage
Vault persists encrypted DEKs through one storage backend. Configure exactly one of `[storage.s3]` or `[storage.disk]`. Setting both, or neither, fails validation at startup.
### S3 storage
S3-compatible object storage is the production backend.
S3-compatible storage configuration.
S3-compatible endpoint URL. Example: `"https://s3.us-east-1.amazonaws.com"`.
Bucket name for encrypted objects. Example: `"unkey-vault"`.
Access key ID for storage. Example: `"AKIA..."`.
Access key secret for storage. Example: `"wJalrXUtnFEMI/K7MDENG/bPxRfiCY"`.
Example:
```toml theme={"theme":"kanagawa-wave"}
[storage.s3]
url = "${UNKEY_S3_URL}"
bucket = "${UNKEY_S3_BUCKET}"
access_key_id = "${UNKEY_S3_ACCESS_KEY_ID}"
access_key_secret = "${UNKEY_S3_ACCESS_KEY_SECRET}"
```
Vault initializes the S3 client on startup and creates the bucket if it does not exist.
### Disk storage
A local filesystem backend exists for local development so you do not need to run a minio container. Object keys are written as forward-slash relative paths under `path`, mirroring the S3 layout. Do not use this backend in production: it does not replicate, it has no durability beyond the host filesystem, and multiple vault replicas pointed at the same path will race.
Local filesystem storage configuration.
Directory where encrypted secrets are persisted. Created at startup if missing. Example: `"./vault-data"`.
Example:
```toml theme={"theme":"kanagawa-wave"}
[storage.disk]
path = "./vault-data"
```
## Observability
Tracing and logging configuration. Each nested section is optional.
Trace sampling rate from 0.0 to 1.0. Example: `0.1`.
Log sampling rate for fast events. Example: `0.01`.
Slow log threshold. Example: `"2s"`.
Vault accepts `observability.metrics` in the config file, but the vault runtime does not start a Prometheus endpoint.
Example:
```toml theme={"theme":"kanagawa-wave"}
[observability.tracing]
sample_rate = 0.1
[observability.logging]
sample_rate = 0.01
slow_threshold = "2s"
```
## Environment variables
The Helm chart provides these variables for the default config template:
Bearer token for vault RPC auth. Example: `"s3cr3t-token"`.
Base64-encoded master key. Example: `"CiV2YXVsdC1rZXktMSIsIk9TQjRvYjBqWnU9"`.
Optional previous master key for rotation. Example: `"CiV2YXVsdC1rZXktMCIsIk9TQjRvYjBqWnU9"`.
S3-compatible endpoint URL. Example: `"https://s3.us-east-1.amazonaws.com"`.
Bucket name for encrypted objects. Example: `"unkey-vault"`.
S3 access key ID. Example: `"AKIA..."`.
S3 access key secret. Example: `"wJalrXUtnFEMI/K7MDENG/bPxRfiCY"`.
Region label for observability. Example: `"us-east-1"`.
OTEL exporter endpoint. Example: `"http://otel-collector.monitoring.svc.cluster.local:4318"`.
OTEL exporter protocol. Example: `"http/protobuf"`.
## Authentication
Vault requires callers to pass an `Authorization: Bearer ` header. The token must match `bearer_token` from the config.
## Key management and rotation
Master keys and bearer tokens are managed manually. When you rotate keys, generate a new master key and update AWS Secrets Manager. Keep the previous master key in `UNKEY_ENCRYPTION_PREVIOUS_MASTER_KEY` until all data is re-encrypted.
### Generate a master key
From the repo root, use the Unkey CLI to generate a base64-encoded master key:
```bash theme={"theme":"kanagawa-wave"}
go run . dev generate-master-key
```
The command prints the encoded key to stdout. Store the value in AWS Secrets Manager as `UNKEY_ENCRYPTION_MASTER_KEY`.
### Generate a bearer token
Vault accepts any non-empty bearer token. Use a strong random value and store it as `UNKEY_VAULT_TOKEN`:
```bash theme={"theme":"kanagawa-wave"}
openssl rand -base64 32
```
High-level flow:
1. Generate a new master key and set it as `UNKEY_ENCRYPTION_MASTER_KEY`.
2. Move the prior master key to `UNKEY_ENCRYPTION_PREVIOUS_MASTER_KEY`.
3. Update AWS Secrets Manager for `unkey/vault`.
4. Re-sync the Helm release to roll the vault pods.
5. Remove `UNKEY_ENCRYPTION_PREVIOUS_MASTER_KEY` after re-encryption is complete.
Vault does not expose a DEK re-encryption RPC. Re-encryption requires running the internal `RollDeks` flow, which walks stored DEKs and rewrites them with the current master key.
### Secret sources
The default Helm chart uses External Secrets to source the vault secrets from `unkey/vault` in AWS Secrets Manager:
Used for RPC authentication. Example: `"s3cr3t-token"`.
Base64-encoded master key. Example: `"CiV2YXVsdC1rZXktMSIsIk9TQjRvYjBqWnU9"`.
Optional key for rotation. Example: `"CiV2YXVsdC1rZXktMCIsIk9TQjRvYjBqWnU9"`.
S3 endpoint. Example: `"https://s3.us-east-1.amazonaws.com"`.
Bucket name. Example: `"unkey-vault"`.
Access key ID. Example: `"AKIA..."`.
Access key secret. Example: `"wJalrXUtnFEMI/K7MDENG/bPxRfiCY"`.
## Example configuration
```toml theme={"theme":"kanagawa-wave"}
http_port = 8060
region = "${UNKEY_REGION}"
instance_id = "${POD_NAME}"
bearer_token = "${UNKEY_VAULT_TOKEN}"
[encryption]
master_key = "${UNKEY_ENCRYPTION_MASTER_KEY}"
previous_master_key = "${UNKEY_ENCRYPTION_PREVIOUS_MASTER_KEY}"
[storage.s3]
url = "${UNKEY_S3_URL}"
bucket = "${UNKEY_S3_BUCKET}"
access_key_id = "${UNKEY_S3_ACCESS_KEY_ID}"
access_key_secret = "${UNKEY_S3_ACCESS_KEY_SECRET}"
[observability.tracing]
sample_rate = 0.1
[observability.logging]
sample_rate = 0.01
slow_threshold = "2s"
```
## Related docs
* [Overview](/architecture/services/vault/overview)
# Overview
Source: https://engineering.unkey.com/architecture/services/vault/overview
Encryption key service backed by object storage
Vault is Unkey's centralized encryption service. It owns data encryption keys (DEKs) and lets runtime services encrypt or decrypt sensitive payloads without embedding key material in application code or databases.
The service stores encrypted DEKs in S3-compatible object storage and protects them with a master key encryption key (KEK). Vault is stateless aside from an in-memory cache, so durable key material lives outside the service and is loaded on demand.
## Place in the stack
Vault sits alongside the core control-plane services and is treated as shared infrastructure for encryption. Services call Vault when they need to encrypt secrets or decrypt stored values. This keeps key material out of MySQL, ClickHouse, and service binaries.
Known runtime callers include the API service, frontline, krane, control worker workflows, and internal analytics. Vault is not exposed to end users and is accessed only by internal services over Connect RPC.
## Responsibilities
Vault is responsible for:
* issuing and storing DEKs per keyring
* encrypting and decrypting payloads with DEKs
* validating encrypted payload structure before decryption
* re-encrypting payloads on demand with the latest DEK
* re-encrypting stored DEKs during master key rotation
## RPC surface
The service uses Connect RPC and exposes these methods:
* `Liveness` returns `ok` and does not require authentication
* `Encrypt` creates or reuses the latest DEK for a keyring and returns a base64-encoded `Encrypted` payload
* `Decrypt` validates and decrypts a base64-encoded `Encrypted` payload
* `ReEncrypt` decrypts a payload, clears the DEK cache, and re-encrypts with the latest DEK for the keyring
`ReEncrypt` ignores the optional `key_id` field in the request and always uses the latest DEK.
`Decrypt` validates the encrypted payload before attempting decryption. It enforces a 12-byte GCM nonce, a ciphertext length of at least 16 bytes, and a non-empty encryption key ID.
## Key model
Vault works with two key types defined in [`svc/vault/proto/vault/v1/object.proto`](https://github.com/unkeyed/unkey/blob/main/svc/vault/proto/vault/v1/object.proto):
* `KeyEncryptionKey` (KEK), the master key used to encrypt DEKs
* `DataEncryptionKey` (DEK), the per-keyring data key used for payload encryption
Encrypted payloads use AES-256-GCM and are serialized into the `Encrypted` message, which includes the nonce, ciphertext, and the KEK or DEK identifier used for encryption.
## Object layout
Vault stores encrypted DEKs in an S3-compatible object store. Objects use a keyring prefix and the DEK ID.
Object keys:
```
keyring//
keyring//LATEST
```
`LATEST` stores the most recent DEK for a keyring and is updated on key creation.
## Object format
Stored object values are serialized `EncryptedDataEncryptionKey` protobuf messages from [`svc/vault/proto/vault/v1/object.proto`](https://github.com/unkeyed/unkey/blob/main/svc/vault/proto/vault/v1/object.proto).
Decoded fields:
* `id` and `created_at` from the DEK
* `encrypted.algorithm` set to `AES_256_GCM`
* `encrypted.nonce` and `encrypted.ciphertext` from the KEK encryption
* `encrypted.encryption_key_id` set to the KEK identifier
* `encrypted.time` set to the encryption timestamp in Unix milliseconds
## Consistency and caching
Vault reads from storage on cache miss and caches DEKs in memory. The cache keeps fresh entries for one hour, allows stale entries for 24 hours, and stores up to 10,000 DEKs per instance.
If storage is eventually consistent, cache misses can return stale keys. The `LATEST` pointer is the source of truth for the newest DEK, so stale reads can temporarily select older keys.
`ReEncrypt` clears the cache to ensure the latest DEK is fetched after key rotation.
## Key rotation and re-encryption
Vault accepts a current master key and an optional previous master key. Both are used for decryption, while new DEKs are always encrypted with the current master key.
Vault can re-encrypt all stored DEKs by walking object storage and rewriting each `EncryptedDataEncryptionKey` with the current master key. This is implemented by `RollDeks` and `Keyring.RollKeys`, and it is not exposed as an RPC method.
## High availability
Vault is stateless aside from the in-memory cache and uses S3 as the system of record. This allows horizontal scaling with multiple replicas.
HA considerations:
* All replicas must share the same S3 bucket and master key.
* Cache state is per-pod and can diverge. Cache TTLs bound staleness.
* If a replica restarts, it repopulates cache on demand from S3.
# Overview
Source: https://engineering.unkey.com/company/index
How Unkey works
This section documents how Unkey works, what the team values, and how communication happens.
# Meetings
Source: https://engineering.unkey.com/company/meetings
Fight for your time and the time of others
Unkey is async-first with teammates in multiple time zones. Minimize meetings and prefer async communication when possible.
Always question if a meeting is necessary. If the information can be shared in a document or Slack message, do that instead.
## No standups
Unkey does not do daily or weekly standups. Use async updates in `#daily-updates` to share progress or blockers.
## 1:1s
One-on-one meetings are personal and optional. They focus on growth, feedback, and issues, not status updates.
## Monthly all hands
Once a month, the team meets to realign, celebrate wins, and discuss improvements. Each person or team can add a slide with updates.
## Request for comments
RFCs can have a follow-up meeting roughly a week after posting. If questions are resolved asynchronously, cancel the meeting.
# Internal workflow
Source: https://engineering.unkey.com/contributing/how-to-contribute
Internal guidelines for working in the Unkey repository
Unkey is not accepting external pull requests at this time. Pull requests from people outside the Unkey team won't be reviewed or merged. Issues remain open for bug reports, feature requests, and documentation feedback.
This guide is for Unkey team members working in this repository.
## Before you start
Discuss significant work before writing code. Use GitHub Issues, Linear, project docs, or the relevant Slack channel so the team can review context asynchronously.
For bug fixes, documentation updates, and small follow-ups, link the issue or context in the pull request.
## House rules
### Issue and PR guidelines
* Before submitting a new issue or PR, check existing issues and PRs .
* Create or link an issue for work that needs tracking.
* Reference the issue in your PR using `fixes #123` or `refs #123`.
* Explain what changed, why it changed, and how you verified it.
### Approval process
Requires prior alignment:
* New features
* Refactoring work
* Changes to core functionality
* UI or UX changes
Can start immediately when scoped:
* Bug fixes
* Security improvements
* Documentation updates
* Typo corrections
## External feedback
Use GitHub Issues for bug reports, feature requests, and documentation feedback. Use Discord for questions and support.
Report security issues by emailing [security@unkey.com](mailto:security@unkey.com) . Do not open public security issues.
# Local development
Source: https://engineering.unkey.com/contributing/local/development
Set up, run, and test Unkey locally
## Prerequisites
We do not support Windows as development environment. It might work, or it might not.
Unkey installs most tools and dependencies automatically. The only required preinstalled dependencies are:
* docker
* git
All other tools are managed via [mise](https://mise.en.dev/), which you'll install in the next step.
## Bootstrap
Clone the repository and install mise and other tools.
```bash theme={"theme":"kanagawa-wave"}
git clone https://github.com/unkeyed/unkey
cd unkey
```
You can set up mise manually or use the install script. It pins mise to a specific version and SHA.
```bash theme={"theme":"kanagawa-wave"}
./dev/install-mise
```
Run the bootstrap task to install the pinned toolchain, create local environment files, and configure the GitHub app.
```bash theme={"theme":"kanagawa-wave"}
mise run bootstrap
```
If GitHub rate limits `mise install`, provide a `GH_TOKEN` when you rerun the task:
```bash theme={"theme":"kanagawa-wave"}
GH_TOKEN=$(gh auth token) mise run bootstrap
```
If you only want to develop on the dashboard, run `mise run dashboard`.
Otherwise continue for a full dev setup.
## Run dev mode
Start the full development setup:
```bash theme={"theme":"kanagawa-wave"}
mise run dev
```
Local deployments use in-cluster BuildKit Jobs by default and don't require Depot credentials. To test the Depot backend, copy `dev/.env.depot.example` to `dev/.env.depot` and set both values to your Depot token before starting the environment.
You get:
* Tilt UI at `http://localhost:10350`
* Various services port-forwarded
* Dashboard at `http://localhost:3000`
## Local HTTPS with Frontline (optional)
Set up local TLS for `*.unkey.local`:
1. Configure local DNS:
```bash theme={"theme":"kanagawa-wave"}
./dev/setup-wildcard-dns.sh
```
2. Start the minikube tunnel in another terminal:
```bash theme={"theme":"kanagawa-wave"}
mise run tunnel
```
3. Open the local domain:
```bash theme={"theme":"kanagawa-wave"}
open https://app.unkey.local
```
Tilt generates trusted TLS certificates using mkcert and Frontline terminates TLS on port 443.
## Stop the development environment
```bash theme={"theme":"kanagawa-wave"}
mise run down
```
## Environment configuration
Dashboard environment variables live in `web/apps/dashboard/.env`. The
bootstrap task creates the file from `web/apps/dashboard/.env.example`.
### Local authentication
Set local auth mode in `web/apps/dashboard/.env`:
```plaintext theme={"theme":"kanagawa-wave"}
AUTH_PROVIDER="local"
```
### Optional services
WorkOS authentication:
Add WorkOS credentials to `web/apps/dashboard/.env`:
```plaintext theme={"theme":"kanagawa-wave"}
AUTH_PROVIDER="workos"
WORKOS_CLIENT_ID=
WORKOS_API_KEY=
WORKOS_COOKIE_PASSWORD=
```
Stripe billing:
The dashboard subscription flow needs all four variables in
`web/apps/dashboard/.env`. If any is missing the dashboard treats Stripe as
unconfigured and billing calls fail.
The product ID lists come pre-filled in `.env.example` (the shared sandbox
catalog), so you only add two values:
* `STRIPE_SECRET_KEY` - a test-mode key (`sk_test_...`) from the shared sandbox.
* `STRIPE_WEBHOOK_SECRET` - the signing secret for forwarded webhook events.
If the stripe CLI is logged in (`stripe login`), `tilt up` handles webhook
forwarding for you: it runs `stripe listen` against both the dashboard
(`localhost:3000/api/webhooks/stripe`) and ctrl-api
(`localhost:7091/webhooks/stripe`), and writes the shared `STRIPE_WEBHOOK_SECRET`
into both `web/apps/dashboard/.env` and `dev/.env.stripe`. No manual step needed.
If the CLI is not logged in, forward events and copy the printed `whsec_...`
yourself:
```bash theme={"theme":"kanagawa-wave"}
stripe listen --forward-to http://localhost:3000/api/webhooks/stripe
```
To set up a fresh Stripe sandbox (products, meters, prices), follow the catalog
guide in the infra repo: Stripe Billing .
Deploy (control plane) billing is separate from the dashboard flow above and is
configured through `dev/` env files that Tilt loads into Kubernetes secrets.
Each is optional: a missing file disables that piece and never breaks startup.
* `dev/.env.stripe` (the `stripe-credentials` secret, read by both ctrl-api and
the worker). Copy `dev/.env.stripe.example` and set:
* `STRIPE_SECRET_KEY` - test-mode key for the hourly usage push and the
month-end invoice finalize.
* `STRIPE_WEBHOOK_SECRET` - the close webhook's signing secret, written
automatically by `tilt up` when the stripe CLI is logged in (as above).
* `STRIPE_DEPLOY_*_LOOKUP_KEY` - the price lookup\_keys `CancelDeploy` uses to
find a subscription's Deploy items. Same handles the dashboard uses; empty
disables cancel.
* The spend-cap budget alert emails need `dev/.env.workos` (`WORKOS_API_KEY`, to
resolve an org's admin recipients) and `dev/.env.resend` (`RESEND_API_KEY`, to
send). Without them the alerts only log; the suspend/resume enforcement is
unaffected.
See [Deploy Billing](/architecture/services/control-plane/worker/workflows/deploy-billing) and [Deploy Spend Cap](/architecture/services/control-plane/worker/workflows/deploy-spend-cap).
### Feature flags
You don't need Vercel Flags setup to run dashboard code that imports `@/lib/flags`. When `FLAGS` is missing, the dashboard uses the noop adapter and resolves each flag to its declared `defaultValue`.
If you're adding flags, testing remote targeting rules, or using Vercel Toolbar overrides, ask Andreas for the dev values of `FLAGS` and `FLAGS_SECRET`. Add them to `web/apps/dashboard/.env`. They're stable, so you set them once and forget. See [Feature flags](/contributing/tooling/feature-flags) for the rest of the workflow.
## Seed local data
```bash theme={"theme":"kanagawa-wave"}
mise run unkey -- dev seed local
```
## Test locally
Run Go tests with Rask:
```bash theme={"theme":"kanagawa-wave"}
mise run test
```
Run a single Go test:
```bash theme={"theme":"kanagawa-wave"}
mise exec -- go test -run TestCacheName ./pkg/cache
```
Run TypeScript tests with pnpm:
```bash theme={"theme":"kanagawa-wave"}
mise exec -- pnpm --dir=web test
```
## Code quality
```bash theme={"theme":"kanagawa-wave"}
mise run fmt
mise run build
```
## Troubleshooting
### Failure: resource\_exhausted: too many requests
If you receive an error message similar to the example below, authenticate your terminal with buf. You can sign up for a free account at [buf.build](https://buf.build/home).
```bash theme={"theme":"kanagawa-wave"}
Failure: resource_exhausted: too many requestssh
Please see https://buf.build/docs/bsr/rate-limits/ for details about BSR rate limiting.
svc/frontline/proto/generate.go:4: running "go": exit status 1
Failure: resource_exhausted: too many requests
Please see https://buf.build/docs/bsr/rate-limits/ for details about BSR rate limiting.
svc/vault/proto/generate.go:3: running "go": exit status 1
mise run generate: command failed
```
# Code quality
Source: https://engineering.unkey.com/contributing/quality/code-quality
Design goals and coding standards for Unkey
This guide captures Unkey's design philosophy and coding standards. It is not about formatting or syntax. Linters handle that. It is about how to think, how to make decisions, and what to value when building software.
Unkey optimizes for safety, performance, and developer experience, in that order. When goals conflict, safety wins. Simplicity is the outcome of iteration, not the first draft.
Unkey has a zero technical debt policy. Do it right the first time. A problem solved in design costs less than one solved in implementation, which costs less than one solved in production. The cost of fixing problems grows exponentially over time.
## Simplicity
Simple and elegant systems are easier to design correctly, more efficient in execution, and more reliable. That simplicity requires hard work and discipline.
Simplicity is not the first attempt. It is the hardest revision. It takes thought, multiple passes, and the willingness to throw work away. The goal is to find the idea that solves multiple problems at once.
## Technical debt
Unkey has a zero technical debt policy. Fix problems when they are discovered. Do not allow issues to slip through with a "fix it later" comment.
## Safety
### Assertions
Crash the request, not the system.
When an invariant is violated, return a 500 error immediately. Do not attempt to recover from programmer errors. Assertions downgrade catastrophic correctness bugs into liveness bugs.
Assertions detect programmer errors, not user errors. A user sending invalid input should get a 400. Code reaching an impossible state should return a 500.
```go theme={"theme":"kanagawa-wave"}
user, err := getUser(ctx, userID)
if err != nil {
return fault.Wrap(err, fault.WithDesc("failed to get user"))
}
// This should never happen if getUser worked correctly.
// If it does, something is deeply wrong. Fail fast.
err = assert.NotEmpty(user.WorkspaceID)
if err != nil {
return err
}
```
Pair assertions. Assert the same property from multiple angles. Validate data before writing to the database and after reading from it. Assert preconditions, postconditions, invariants, and impossible states like default cases in switches or else branches that cannot happen.
### Error handling
All errors must be handled. Never ignore errors. If you think an error cannot happen, assert that assumption explicitly.
```go theme={"theme":"kanagawa-wave"}
// Never do this
result, _ := doSomething()
// Always handle or propagate
result, err := doSomething()
if err != nil {
return fault.Wrap(err, fault.WithDesc("doSomething failed"))
}
```
### Scope
Declare variables at the smallest possible scope. Minimize the number of variables in play at any point. This reduces the probability of using the wrong variable and makes code easier to reason about.
Calculate or check variables close to where they are used. Do not introduce variables before they are needed or leave them around when they are not.
## Failure
Unkey builds distributed systems. Networks partition. Disks fail. Processes crash. Memory runs out. Clocks drift. The question is not whether things will fail, but how gracefully they fail.
### Expect failure
Design for failure from the start. Every external call can fail. Every database query can time out. Every message can be lost. Write code that assumes these things will happen.
### Fail gracefully
Contain the blast radius. One bad request should not bring down the system. One slow dependency should not cascade into total unavailability.
Use circuit breakers to stop calling failing dependencies and give them time to recover. Unkey provides `pkg/circuitbreaker` for this.
### Retry with backoff
Transient failures are common. Retry them, but retry intelligently. Unkey provides `pkg/retry` for this.
Use exponential backoff so you do not hammer a struggling service. Add jitter to randomize retry timing and prevent thundering herds. Set budgets to limit total retry time.
### Idempotency
Exactly-once delivery is a myth. Messages arrive zero times, once, or multiple times. Design operations to be safe to retry. Use idempotency keys for operations that must not be duplicated.
### Observability
Instrument critical paths. Log errors with context. Emit metrics for failure rates. Trace requests across service boundaries.
## Developer experience
### Order matters
Order matters for readability even when it does not affect semantics. In Go files, exported functions come before unexported ones. The primary type or function that the file is named after comes first.
### Comments
A good code comment should explain the reasoning, tradeoffs, and context that future readers need.
Think of comments as breadcrumbs of context left for future readers. Use them sparingly, and only when the breadcrumb carries enough signal to justify its existence.
Keep existing comments accurate when changing the code they describe.
For symbol and package documentation requirements, see [Documentation](/contributing/quality/documentation).
Test docstrings and comments follow the same rule. They explain the guarantee a test protects, especially for security properties, regressions, invariants, and business rules. They do not repeat the mechanics of setup or assertions.
### Function length
Keep functions short. If a function does not fit on a screen, it is probably doing too much. Aim for functions under 50 lines. If a function grows to 200 lines, step back and consider whether it should be broken up.
## Naming
Great names capture what a thing is or does. Append qualifiers to names. Units, bounds, and modifiers come at the end. This groups related variables together and makes scanning easier.
```go theme={"theme":"kanagawa-wave"}
// Bad
maxLatency
minLatency
p99Latency
// Good: qualifiers last, sorted by significance
latencyMsMax
latencyMsMin
latencyMsP99
timeoutSeconds
bufferSizeBytes
```
Do not abbreviate. Use `ctx`, `err`, `req`, `res`, `db`, `id` as the exceptions.
## Dependencies
Unkey integrates with external systems and cannot adopt a zero dependency approach. Every dependency has costs: supply chain risk, maintenance burden, build complexity, and cognitive load.
Prefer the standard library. Prefer single purpose packages. Prefer packages with few transitive dependencies. Avoid adding a dependency when a small local implementation is simpler.
Do not roll your own cryptography. Use official or well-maintained database drivers. Core frameworks like Next.js, Hono, and Drizzle are foundational choices.
## Enforcement
These rules are enforced through tooling and review. Linters, formatters, and CI checks run on every pull request.
Reviewers verify that loops are bounded or justified, timeouts are explicit on external calls, errors are handled, invariants are asserted at boundaries, variable names are clear with units where applicable, and new dependencies are necessary.
Before opening a pull request, ask: did you do the hard thing today, or take a shortcut that creates debt? Would you be confident if this code ran at 10x the current load? Will someone reading this in six months understand why it works this way?
## Acknowledgments
This guide is inspired by TigerStyle and adapted for Unkey's Go and TypeScript codebase.
# Documentation
Source: https://engineering.unkey.com/contributing/quality/documentation
Standards for internal documentation and code comments
## The problem
Documentation serves two masters: the engineer who writes it and the engineer who reads it six months later. Too little documentation leaves readers guessing. Too much buries the signal in noise. The goal is documentation that helps engineers understand and use code correctly. Nothing more, nothing less.
The principles in this guide apply to all languages. The examples are in Go since that is most of the backend, but the philosophy is universal.
## Quick checklist
Before submitting documentation, verify each item.
**Accuracy**
* [ ] Every claim matches actual code behavior
* [ ] Return values match what code returns
* [ ] Error conditions listed are possible and described correctly
* [ ] Default values match actual defaults
* [ ] Constraints documented are enforced, and the docs note when and how
**Completeness**
* [ ] Every symbol has a doc comment
* [ ] Package has a `doc.go` if it has non-trivial behavior
* [ ] Non-obvious behavior is documented (edge cases, nil handling, concurrency)
* [ ] The why is explained for design choices that are not self-evident
* [ ] Every named SQL query has a doc comment block
**Quality**
* [ ] Doc comments start with the symbol name
* [ ] Uses prose, not bullet lists, unless items are parallel
* [ ] Depth matches complexity
* [ ] SQL comments add non-obvious context instead of restating obvious clauses
* [ ] Cross-references use bracket syntax: `[TypeName]`, `[FuncName]`
* [ ] No stale documentation from copy-paste or refactoring
**Verification**
* [ ] You read the implementation, not just the signature
* [ ] For value plus error returns, you checked what value returns on failure
* [ ] For unmarshal operations, you verified whether partial values return
* [ ] Examples compile and run
* [ ] SQL comment examples match real query behavior (ordering, fallback, joins)
## Writing style
Write naturally. Use prose for explanations, not bullet points. Bullet lists are for parallel items or steps. A list of single sentence bullets is often better as a paragraph.
```go theme={"theme":"kanagawa-wave"}
// Bad: bullet spam
// This function:
// - Takes a user ID
// - Validates the input
// - Queries the database
// - Returns the user or an error
// Good: prose
// GetUser retrieves a user by ID from the database. Returns ErrNotFound
// if no user exists with that ID.
```
## Document the why, not the what
The code shows what it does. Documentation should explain why it exists, why it works this way, and what could go wrong.
```go theme={"theme":"kanagawa-wave"}
// IncrementCounter adds one to the counter.
func IncrementCounter() { counter++ }
```
```go theme={"theme":"kanagawa-wave"}
// IncrementCounter updates the request count for rate limiting.
// Not safe for concurrent use; caller must hold the mutex.
func IncrementCounter() { counter++ }
```
### Documenting design choices
When you choose between reasonable alternatives, explain the reasoning in a sentence.
```go theme={"theme":"kanagawa-wave"}
// Package retry provides configurable retry logic for transient failures.
//
// The package uses functional options rather than a config struct because
// retry behavior is usually customized one parameter at a time, and options
// compose better when wrapping retry logic around existing functions.
package retry
```
```go theme={"theme":"kanagawa-wave"}
// Validate checks the request and returns all validation errors at once.
// We return a slice rather than failing on the first error because API
// clients can fix multiple issues in a single round trip.
func Validate(req *Request) []ValidationError
```
## Public API documentation
Every exported function, type, constant, and variable must be documented. This is the contract with users of the code.
Unexported functions and methods must be documented too. They have no external contract, but the next reader still needs to know what they do and why they exist without reverse-engineering the body. A single sentence is usually enough.
The depth of documentation should match complexity. A simple getter needs one line. A distributed algorithm needs paragraphs.
### Simple functions
```go theme={"theme":"kanagawa-wave"}
// GetUserID extracts the user ID from the request context.
// Returns an empty string if no user ID is present.
func GetUserID(ctx context.Context) string
// Close releases all resources held by the client, including network connections
// and background goroutines. After calling Close, the client must not be used.
func (c *Client) Close() error
// SetTimeout updates the request timeout duration for all future requests.
func (c *Client) SetTimeout(d time.Duration)
```
### Complex functions
```go theme={"theme":"kanagawa-wave"}
// Allow determines whether the specified identifier can perform the requested
// number of operations within the configured rate limit window.
//
// This method implements distributed rate limiting with strong consistency
// guarantees across all nodes in the cluster. It uses a lease-based algorithm
// to coordinate between nodes and ensure accurate limiting under high concurrency.
//
// The identifier should be a stable business identifier (user ID, API key, IP).
// The cost is typically 1 for single operations, but can be higher for batch
// requests. Cost must be positive or an error is returned.
//
// Returns (true, nil) if allowed, (false, nil) if rate limited, or (false, error)
// if a system error occurs. Possible errors include ErrInvalidCost for invalid
// cost values, ErrClusterUnavailable when less than 50% of cluster nodes are
// reachable, context.DeadlineExceeded on timeout (default 5s), and network
// errors on storage failures.
//
// Safe for concurrent use. If context is cancelled, no rate limit counters
// are modified.
func (r *RateLimiter) Allow(ctx context.Context, identifier string, cost int) (bool, error)
```
### When to include specific details
**Parameters**: Document when the purpose is not obvious from the name and type, or when there are constraints like must be positive.
**Return values**: Explain when return patterns are subtle or when multiple success states exist. For functions that return a value plus an error, document what value returns on failure.
**Error conditions**: List specific errors only when callers need to handle them differently.
**Concurrency**: Document when a function or type is safe or unsafe for concurrent use.
**Performance**: Mention non-obvious characteristics that affect usage decisions.
**Context**: Document context behavior only if it is non-standard.
## SQL query documentation
Named SQL queries are part of the public contract between application code and the database. Treat query comments like API documentation.
For SQL query docs, explain why this query exists, how it resolves non-obvious behavior, and what guarantees callers can rely on. Do not just restate the `SELECT` clause.
Keep SQL comments concise. In most cases, two to five lines are enough. If a comment is longer than the query, each sentence must carry non-obvious information such as fallback guarantees, deterministic ordering, intentional join behavior, or performance tradeoffs.
Avoid duplicate explanations. If the SQL already makes behavior obvious, for example `AND s.health = 'healthy'`, do not repeat that in prose unless you are documenting a non-obvious guarantee that depends on it.
For selection queries with fallback logic, document deterministic behavior explicitly. If exact matching must win over wildcard matching, explain how SQL enforces it, for example with `ORDER BY` and not candidate list order.
For query docs that are not obvious from a quick read, include a small concrete example with inputs and the expected returned row. This is required for logic that depends on ordering, fallback, or tie-breaking.
Document performance-sensitive choices when they are intentional, for example using `LIMIT 1` to avoid transferring large payload rows that are not selected.
```sql theme={"theme":"kanagawa-wave"}
-- name: FindBestCertificateByCandidates :one
-- FindBestCertificateByCandidates returns one certificate row for the provided
-- hostnames, preferring an exact hostname over wildcard matches.
-- MySQL does not preserve IN-list order, so exact-first behavior is enforced by
-- ORDER BY against exact_hostname, not by candidate position.
--
-- Example: with candidates ['api.example.com', '*.example.com'] and
-- exact_hostname 'api.example.com', this query returns 'api.example.com' when
-- both rows exist. If only '*.example.com' exists, it returns the wildcard row.
--
-- LIMIT 1 avoids returning non-selected certificate and key payload rows.
SELECT
hostname,
workspace_id,
certificate,
encrypted_private_key
FROM certificates
WHERE hostname IN (sqlc.slice('hostnames'))
ORDER BY hostname = sqlc.arg(exact_hostname) DESC
LIMIT 1;
```
## What not to document
Do not document implementation details in doc comments. Those belong inside the function. Do not explain that context is used for cancellation or mention O(1) performance unless it is surprising.
## Package documentation
Every significant package should have a `doc.go` file with the package comment and declaration. It should explain what the package does, why it exists, how it fits into the system, key concepts, usage, and cross-references.
### Structure of doc.go
Use `#` headers to organize sections. Include a usage example.
```go theme={"theme":"kanagawa-wave"}
// Package ratelimit implements distributed rate limiting with lease-based coordination.
//
// The package uses a two-phase commit protocol to ensure consistency across
// multiple nodes in a cluster. Rate limits are enforced through sliding time
// windows with configurable burst allowances.
//
// This implementation was chosen over simpler approaches because it needs
// strong consistency guarantees for billing and security use cases.
//
// # Key Types
//
// The main entry point is [RateLimiter], which provides the [RateLimiter.Allow]
// method for checking rate limits. Configuration is handled through [Config].
//
// # Usage
//
// Basic rate limiting:
//
// cfg := ratelimit.Config{Window: time.Minute, Limit: 100}
// limiter := ratelimit.New(cfg)
// allowed, err := limiter.Allow(ctx, "user:123", 1)
// if err != nil {
// // Handle system error
// }
// if !allowed {
// // Rate limited - reject request
// }
//
// # Error handling
//
// The package distinguishes between rate limiting (expected behavior) and
// system errors (unexpected failures). See [ErrRateLimited] and [ErrClusterUnavailable].
package ratelimit
```
## Internal code
Internal functions have different documentation needs. The audience is teammates maintaining this code. The why matters even more than the what.
```go theme={"theme":"kanagawa-wave"}
// retryWithBackoff handles retries for failed lease acquisitions.
//
// Exponential backoff with jitter spreads retry attempts and reduces system load.
// Max retry count is limited to prevent infinite loops during outages.
func (r *RateLimiter) retryWithBackoff(ctx context.Context, fn func() error) error
```
## Complex algorithm documentation
For complex internal logic, explain the approach and reasoning.
```go theme={"theme":"kanagawa-wave"}
// distributeTokens implements the token bucket algorithm with cluster coordination.
//
// Token bucket is chosen for burst handling, simpler math, and predictable memory use.
// The algorithm runs in two phases: local calculation, then cluster consensus.
func (r *RateLimiter) distributeTokens(ctx context.Context, required int64) (granted int64, err error)
```
## Types and interfaces
Type documentation should explain what the type represents and any constraints or invariants. Interface documentation should focus on the contract and concurrency guarantees.
```go theme={"theme":"kanagawa-wave"}
// Config holds the configuration for a rate limiter instance.
//
// Window and Limit work together to define rate limiting behavior.
// For example, Window=1m and Limit=100 means 100 operations per minute.
type Config struct {
Window time.Duration
Limit int64
ClusterNodes []string
}
```
```go theme={"theme":"kanagawa-wave"}
// Cache provides a generic caching interface with support for distributed invalidation.
//
// Implementations must be safe for concurrent use. The cache may return stale data
// during network partitions to maintain availability, but will converge after recovery.
type Cache[T any] interface {
Get(ctx context.Context, key string) (value T, found bool, err error)
Set(ctx context.Context, key string, value T) error
}
```
## Error documentation
Document sentinel errors with meaning and conditions.
```go theme={"theme":"kanagawa-wave"}
var (
// ErrRateLimited is returned when an operation exceeds the configured rate limit.
ErrRateLimited = errors.New("rate limit exceeded")
// ErrClusterUnavailable indicates insufficient cluster nodes are reachable.
ErrClusterUnavailable = errors.New("insufficient cluster nodes available")
)
```
## Constants and variables
Document purpose and reasoning when it is not obvious.
```go theme={"theme":"kanagawa-wave"}
const (
// DefaultWindow is the standard rate limiting window.
DefaultWindow = time.Minute
)
```
## Examples
Use Go example tests for non-trivial usage patterns. Examples compile and run, so they do not go stale.
## Test documentation
Document test helpers and complex test scenarios so future maintainers understand the purpose.
## Document what not to do
Warn against common mistakes when a misuse would be easy and costly.
## Verify before you document
The most dangerous documentation is confident and wrong. Read the implementation. Document what the code does, not what you think it should do.
Common verification failures include return values on error, partial values from unmarshal, constraint enforcement timing, defaults, and context behavior.
## Common mistakes
Restating the signature adds no value. Documenting irrelevant details creates noise. Missing critical information is dangerous. Stale documentation is worse than no documentation.
## Go conventions
Start doc comments with the name of the thing being documented. Use present tense. Write complete sentences. Use bracket syntax for references, for example `[TypeName]` and `[FuncName]`.
## Deprecation
When deprecating an API, provide a migration path.
```go theme={"theme":"kanagawa-wave"}
// Deprecated: Use [NewRateLimiterV2] instead. This function will be removed in v2.0.
//
// Migration example:
//
// // Old:
// limiter := NewRateLimiter(100, time.Minute)
//
// // New:
// limiter := NewRateLimiterV2(Config{Limit: 100, Window: time.Minute})
func NewRateLimiter(limit int, window time.Duration) *RateLimiter
```
## Keeping documentation alive
Update documentation whenever you change behavior, parameters, or error conditions. When reviewing code, check that documentation still matches implementation.
# Screenshots and recordings
Source: https://engineering.unkey.com/contributing/quality/screenshots-and-recordings
How to capture screenshots and screen recordings for pull requests with visual changes
* All unkey employees are encouraged to pick up a `Cloud Basic` [http://cleanshot.com/](http://cleanshot.com/) license and learn how to use it.
* For employees who would like to frequently record explainer videos, feel free to ask for a Supercut license [https://supercut.ai/](https://supercut.ai/)
## Screenshot guidelines
Screenshots are incredibly useful tools, but a bad screenshot can sometimes do more harm than good, and force people to think harder than they need to about what they're looking at.
Here's some general rules on how to create a nice screenshot.
* When using Cleanshot, please screenshot the ENTIRE browser window that the change / fix is present on. You can set up and configure a shortcut this from within the Cleanshot Settings page.
* If your change only uses a tiny portion of the current view, use cleanshots annotations feature to add arrows and point out the relevant portion of a change.
* Do not screenshot a portion of the app if at all possible, as it can make it harder for people to grep what context the item sits within.
### Example
Both screenshots show the same change: the email input on the sign-in page.
| Bad | Good |
| --------------------------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------- |
| /` so it becomes part of the seed corpus.
## What to do when fuzzing finds a bug
Write a deterministic unit test for the failing input, then fix the bug. Keep the fuzz corpus in `testdata` to prevent regressions.
## Corpus files
Fuzz tests live in regular Go test files. Commit saved failure inputs under
`testdata/fuzz//` so normal test runs replay them.
# HTTP handler tests
Source: https://engineering.unkey.com/contributing/quality/testing/http-handler-tests
Testing API endpoints with the test harness
## Testing the full stack
HTTP handler tests exercise API endpoints from request to response. A single request might authenticate a user, check permissions, validate input, query a database, update a cache, and write an audit log. Testing handlers end to end catches bugs that unit tests miss.
Every handler must have tests for success, validation errors, authentication errors, and authorization errors.
Each handler test must make the API guarantee visible. The reader must know which contract the endpoint promises, which client-visible response is expected, and which side effects are required or forbidden.
## Anatomy of a handler test
Create a test harness, configure the handler with harness dependencies, register the handler, create credentials, make a request, and verify the response.
```go theme={"theme":"kanagawa-wave"}
func TestCreateApi_Success(t *testing.T) {
h := testutil.NewHarness(t)
route := &handler.Handler{
Logger: h.Logger,
DB: h.DB,
Keys: h.Keys,
Auditlogs: h.Auditlogs,
}
h.Register(route)
rootKey := h.CreateRootKey(h.Resources().UserWorkspace.ID, "api.*.create_api")
headers := http.Header{
"Content-Type": {"application/json"},
"Authorization": {fmt.Sprintf("Bearer %s", rootKey)},
}
req := handler.Request{Name: "my-new-api"}
res := testutil.CallRoute[handler.Request, handler.Response](h, route, headers, req)
require.Equal(t, http.StatusOK, res.Status)
require.NotEmpty(t, res.Body.ApiID)
}
```
## Organizing test files
Organize tests by behavior so the intent is clear. Example directory: [`svc/api/routes/v2_apis_create_api/`](https://github.com/unkeyed/unkey/blob/main/svc/api/routes/v2_apis_create_api/).
Typical files:
* [`svc/api/routes/v2_apis_create_api/handler.go`](https://github.com/unkeyed/unkey/blob/main/svc/api/routes/v2_apis_create_api/handler.go)
* [`svc/api/routes/v2_apis_create_api/success_test.go`](https://github.com/unkeyed/unkey/blob/main/svc/api/routes/v2_apis_create_api/success_test.go)
* [`svc/api/routes/v2_apis_create_api/validation_test.go`](https://github.com/unkeyed/unkey/blob/main/svc/api/routes/v2_apis_create_api/validation_test.go)
* [`svc/api/routes/v2_apis_create_api/auth_test.go`](https://github.com/unkeyed/unkey/blob/main/svc/api/routes/v2_apis_create_api/auth_test.go)
## Testing success cases
Test minimal requests, full requests, and any important variations. Verify side effects such as audit log writes when relevant.
## Testing validation errors
Test boundary conditions, missing required fields, and invalid formats. These tests document the API contract.
Use test names and docstrings to identify the contract. For example, document whether invalid input must return `400`, whether the error code is stable for SDKs, and whether the handler must avoid writing partial state.
## Testing authentication
Reject missing headers, malformed tokens, and revoked credentials.
Authentication tests protect security guarantees. Document the guarantee when a case covers token confusion, revoked credentials, replay behavior, or information disclosure.
## Testing authorization
Verify that permissions are enforced and cross-workspace access is rejected.
Authorization tests must state the resource boundary they protect. If a response intentionally hides existence with `404`, document that behavior in the test so future changes do not weaken it by accident.
## Helper functions
Extract repeated setup into helpers. If a helper asserts, it must call `t.Helper()`.
## Debugging failed requests
Use the raw response body to inspect failures:
```go theme={"theme":"kanagawa-wave"}
if res.Status != http.StatusOK {
t.Logf("Response body: %s", res.RawBody)
}
```
# Testing
Source: https://engineering.unkey.com/contributing/quality/testing/index
Testing standards and patterns for Unkey
## Why we test
Tests exist to give confidence. Confidence to ship changes quickly, confidence that refactoring will not break production, confidence that the system behaves as expected. A test suite with 90 percent coverage that misses critical edge cases is less valuable than one with 60 percent coverage that catches real bugs.
We prioritize quality over quantity. A single well-designed test that validates complex business logic is worth more than a dozen tests that exercise trivial code paths. When writing tests, ask what could go wrong in production that this test would catch.
Treat test code as production code. Tests are part of the system contract, not a safety net added after implementation. They must be readable, typed, deterministic, reviewed with the same care as application code, and maintained when behavior changes.
Every test must make its guarantee clear to the reader. The guarantee is the production behavior that would break if the test failed. A future reader must be able to understand what risk the test covers without reverse engineering setup, mocks, or fixtures.
Document non-obvious guarantees with docstrings or comments. Use them when a test protects a security property, regression, invariant, concurrency condition, failure mode, or business rule that the test name cannot fully explain.
```go theme={"theme":"kanagawa-wave"}
// TestTokenParserRejectsAlgorithmConfusion guarantees that tokens signed with
// one algorithm cannot be accepted under another algorithm. This protects the
// verifier from accepting attacker-controlled headers as trusted policy.
func TestTokenParserRejectsAlgorithmConfusion(t *testing.T) {
// ...
}
```
Do not document obvious mechanics. A comment that says "creates a workspace" above `createWorkspace(t)` adds noise. A comment that explains why cross-workspace access must return 404 instead of 403 documents a guarantee.
## What to test
Invest testing effort where bugs would hurt most.
**High value targets:** Business logic with complex conditionals, error handling paths, concurrent code with race potential, security sensitive operations, data transformations that could silently corrupt.
**Lower value targets:** Simple getters and setters, straightforward pass-through functions, code that delegates to well-tested libraries.
**Skip entirely:** Tests that verify the programming language works.
Ask what bug this test would catch that the compiler, a code review, or a more meaningful test would not.
### Testing observability
Do not verify every log line or metric increment. Test metrics that drive alerts or SLOs. Test that error conditions produce the logs operators need for debugging.
```go theme={"theme":"kanagawa-wave"}
func TestRateLimiter_EmitsRejectionMetric(t *testing.T) {
collector := &testMetricCollector{}
limiter := NewRateLimiter(Config{Limit: 1}, collector)
limiter.Allow()
limiter.Allow()
require.Equal(t, 1, collector.Count("rate_limit_rejected_total"))
}
```
## Go testing
All Go tests use `github.com/stretchr/testify/require` for assertions.
Run normal Go test suites with Rask through mise. Rask keeps package-level test
runs fast while preserving the standard Go test behavior.
## Test organization
Tests live alongside the code they test. A file `cache.go` has its tests in `cache_test.go` in the same directory.
For integration tests that require substantial setup or external dependencies, create an `integration/` subdirectory when it improves clarity.
Organize tests around guarantees, not implementation details. File names, test names, table cases, and helper names must help a reader answer which behavior is protected and why it matters.
Use package structure, test names, and helper names to communicate test scope.
Keep unit tests close to the package under test. Put expensive cross-service
tests in an `integration/` subdirectory when that makes the boundary clearer.
## Writing test helpers
Every helper function must call `t.Helper()` as its first line.
```go theme={"theme":"kanagawa-wave"}
func createTestWorkspace(t *testing.T) *Workspace {
t.Helper()
ws, err := db.CreateWorkspace(ctx, "test-workspace")
require.NoError(t, err)
return ws
}
```
## Resource cleanup
Tests that acquire resources must clean them up. Use `t.Cleanup()` instead of `defer`.
```go theme={"theme":"kanagawa-wave"}
func TestWithDatabase(t *testing.T) {
db := setupTestDatabase(t)
t.Cleanup(func() {
db.Close()
})
}
```
## Running tests
During development, run tests for the package you are working on:
```bash theme={"theme":"kanagawa-wave"}
mise exec -- rask ./pkg/cache
```
Before pushing, run the full test suite:
```bash theme={"theme":"kanagawa-wave"}
mise run test
```
## What is next
Use these guides for deeper patterns:
* [Unit tests](/contributing/quality/testing/unit-tests)
* [Integration tests](/contributing/quality/testing/integration-tests)
* [Anti-patterns](/contributing/quality/testing/anti-patterns)
# Integration tests
Source: https://engineering.unkey.com/contributing/quality/testing/integration-tests
Testing components with real dependencies
## When to use integration tests
Use integration tests to cover behavior across service boundaries, real databases, caches, and storage. These tests catch failures that mocks miss.
Integration tests must document the cross-boundary guarantee they protect. Make it clear which production contract would be broken if the test failed, such as transaction rollback, idempotency, permission propagation, cache invalidation, or persistence after restart.
```go theme={"theme":"kanagawa-wave"}
// TestCreateKeyRollsBackAuditLogOnDatabaseFailure guarantees that key creation
// and audit logging remain atomic. Operators must not see an audit log for a
// key that was never committed.
func TestCreateKeyRollsBackAuditLogOnDatabaseFailure(t *testing.T) {
// ...
}
```
## Container patterns
Use `pkg/testutil/containers` for shared test containers. Helpers lazily start
containers through `pkg/testutil/docker-compose.test.yaml` and reuse one Docker
Compose project per worktree, so tests only start the services they need.
`mise run test` removes the shared containers for that worktree after the test
suite exits.
Restate is the exception to container reuse. `containers.Restate` starts a
server per test and removes it afterwards, because Restate identifies services
by name and those names come from protobuf packages. Two tests registering
their own workers on one server would overwrite each other's routing and share
virtual object state.
A private Restate costs about a second to start. Since Go runs a package's
tests sequentially, at most one such container per test binary is alive at a
time.
To inspect a failure, set `UNKEY_TEST_KEEP_RESTATE=1`. The container of a
failed test is left running and its admin URL is logged, so the invocation
journal and state stay queryable:
```bash theme={"theme":"kanagawa-wave"}
UNKEY_TEST_KEEP_RESTATE=1 mise exec -- rask ./svc/ctrl/worker/cron
```
Containers left behind by a test process that was killed are removed by the
next run.
For full-suite runs, use `mise run test`. Direct `rask` does not run the
cleanup trap from the mise task.
```go theme={"theme":"kanagawa-wave"}
redisURL := containers.Redis(t)
```
## Test harness
Use `pkg/testutil` when you need a full service graph and seeded data:
```go theme={"theme":"kanagawa-wave"}
h := testutil.NewHarness(t)
workspace := h.CreateWorkspace()
```
Keep harness setup explicit when it affects the guarantee. Do not hide permissions, feature flags, tenants, or seeded records behind generic helpers unless the helper name or docstring explains the resulting system state.
## Suite cost
Integration tests that start containers must use the shared container helpers.
Keep expensive setup explicit so readers can see why the test belongs in the
integration suite.
## Debugging failures
Use verbose output for failing tests:
```bash theme={"theme":"kanagawa-wave"}
mise exec -- rask -v ./pkg/vault/integration
```
# Simulation tests
Source: https://engineering.unkey.com/contributing/quality/testing/simulation-tests
Property-based testing with the simulation framework
## Beyond example-based testing
Example tests verify specific inputs and outputs. Simulation tests verify invariants across random sequences of operations. This is valuable for stateful systems like caches, rate limiters, and state machines.
Unkey uses `pkg/sim` for simulation testing.
Simulation tests must document the invariant they protect. Random events can obscure intent, so the test docstring and validator names must state the production guarantee directly.
## The mental model
A simulation has state, events, and validators.
* State is the system under test plus any bookkeeping.
* Events modify state in random order.
* Validators check invariants after each step.
## A simple example
```go theme={"theme":"kanagawa-wave"}
type state struct {
cache cache.Cache[uint64, uint64]
keys []uint64
clk *clock.TestClock
}
type setEvent struct{}
func (e *setEvent) Name() string { return "set" }
func (e *setEvent) Run(rng *rand.Rand, s *state) error {
key := rng.Uint64()
val := rng.Uint64()
s.keys = append(s.keys, key)
s.cache.Set(context.Background(), key, val)
return nil
}
```
## Running the simulation
```go theme={"theme":"kanagawa-wave"}
// TestCacheSimulation guarantees that cache operations preserve basic cache
// invariants across random event orderings. No event may leave the cache in a
// nil or unreadable state.
func TestCacheSimulation(t *testing.T) {
seed := sim.NewSeed()
simulation := sim.New[state](seed,
sim.WithState(func(rng *rand.Rand) *state {
clk := clock.NewTestClock(time.Now())
c, _ := cache.New(cache.Config[uint64, uint64]{
Clock: clk,
Fresh: time.Second,
Stale: time.Minute,
MaxSize: rng.IntN(1000) + 1,
})
return &state{cache: c, keys: []uint64{}, clk: clk}
}),
)
simulation = sim.WithValidator(func(s *state) error {
if s.cache == nil {
return fmt.Errorf("cache should not be nil")
}
return nil
})(simulation)
err := simulation.Run([]sim.Event[state]{&setEvent{}})
require.NoError(t, err)
}
```
## Reproducibility
When a simulation fails, save the seed and rerun with `sim.SeedFromString` to reproduce the sequence.
## Writing effective events
Events must be self-contained and valid regardless of current state. If preconditions are not met, return early.
Name events and validators after domain behavior. `rejectExpiredKeyValidator` is clearer than `validator1` because it tells the reader which guarantee failed.
## When to use simulations
Use simulations for caches, rate limiters, and state machines with many valid transitions. Skip them for simple stateless logic.
## Test placement
Simulation tests are regular Go tests. Keep them near the package under test
unless the simulation covers a cross-package contract.
# Unit tests
Source: https://engineering.unkey.com/contributing/quality/testing/unit-tests
Table-driven patterns and unit test conventions
## Table-driven tests
Use table-driven tests when cases share setup and assertions. Add `t.Run` so each case is reported by name.
```go theme={"theme":"kanagawa-wave"}
func TestValidateEmail(t *testing.T) {
tests := []struct {
name string
email string
wantErr bool
}{
{name: "valid email", email: "user@example.com", wantErr: false},
{name: "missing @", email: "userexample.com", wantErr: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
err := ValidateEmail(tt.email)
if tt.wantErr {
require.Error(t, err)
return
}
require.NoError(t, err)
})
}
}
```
Use individual tests when setup or assertions diverge meaningfully.
## Naming
Name test functions `Test_` or `Test_`. Name table cases so failures read like a sentence.
Prefer names that state the guarantee under test. `TestParseURN_RejectsCrossWorkspaceResources` is better than `TestParseURN_InvalidInput` because it tells the reader which production boundary the test protects.
Add a docstring when the guarantee is not obvious from the name. Use the docstring to explain the invariant, regression, or business rule, not the steps the test performs.
```go theme={"theme":"kanagawa-wave"}
// TestAuthorizeRejectsCrossWorkspaceAccess guarantees that a valid key from one
// workspace cannot authorize reads in another workspace. Returning not found
// prevents callers from discovering that the target resource exists.
func TestAuthorizeRejectsCrossWorkspaceAccess(t *testing.T) {
// ...
}
```
## Parallel execution
Use `t.Parallel()` only when tests do not share mutable state or external resources.
## Helpers and cleanup
Helpers that assert must call `t.Helper()`. Use `t.Cleanup()` for resource cleanup so subtests complete before cleanup runs.
Helpers are production test APIs. Keep them small, typed, and named after the domain object or state they create. If a helper hides important setup, document the guarantee it establishes for callers.
## Test data
Inline small fixtures. Use `testdata/` for larger files that belong with the
package under test.
Name fixtures by the behavior they exercise. A fixture named `expired_root_key.json` is better than `case_3.json` because it carries the test guarantee into the filesystem.
## Time-dependent logic
Use `pkg/clock` to control time rather than sleeping.
# Builds
Source: https://engineering.unkey.com/contributing/tooling/builds
How Unkey builds service images with Docker
Service images are built with Docker. For how a built image then ships to
production, see [Releases](./releases).
## Why Docker produces the images
Unkey uses Docker for image production for three reasons:
1. **Plain Dockerfiles.** Local dev, image loading, and releases use Docker
files.
2. **Multi-arch releases.** The release workflow uses Docker Buildx to publish
`linux/amd64` and `linux/arm64` variants.
3. **Simple local builds.** Tilt compiles binaries on the host for fast
incremental rebuilds. Dashboard Compose compiles its services once in a
shared Docker build stage.
## What ships in a release image
Every release service image is the same shape:
* **A single static Go binary** at `/unkey`. No libc, no shared libraries, no
init scripts. The image entrypoint is the binary path directly, not a shell
invocation.
* **Distroless base.** No shell, no busybox, no package manager. Attack
surface is minimal and the image contains exactly what the service needs to
run. This is also why container probes have to be HTTP-based; there is no
`sh` to exec.
* **Multi-arch manifest.** Each release publishes an OCI index referencing
amd64 and arm64 variants built in parallel from the same Go source.
`docker pull` resolves to the right architecture automatically, so
deployments don't care whether the cluster is x86 or Graviton.
These choices are tightly coupled. Static binaries are required because the
distroless base has no dynamic loader. The distroless base is what lets us
use a direct binary entrypoint. The multi-arch index lets the same tag work
across our cluster types.
## Where things live
Build configuration is split by concern:
* `build//main.go` is the entrypoint. It wires a TOML config
command via `build/util` and calls into the service's `Run` function.
Runtime behavior (instance IDs, clocks, TLS loading, normalization) stays
in `svc/.../run.go`. The entrypoint is intentionally thin so it does not
become a second service runtime.
* `dev/Dockerfile` compiles the services required by Dashboard Compose in one
shared stage, then packages each binary in a named busybox stage.
* `Dockerfile.release` owns release image packaging. It copies the prebuilt
binary that GoReleaser produces for each platform. The image carries only the
repository source label.
* `.goreleaser.service.yaml` builds release binaries and publishes multi-arch
images with `Dockerfile.release`.
* `dev/Tiltfile` compiles each service on the host with the local Go build
cache, then packages the binary with `dev/Dockerfile.binary`. The image is
built once per service; after that, Tilt's `live_update` syncs the rebuilt
binary into the running container and restarts the process in place, so a
code change never rebuilds an image or rolls a pod.
* `web/apps/dashboard/dev/docker-compose.yaml` selects the service stages from
`dev/Dockerfile`.
## Local images
Tilt and the dashboard `docker-compose` setup both produce local
`unkey/:dev` tags:
| Local environment | Image source |
| ----------------- | --------------------------------------------------------------------------- |
| `dev/k8s` (Tilt) | host `go build` + `dev/Dockerfile.binary` + `live_update` in `dev/Tiltfile` |
| Dashboard Compose | shared Docker builder + named busybox stages |
Local development images use a busybox base. Tilt needs `sh`, `date`, and `tar`
for `live_update`, while Dashboard Compose uses the same base for consistency.
Release images use distroless.
## Adding a new service image
1. Add `build//main.go` with a call to `util.RunServiceCommand`.
2. Add the service's tag pattern to `.depot/workflows/service-release.yaml`
so the release workflow picks up `/vx.y.z` pushes.
3. Add the service to `.goreleaser.service.yaml`.
4. Wire it into local dev: a `go_service_image` call in `dev/Tiltfile`, and
any compose file that needs it.
Keep new entrypoints thin. Anything that looks like service behavior, such
as defaults, injected clocks, or generated instance IDs, belongs in the
service package, not in `build//main.go`.
# Transactional emails
Source: https://engineering.unkey.com/contributing/tooling/emails
How Go services send email and how to edit the templates
Go services send email through `pkg/email` by naming a published Resend
template and its variables. The template content is authored in this repo as
react-email components in `web/internal/resend/emails/` and uploaded to
Resend by `web/internal/resend/scripts/sync-templates.tsx`. Copy or design
changes are TypeScript edits plus one command, no Go deploy.
Without a `RESEND_API_KEY`, `pkg/email` falls back to a noop sender that only
logs, so local and CI never send real mail.
## Editing an email
Edit the component in `web/internal/resend/emails/`, then:
```bash theme={"theme":"kanagawa-wave"}
# live preview
mise exec -- pnpm --dir=web/internal/resend dev
# render test
mise exec -- pnpm --dir=web/internal/resend test
```
## Publishing
```bash theme={"theme":"kanagawa-wave"}
# upload draft versions
RESEND_API_KEY=... mise exec -- pnpm --dir=web/internal/resend sync-templates
# upload and publish
RESEND_API_KEY=... mise exec -- pnpm --dir=web/internal/resend sync-templates --publish
```
Drafts are invisible to sends; Go keeps sending the last published version
until you publish. Publishing takes effect immediately. The key lives in
`dev/.env.resend` locally (see `dev/.env.resend.example`).
## Adding a new email
Write the component with string props for every dynamic value, then register
it in the `templates` array in `scripts/sync-templates.tsx`: alias, name,
subject, from, the element with `{{{VARIABLE}}}` placeholder props, and the
variable declarations with fallbacks. Include a `Preview` component for the
inbox snippet.
Do not rename an alias a deployed service sends with, and keep the variable
keys identical to the map the Go caller passes; Resend rejects sends with
missing variables.
Set `Email.IdempotencyKey` when a caller may retry after a transient failure
(e.g. a Restate handler re-invocation). Resend dedupes identical sends for 24
hours. Use a stable key per logical message, such as
`budget-alert/{workspace_id}:{period}`.
# Feature flags
Source: https://engineering.unkey.com/contributing/tooling/feature-flags
How feature flags work in the dashboard
Flags are declared in `web/apps/dashboard/lib/flags/index.ts`. The `identify` helper reads the current session and passes stable `user.id` and `org.id` entities to each flag, so targeting by user or org doesn't need extra wiring per flag.
The flag registry uses a local adapter wrapper instead of calling `vercelAdapter()` directly. When the Vercel `FLAGS` setup is present, the wrapper returns the Vercel adapter. When `FLAGS` is absent, it returns a noop adapter that resolves each flag to its declared `defaultValue`.
## Adding a flag
From `web/apps/dashboard`:
```bash theme={"theme":"kanagawa-wave"}
vercel flags create my-flag --kind boolean --description "What it gates"
```
Declare it alongside the existing ones:
```ts theme={"theme":"kanagawa-wave"}
export const myFlag = flag({
key: "my-flag",
description: "Gates the new dashboard workflow",
defaultValue: false,
options: [
{ value: false, label: "Off" },
{ value: true, label: "On" },
],
identify,
adapter: adapter(),
});
```
Every flag must declare a `defaultValue`. The noop adapter uses that value in local development and self-hosted environments that don't configure Vercel Flags.
Add the flag to `web/apps/dashboard/lib/flags/resolve.ts` so the `FlagsProvider` in the root layout exposes it to client components:
```ts theme={"theme":"kanagawa-wave"}
export async function resolveAll() {
const [helloWorld, myFlag] = await Promise.all([flags.helloWorld(), flags.myFlag()]);
return { helloWorld, myFlag };
}
```
The `Flags` type is derived from `resolveAll`'s return shape, so any flag missing from this list will fail to type-check at every `useFlag(key)` call site.
Use it from a client component:
```tsx theme={"theme":"kanagawa-wave"}
import { useFlag } from "@/lib/flags/provider";
const enabled = useFlag("myFlag");
```
Or from server code by awaiting the flag directly:
```ts theme={"theme":"kanagawa-wave"}
import { myFlag } from "@/lib/flags";
const enabled = await myFlag();
```
## Toggling
From the Vercel dashboard, or from the CLI:
```bash theme={"theme":"kanagawa-wave"}
vercel flags enable my-flag development
vercel flags disable my-flag development
```
In dev, the Vercel Toolbar appears in the bottom-right. You can override a flag locally there, but flipping the switch alone does nothing. You have to click **Apply** for the override to write its cookie and reload the page.
## Self-hosted dashboards
Self-hosted dashboards can import `@/lib/flags` without configuring Vercel Flags. If `FLAGS` is missing, the adapter wrapper logs a warning and evaluates flags through the noop adapter.
The noop adapter doesn't contact Vercel, doesn't report flag values, and doesn't load remote targeting rules. It returns each flag's `defaultValue`, so choose safe defaults before using a flag in a self-hosted path.
# Mise
Source: https://engineering.unkey.com/contributing/tooling/mise
How Unkey uses mise for local tooling and repository tasks
## What mise owns
Unkey uses mise as the local toolchain and task runner. The source of truth is
`mise.toml`, `mise.lock`, and `.mise/tasks/*`.
Use mise for local repository commands:
```bash theme={"theme":"kanagawa-wave"}
mise run build
mise run test
mise run fmt
mise exec -- rask ./pkg/cache
```
Do not use Makefiles for repository workflows. They are legacy and can drift
from the pinned toolchain.
## Install the toolchain
Install the pinned mise binary, then install the tools from `mise.toml`:
```bash theme={"theme":"kanagawa-wave"}
./dev/install-mise
mise install
```
If you run into github ratelimit issues, mise can use an auth token to get higher limits
You can manually create a token, or use the one from your `gh` cli:
```bash theme={"theme":"kanagawa-wave"}
GITHUB_TOKEN=$(gh auth token) mise install
```
`mise install` installs languages, CLIs, and package managers. It does not run
repository setup tasks. Repository tasks declare setup work as dependencies, so
you don't need to run setup-only tasks manually.
## How tools are pinned
Add tools to the `[tools]` section in `mise.toml`. Pin exact versions instead
of floating versions like `latest`.
```toml theme={"theme":"kanagawa-wave"}
[tools]
node = "24.19.0"
"npm:pnpm" = "8.6.9"
"github:depot/cli" = "2.101.63"
```
After changing tools, update `mise.lock`:
```bash theme={"theme":"kanagawa-wave"}
mise lock
```
Use the backend-qualified name when needed, for example:
```bash theme={"theme":"kanagawa-wave"}
mise lock npm:pnpm
mise lock github:depot/cli
```
Run `mise install --locked --yes` after updating the lockfile. This catches
missing URLs, checksums, and platform entries before another developer hits the
same issue.
## How tasks work
Repository tasks are executable files under `.mise/tasks/`. The file name is the
task name. Metadata is declared with `#MISE` comments near the top of the file.
```bash .mise/tasks/example theme={"theme":"kanagawa-wave"}
#!/usr/bin/env bash
#MISE description="Run an example workflow"
#MISE depends=["setup-example"]
set -euo pipefail
example-tool run
```
Use task dependencies for setup work. If a task needs another task to prepare
inputs or local state, add `#MISE depends=["task-name"]` instead of telling
users to run setup steps by hand.
## Hidden dependency tasks
Some tasks exist only to prepare other tasks. Mark those tasks hidden:
```bash theme={"theme":"kanagawa-wave"}
#MISE hide=true
```
Hidden tasks can still run as dependencies. Use them for setup work that users
usually don't need to invoke directly.
You can inspect hidden tasks when debugging:
```bash theme={"theme":"kanagawa-wave"}
mise tasks --hidden
mise tasks deps
```
## Cache task outputs
Tasks can declare source and output files. Mise skips the task when all outputs
are newer than the sources.
```bash theme={"theme":"kanagawa-wave"}
#MISE sources=["path/to/input.json"]
#MISE outputs=["path/to/output"]
```
Use this for expensive setup steps with clear inputs and outputs. Do not add
outputs for tasks that must always run, such as tests or formatters.
## Add a task
When you add a task, follow this checklist:
1. Create an executable script in `.mise/tasks/`.
2. Add a clear `#MISE description`.
3. Add `set -euo pipefail` for Bash tasks.
4. Add `#MISE depends=["task-name"]` when the task has setup dependencies.
5. Add `#MISE sources` and `#MISE outputs` only when skip behavior is safe.
6. Mark setup-only tasks with `#MISE hide=true`.
7. Run `mise tasks validate`.
8. Run the smallest task or command that proves the task works.
Prefer a task when the command is part of a repeated repository workflow. Use
`mise exec -- ` for one-off direct tool calls.
## What to expect
Mise tasks run from the repository root unless configured otherwise. Keep task
scripts explicit about paths, for example `pnpm --dir=web` instead of changing
directories for the rest of the script.
Task dependencies run before the requested task. Hidden tasks can still run as
dependencies. `mise tasks` hides them by default, and `mise tasks --hidden`
shows them.
The lockfile is part of the review surface. If `mise.toml` changes, expect a
matching `mise.lock` change unless the edit only affects task configuration or
environment variables.
# Releases
Source: https://engineering.unkey.com/contributing/tooling/releases
How Unkey ships service images from a git tag to production
Releases are tag-driven. Pushing a `/vx.y.z` tag from `main` causes
Depot CI to build the image, push it to GHCR, and cut a GitHub release.
Promotion through canary into production then happens in the
[`unkeyed/infra`](https://github.com/unkeyed/infra) repo. For how the image
itself is built, see [Builds](./builds).
## Design choices worth knowing
* **One image per service, not a monolith.** Each service gets its own
GHCR repository (`ghcr.io/unkeyed/`) and its own tag namespace
(`/vx.y.z`). Services release independently; bumping `api` does
not force a release of `vault`. Helm charts and promotion pins are also
per-service for the same reason.
* **Stable tags must be on `main`.** Stable releases ship code that has
already passed CI. The release workflow refuses to publish a stable tag
whose commit is not reachable from `origin/main`, so a feature branch can
never become a stable release. Pre-release tags (e.g. `-rc.1`) are exempt
and may be cut from any branch so they can be canaried before merging.
* **The release workflow does not re-test.** Tests ran when the code merged
to `main`. The release job's only responsibilities are build, push, and
cut a GitHub release.
* **Tags are immutable.** Once published, a tag (and its image) sticks
around as historical record. Bad releases are addressed by forward-fix
tags, never by deleting or re-pushing the broken one.
## Releasing a service
The unkey side is two steps: tag the service, then wait for CI to publish
the image. The infra side (canary rollout, production promotion, rollback)
lives in the
[infra deploy guide](https://github.com/unkeyed/infra/blob/main/README.md#deploying-unkey-api)
and picks up from there.
### 1. Tag the service with `mise run release`
`mise run release` is the supported way to cut a release. Everything after the
`--` is passed to the tool. It fetches `origin/main` and all tags first, so the
version it picks and the "already exists" check reflect origin rather than your
local state. It then prints the plan and the commits and PRs merged since the
last release, asks for confirmation, and pushes one tag at a time (GitHub drops
tag-push events when more than three land at once).
```bash theme={"theme":"kanagawa-wave"}
# preview without tagging anything
mise run release -- --dry-run api
# patch-bump api and frontline from their latest tags, confirm, push
mise run release -- api frontline
```
Supported services are `api`, `frontline`, `vault`, `heimdall`, `krane`,
`logdrain`, `control-api`, `control-worker`, and `cli`. The version is
auto-numbered from the service's existing tags (patch bump by default). For a
service's very first release there is nothing to number from, so pass an
explicit version (e.g. `api@v0.1.0`).
Common variations:
```bash theme={"theme":"kanagawa-wave"}
mise run release -- --bump minor api # minor bump instead of patch
mise run release -- --rc api # next release candidate (-rc.N)
mise run release -- --version v1.2.3 api vault # pin an exact version for both
mise run release -- api@v1.2.3 vault@v0.4.0-rc.1 # per-service explicit versions
```
`--version` pins an exact version and cannot be combined with
`--bump`/`--rc`/`--pre`. `--yes` skips the confirmation prompt, `--no-log`
hides the changelog, and `--no-fetch` skips the origin fetch for offline use.
#### Pre-release tags
`--rc` (shorthand for `--pre rc`) cuts the next release candidate; `--pre ` does the same for any SemVer pre-release label (`beta`, `alpha`, ...).
Pre-releases build and push the image the same way but produce a GitHub release
marked as a pre-release, so it does not show up as "Latest". Use them when you
want a real, immutable artifact to canary in infra without claiming it as a
stable version. Unlike stable tags, a pre-release may be cut from any branch,
so you can canary a feature branch before merging it to `main`. Tags are
immutable, so iterate by bumping the suffix (`-rc.1` -> `-rc.2`); the tool does
this automatically on each run.
```bash theme={"theme":"kanagawa-wave"}
mise run release -- --rc api # api/v1.2.3-rc.1, then -rc.2 on the next run
```
### 2. Wait for CI
`depot ci run list --repo unkeyed/unkey --trigger push --status running`
shows the in-flight workflow. When it finishes, confirm both artifacts exist
before moving on:
* The image at `ghcr.io/unkeyed/:v..` (visible in the
GitHub Packages tab, or via `docker manifest inspect`).
* A GitHub release at `releases/tag//v..`.
If the workflow fails, fix the underlying issue on `main` and cut a new
patch tag. Do not delete and re-push the failed one.
## After CI: deploy in `unkeyed/infra`
Follow the
[infra deploy guide](https://github.com/unkeyed/infra/blob/main/README.md#deploying-unkey-api).
Two things to be aware of for per-service releases: helm values live at
`eks-cluster/helm-chart//values.yaml` with
`image.repository: ghcr.io/unkeyed/`, and each service has its own
production promotion pin under
`eks-cluster/promotions/production001/.yaml`.
## Walkthrough: shipping a new `api` minor
A full run from code on `main` to production, canarying a release candidate
first. Substitute your own service and version.
1. **Preview the plan.** See what the tool would tag without touching origin.
```bash theme={"theme":"kanagawa-wave"}
mise run release -- --dry-run --bump minor --rc api
```
It prints the next version (e.g. `api/v1.3.0-rc.1`) and the commits and PRs
merged since the last `api` release.
2. **Cut the release candidate.** Drop `--dry-run` and confirm at the prompt.
```bash theme={"theme":"kanagawa-wave"}
mise run release -- --bump minor --rc api
```
The tag is pushed and the tool prints the `depot ci run list` command.
3. **Wait for CI and confirm artifacts.** Watch the run, then check the image
at `ghcr.io/unkeyed/api:v1.3.0-rc.1` and the GitHub pre-release exist
(see [step 2 above](#2-wait-for-ci)).
4. **Canary the RC in infra.** Pin `api` to `v1.3.0-rc.1` and roll it out per
the [infra deploy guide](https://github.com/unkeyed/infra/blob/main/README.md#deploying-unkey-api).
Watch metrics until you trust it.
5. **Cut the stable tag.** Same commit, now without `--rc`. The bump matches
the RC so you get `api/v1.3.0`.
```bash theme={"theme":"kanagawa-wave"}
mise run release -- --bump minor api
```
6. **Wait for CI, then promote.** Confirm `ghcr.io/unkeyed/api:v1.3.0` and the
GitHub release exist, then update the production promotion pin in infra.
If anything looks wrong, never delete or re-push a tag. Fix forward on `main`
and cut the next patch or RC.
## Tagging by hand
`mise run release` is preferred, but the tags it pushes are ordinary git tags,
so you can create them directly when the tool is unavailable (for example from
a machine without the Go toolchain). Sync `main` first, and remember that
stable tags must be on `main` and that CI drops tag-push events when more than
three are pushed at once, so push one at a time.
```bash theme={"theme":"kanagawa-wave"}
git fetch origin main --tags
git checkout main
git pull --ff-only
git tag /v..
git push origin /v..
```
## CLI releases
The CLI ships through a separate path because it is an npm package, not a
container image. `cli/vx.y.z` tags trigger `.github/workflows/release.yaml`,
which uses GoReleaser to build the CLI binaries and publish to npm. The
GoReleaser config is scoped to the `cli/` tag prefix, so a service tag
never produces CLI artifacts and vice versa.
Tag it the same way as a service; only the downstream CI differs:
```bash theme={"theme":"kanagawa-wave"}
mise run release -- cli # auto patch-bump from the latest cli tag
mise run release -- cli@v1.2.3 # or pin an explicit version
```
# Users & Roles
Source: https://engineering.unkey.com/infra/clickhouse/index
Overview of ClickHouse users, roles, and their permissions.
Users managed by ClickHouse (`*-internal`, `default`) and SQL console users (`sql-console*`) are not documented here.... they are provisioned automatically.
Workspace users (`ws_*`) are also excluded, cuz we create them ourselves.
## Users
| User | Purpose | Roles / Grants |
| ---------------------------------------------- | --------------------------------------------------------- | ----------------------------------- |
| [`grafana`](./users/grafana) | Grafana dashboards | `grafana_readonly`, `readonly_role` |
| [`apiv2`](./users/apiv2) | API v2 service | `insertonly_role`, `readonly_role` |
| [`ctrl`](./users/ctrl) | Control plane data writer/reader (ctrl-api + ctrl-worker) | direct grants (see script) |
| [`frontline`](./users/frontline) | Frontline service | direct grants (see script) |
| [`vector`](./users/vector) | Runtime logs | INSERT on `runtime_logs_raw_v1` |
| [`eve`](./users/eve) | Ad-hoc analytics exploration (human) | direct grants (see script) |
| [`github`](./users/github) | GitHub integrations | `readonly_role` |
| [`vercel_dashboard`](./users/vercel-dashboard) | Vercel dashboard | `readonly_role` |
| [`unkey_admin`](./users/unkey-admin) | Admin access | all roles |
## Roles
| Role | Purpose |
| ---------------------------------------------- | ------------------------------------------ |
| [`readonly_role`](./roles/readonly-role) | `SELECT` on `default.*` |
| [`insertonly_role`](./roles/insertonly-role) | `INSERT` on `default.*` |
| [`grafana_readonly`](./roles/grafana-readonly) | Column-level `SELECT` on `system.*` tables |
# grafana_readonly
Source: https://engineering.unkey.com/infra/clickhouse/roles/grafana-readonly
Principle of least privilege: only the specific columns Grafana actually queries are granted, not full table access.
If a new Grafana dashboard needs additional columns, add them here explicitly.
```sql theme={"theme":"kanagawa-wave"}
CREATE ROLE IF NOT EXISTS grafana_readonly;
-- Cluster topology
GRANT SELECT(
cluster, shard_num, replica_num, host_name, host_address,
port, is_local, errors_count, slowdowns_count
) ON system.clusters TO grafana_readonly;
-- Ongoing merges
GRANT SELECT(
database, `table`, elapsed, progress, is_mutation, partition_id,
result_part_path, source_part_paths, num_parts,
total_size_bytes_compressed, bytes_read_uncompressed,
bytes_written_uncompressed, columns_written, memory_usage, thread_id
) ON system.merges TO grafana_readonly;
-- Mutations
GRANT SELECT(
database, `table`, mutation_id, parts_to_do_names,
command, create_time, is_done,
latest_failed_part, latest_fail_time, latest_fail_reason
) ON system.mutations TO grafana_readonly;
-- Replication status
GRANT SELECT(
database, `table`, queue_size, absolute_delay,
is_leader, is_readonly, inserts_in_queue, merges_in_queue
) ON system.replicas TO grafana_readonly;
-- Disk usage
GRANT SELECT(name, path, free_space, total_space)
ON system.disks TO grafana_readonly;
-- Part metadata
GRANT SELECT(
database, `table`, partition, partition_id, bytes_on_disk,
data_uncompressed_bytes, rows, active, modification_time, name,
part_type, level, disk_name, path, marks, refcount,
min_block_number, max_block_number
) ON system.parts TO grafana_readonly;
-- Detached parts
GRANT SELECT(database, `table`, partition_id, name, disk, level)
ON system.detached_parts TO grafana_readonly;
-- Dictionary health
GRANT SELECT(source, type, status)
ON system.dictionaries TO grafana_readonly;
-- Metric logs (accessed via merge('system', '^metric_log'))
-- Full SELECT needed because merge() table function requires table-level access
GRANT SELECT ON system.metric_log TO grafana_readonly;
GRANT SELECT ON system.metric_log_1 TO grafana_readonly;
GRANT SELECT ON system.metric_log_2 TO grafana_readonly;
GRANT SELECT ON system.metric_log_3 TO grafana_readonly;
-- Async metric logs (accessed via merge('system', '^asynchronous_metric_log'))
GRANT SELECT ON system.asynchronous_metric_log TO grafana_readonly;
GRANT SELECT ON system.asynchronous_metric_log_1 TO grafana_readonly;
GRANT SELECT ON system.asynchronous_metric_log_2 TO grafana_readonly;
GRANT SELECT ON system.asynchronous_metric_log_3 TO grafana_readonly;
-- Query log (performance dashboards)
GRANT SELECT(
event_time, query_start_time, query_duration_ms, type,
initial_user, query_kind, query_id, query, normalized_query_hash,
memory_usage, read_rows, read_bytes, written_rows, written_bytes,
result_rows, result_bytes
) ON system.query_log TO grafana_readonly;
-- Remote access for clusterAllReplicas() queries
GRANT REMOTE ON *.* TO grafana_readonly;
-- ============================================================
-- Assign to the grafana user and set as default
-- ============================================================
-- CREATE USER grafana IDENTIFIED WITH sha256_password BY '';
GRANT grafana_readonly TO grafana;
SET DEFAULT ROLE grafana_readonly, readonly_role TO grafana;
```
# insertonly_role
Source: https://engineering.unkey.com/infra/clickhouse/roles/insertonly-role
```sql theme={"theme":"kanagawa-wave"}
CREATE ROLE IF NOT EXISTS insertonly_role;
GRANT INSERT ON default.* TO insertonly_role;
```
# readonly_role
Source: https://engineering.unkey.com/infra/clickhouse/roles/readonly-role
Read-only access to the default database
```sql theme={"theme":"kanagawa-wave"}
CREATE ROLE IF NOT EXISTS readonly_role;
GRANT SELECT ON default.* TO readonly_role;
```
# apiv2
Source: https://engineering.unkey.com/infra/clickhouse/users/apiv2
```sql theme={"theme":"kanagawa-wave"}
CREATE USER IF NOT EXISTS apiv2 IDENTIFIED WITH sha256_password BY '';
GRANT insertonly_role, readonly_role TO apiv2;
```
# eve
Source: https://engineering.unkey.com/infra/clickhouse/users/eve
Read-only analytics user for ad-hoc data exploration
Human exploration user. Gets read access to the analytics tables but not the
raw request/response streams.
Uses a direct `GRANT SELECT ON default.*` plus targeted `REVOKE`s instead of
`readonly_role` because the role grants every table in `default`, including the
raw tables that store request and response bodies, headers, IP addresses, and
free-form logs. Those can contain customer PII and secrets (for example
`Authorization` headers or API keys in request bodies), which an exploration
user has no reason to read. The direct-grant-then-revoke shape still auto-picks
up new rollup tables while keeping the four raw streams out.
```sql theme={"theme":"kanagawa-wave"}
CREATE USER IF NOT EXISTS eve IDENTIFIED WITH sha256_password BY '';
-- Read access to the default database.
GRANT SELECT ON default.* TO eve;
-- Exclude the raw streams that carry bodies / headers / IPs / free-form logs.
REVOKE SELECT ON default.api_requests_raw_v2 FROM eve; -- request/response bodies, headers, IP, UA
REVOKE SELECT ON default.frontline_requests_raw_v1 FROM eve; -- request/response bodies, headers, IP, UA
REVOKE SELECT ON default.runtime_logs_raw_v1 FROM eve; -- free-form app log messages + attributes
REVOKE SELECT ON default.audit_logs_raw_v1 FROM eve; -- actor / target / meta JSON
```
# frontline
Source: https://engineering.unkey.com/infra/clickhouse/users/frontline
Uses direct grants instead of roles because it needs a mix of read and write on specific tables.
```sql theme={"theme":"kanagawa-wave"}
CREATE USER IF NOT EXISTS frontline IDENTIFIED WITH sha256_password BY '';
-- Read access to verification aggregation tables
GRANT SELECT ON default.key_verifications_per_day_v3 TO frontline;
GRANT SELECT ON default.key_verifications_per_hour_v3 TO frontline;
GRANT SELECT ON default.key_verifications_per_minute_v3 TO frontline;
GRANT SELECT ON default.key_verifications_per_month_v3 TO frontline;
-- Read + write access to raw verifications
GRANT SELECT, INSERT ON default.key_verifications_raw_v2 TO frontline;
-- Write access to the key-last-used MV target
-- (the MV trigger runs in the inserting user's context and writes here)
GRANT SELECT ON default.key_last_used_v1 TO frontline;
-- Read + write access to raw frontline requests
GRANT SELECT, INSERT ON default.frontline_requests_raw_v1 TO frontline;
-- Frontline request rollup cascade: raw -> per_minute -> per_5m -> per_15m -> per_hour -> per_day.
-- The MVs trigger in the inserting user's context, and each one reads its source table
-- (SELECT) and writes its target table, so frontline needs SELECT on every level.
GRANT SELECT ON default.frontline_requests_per_minute_v1 TO frontline;
GRANT SELECT ON default.frontline_requests_per_5m_v1 TO frontline;
GRANT SELECT ON default.frontline_requests_per_15m_v1 TO frontline;
GRANT SELECT ON default.frontline_requests_per_hour_v1 TO frontline;
GRANT SELECT ON default.frontline_requests_per_day_v1 TO frontline;
```
# github
Source: https://engineering.unkey.com/infra/clickhouse/users/github
```sql theme={"theme":"kanagawa-wave"}
CREATE USER IF NOT EXISTS github IDENTIFIED WITH sha256_password BY '';
GRANT readonly_role TO github;
```
# grafana
Source: https://engineering.unkey.com/infra/clickhouse/users/grafana
```sql theme={"theme":"kanagawa-wave"}
CREATE USER IF NOT EXISTS grafana IDENTIFIED WITH sha256_password BY '';
GRANT grafana_readonly, readonly_role TO grafana;
SET DEFAULT ROLE grafana_readonly, readonly_role TO grafana;
```
# unkey_admin
Source: https://engineering.unkey.com/infra/clickhouse/users/unkey-admin
Has `default_roles_all: true` set in ClickHouse Cloud config, so it automatically inherits every role without explicit grants.
```sql theme={"theme":"kanagawa-wave"}
CREATE USER IF NOT EXISTS unkey_admin IDENTIFIED WITH sha256_password BY '';
```
# vector
Source: https://engineering.unkey.com/infra/clickhouse/users/vector
```sql theme={"theme":"kanagawa-wave"}
CREATE USER IF NOT EXISTS vector IDENTIFIED WITH sha256_password BY '';
GRANT INSERT ON default.runtime_logs_raw_v1 TO vector;
```
# vercel_dashboard
Source: https://engineering.unkey.com/infra/clickhouse/users/vercel-dashboard
```sql theme={"theme":"kanagawa-wave"}
CREATE USER IF NOT EXISTS vercel_dashboard IDENTIFIED WITH sha256_password BY '';
GRANT readonly_role TO vercel_dashboard;
```
# Overview
Source: https://engineering.unkey.com/infra/index
Documentation for Unkey's infrastructure.
These docs cover how our infrastructure is configured and operated — clusters, deployments, observability, secrets, and ClickHouse.
The actual infrastructure code lives in the private [unkeyed/infra](https://github.com/unkeyed/infra) repo. These docs exist here for a few reasons:
* they're easily searchable on [engineering.unkey.com](https://engineering/unkey.com)
* they can be cross-referenced with other documentation
* we can use mintlify's editor
# PlanetScale query tags
Source: https://engineering.unkey.com/infra/planetscale/query-insights-tags
SQLCommenter metadata for attributing MySQL load in PlanetScale Query Insights
Unkey annotates every MySQL statement with [SQLCommenter](https://google.github.io/sqlcommenter/)-compatible metadata before it reaches PlanetScale. Tags show up in Query Insights so we can attribute query time to services, deploys, sqlc operations, and (for dashboard) tRPC routes.
## Why
PlanetScale Query Insights groups queries by fingerprint. Without tags, hot paths like `FindKeyForVerification` appear as anonymous load. Tags let us answer:
* Which service issued this query?
* Which deploy introduced the regression?
* Which sqlc operation or tRPC route drove the spike?
Keep tag cardinality low. Insights indexes tag keys; high-cardinality values (user ids, key ids) must never appear in comments.
Tag values are URL-encoded per the [SQLCommenter spec](https://google.github.io/sqlcommenter/spec/#value-serialization) before they are quoted, so routes like `POST /v2/keys.verifyKey` serialize safely.
## Tag schema
| Key | Source | Example |
| ------------- | ---------------------------------------- | ------------------------------------ |
| `application` | constant | `unkey` |
| `service` | process name | `api`, `frontline`, `dashboard` |
| `region` | `UNKEY_REGION` (set in helm per cluster) | `us-east-1` |
| `release_sha` | link-time git SHA (7 chars) | `a1b2c3d` |
| `operation` | sqlc `-- name:` header (Go only) | `FindKeyForVerification` |
| `mode` | connection role (Go only) | `rw`, `ro` |
| `route` | request context | `deploy.envVars.create` (tRPC path) |
| `source` | request context | `http`, `restate`, `trpc`, `webhook` |
We intentionally omit an `environment` tag. Infra labels clusters `production001`, `canary`, or legacy `staging`, but those labels are not injected into platform pod env vars today. Use `release_sha` to identify the deployed commit (goreleaser sets `buildinfo.Revision` on Go binaries; Vercel sets `UNKEY_GIT_COMMIT_SHA` / `GIT_COMMIT` for dashboard).
Example comment appended to SQL:
```sql theme={"theme":"kanagawa-wave"}
SELECT ... FROM keys WHERE ... /*application='unkey',service='api',region='us-east-1',release_sha='a1b2c3d',operation='FindKeyForVerification',mode='ro'*/
```
## Go services
Annotation happens in [`pkg/mysql.Replica`](https://github.com/unkeyed/unkey/blob/main/pkg/mysql/replica.go) (and the mirrored [`pkg/db`](https://github.com/unkeyed/unkey/blob/main/pkg/db/database.go) package). sqlc call sites stay unchanged.
1. Pass static tags when opening the database:
```go theme={"theme":"kanagawa-wave"}
tags := sqlcomment.ForService("api", cfg.Region)
database, err := db.New(db.Config{
PrimaryDSN: cfg.DatabasePrimary,
ReadOnlyDSN: cfg.DatabaseReadOnly,
Tags: tags,
})
```
2. Optional dynamic tags via context (HTTP zen services set these automatically with [`zen.WithSQLComment`](https://github.com/unkeyed/unkey/blob/main/pkg/zen/middleware_sqlcomment.go)):
```go theme={"theme":"kanagawa-wave"}
ctx = sqlcomment.WithDynamic(ctx, sqlcomment.Dynamic{Route: "POST /v2/keys.verifyKey", Source: "http"})
rows, err := database.RO().QueryContext(ctx, query, args...)
```
`zen.WithSQLComment()` reads `http.Request.Pattern` (for example `POST /v2/keys.verifyKey`) and is registered in `api` and `frontline` middleware stacks.
`ctrl-worker` wraps the Restate ingress with `sqlcomment.WrapRestateInvokeHandler`, tagging routes like `hydra.v1.DeployService/Deploy` with `source=restate`.
`sqlcomment.Static{}` disables annotation (tests and dev seeds).
Package reference: [`pkg/mysql/sqlcomment`](https://github.com/unkeyed/unkey/blob/main/pkg/mysql/sqlcomment/doc.go).
## TypeScript apps
Dashboard and portal use Drizzle on top of [`createCommentedPool`](https://github.com/unkeyed/unkey/blob/main/web/internal/db/src/commented-pool.ts) from `@unkey/db`. The pool proxy annotates `query` and `execute` on the pool and on connections from `getConnection`, which is what Drizzle uses inside transactions.
Useful tags on the TypeScript side:
* **Static** (per pool): `service`, `region`, `release_sha`
* **Dynamic** (per request): `route` (tRPC procedure path) and `source` (`trpc`, `webhook`, etc.)
Dashboard tRPC sets `route` and `source` through `runWithSqlCommentTags` in the base procedure middleware. Non-tRPC entrypoints (Stripe webhooks, server actions) should wrap their handler with `runWithSqlCommentTags` and set `source` accordingly.
TypeScript pools do not emit `mode`: mysql2 uses a single pool without the read/write split that Go services have, so a `mode` tag would be misleading.
Static tags read `UNKEY_REGION` / `REGION` and `UNKEY_GIT_COMMIT_SHA` / `GIT_COMMIT` for `release_sha`.
## Verification
After deploy to staging:
1. Open PlanetScale Query Insights for the `unkey` database.
2. Find a high-traffic fingerprint (for example `FindKeyForVerification`).
3. Confirm `service`, `operation`, and `release_sha` appear on sampled queries.
4. For dashboard traffic, confirm `route` tags on tRPC-backed queries.
Local unit tests:
```bash theme={"theme":"kanagawa-wave"}
mise exec -- rask ./pkg/mysql/sqlcomment
mise exec -- pnpm --dir=web exec vitest run internal/db/src/sqlcomment.test.ts internal/db/src/commented-pool.test.ts
```
## Agent notes
* Do not add high-cardinality values to SQL comments.
* New Go services: wire `sqlcomment.ForService` in `run.go` when constructing `db.Config`.
* New TS apps using `@unkey/db`: use `createCommentedPool` instead of `mysql.createPool`.
* HTTP handlers that bypass tRPC can set `sqlcomment.WithDynamic` (Go) or `runWithSqlCommentTags` (TS) per request.
# Deploy MySQL database changes
Source: https://engineering.unkey.com/infra/planetscale/schema-changes
Make safe changes to our database through PlanetScale
Use this runbook after changing Unkey's MySQL schema.
Important distinction:
* `pull request`: A proposed code change on GitHub.
* `deploy request`: A proposed database schema change on PlanetScale.
## 1. Prepare the GitHub pull request
Update the
Drizzle schema ,
then regenerate the SQL files used by Go tooling and code generation:
```bash theme={"theme":"kanagawa-wave"}
mise run generate-sql
```
Review and commit all generated files. Push the branch, then open a non-draft GitHub pull request from the Unkey repository.
The PlanetScale PR Branch
The
Database Schema Gate / Compare production schema
## 2. Open and review the staging deploy request
After the deploy request on PlanetScale is created, open the PlanetScale [dashboard](https://app.planetscale.com/unkey/unkey/deploy-requests) and find the deploy request
from `pr-` into `staging`.
Review the schema diff and ask for a sanity check from other engineers.
Usually you want to merge the deploy request right away, because otherwise you cannot test anything in our canary environment.
A bad migration here is recoverable, but due diligence can prevent canary disruption and cleanup work.
If you update the GitHub pull request, CI reapplies its Drizzle schema to the
same `pr-` branch and reuses the open deploy request. Review the latest
diff before continuing.
## 3. Merge the deploy request into staging
After the staging deploy request has been reviewed, merge it into `staging`.
Wait for PlanetScale to report that the deployment completed successfully.
## 4. Gather reviews on the GitHub pull request
Get the required reviews on the GitHub pull request while the database change
is in PlanetScale `staging`. Address feedback and repeat the staging review if a
new commit changes the Drizzle schema.
Don't merge the GitHub pull request yet. Continue after it has sufficient
approvals.
## 5. Create the production deploy request
In PlanetScale, create a deploy request from `staging` into `main`. This deploy
request promotes the reviewed staging schema to production.
## 6. Get approval for the production deploy request
Review the final schema diff and request approval from another engineer. Confirm
that the deploy request contains the expected database change before approving
it.
## 7. Merge the deploy request into main
After approval, merge the deploy request into `main`. Wait for PlanetScale to
report that the production deployment completed successfully.
## 8. Rerun the production schema gate
PlanetScale doesn't trigger GitHub checks when a deploy request completes.
Manually rerun
Database Schema Gate / Compare production schema
Don't merge the GitHub pull request before the database change reaches
PlanetScale `main`.
After merging, you can optionally release and deploy affected services
through ArgoCD. Application deployment is outside the scope of this runbook.
If you close a GitHub pull request without merging it, the
PlanetScale Cleanup` branch. If the
pull request merges, the job leaves the branch and deploy request available so
an unfinished database promotion can still be completed.
## Changing an existing column
Adding a column or a table follows the runbook above with no special handling.
Changing a column that already exists in production needs more care, because
PlanetScale rejects some shapes of change outright.
### Renames are rejected
Vitess refuses any deploy request whose diff renames a column:
```
Table `portal_sessions` has a column renamed from `portal_config_id` to `portal_id`.
Column renames are not supported. Please first add a new column, copy the data
and then drop the old column.
```
A deploy request compares the desired end state of its source branch against the
current state of its target branch, and in an end state comparison "renamed `a`
to `b`" is indistinguishable from "dropped `a`, added `b`". PlanetScale refuses
rather than risk discarding the column's data.
Three consequences that are easy to get wrong:
* It fires even when the table is empty. Row count is irrelevant; only the shape
of the diff matters.
* No sequence of statements on the branch avoids it. Dropping and recreating the
table on your branch does not help, because only the end state is compared.
* **It applies to every deploy request, not just the one out of your
`pr-` branch.** The `staging` into `main` deploy request is diffed the
same way. If `staging` has both the add and the drop while `main` has neither,
that diff is rename shaped and is rejected, even though each pass looked clean
on its way into `staging`.
Renaming or dropping a *table* is fine. Only column renames are blocked.
### What counts as a rename
The check pairs a dropped column with an added one of the same type. That makes
some changes rename shaped even when no rename was intended:
* Replacing `branding json` with `logo_url varchar(500)` and
`primary_color varchar(7)` is safe, because no dropped column matches an added
one by type.
* But dropping `return_url varchar(500)` from the same table while adding
`logo_url varchar(500)` **is** rename shaped, since the types match exactly and
the other addition differs in length. Two unrelated changes to one table can
pair up by accident.
Moving a column between two tables is not a rename. The check is per table, so
dropping `portals.return_url` while adding `portal_sessions.return_url` needs no
special handling.
Do not rely on ambiguity to save you. A drop can slip through unflagged when
several additions share its type, but that is an accident of the heuristic rather
than a guarantee.
### Split the change into two deploy requests
The two passes are two separate trips through steps 1 to 8. Take the first one
all the way to `main` before you start the second:
```
DR1 (additive): pr- -> staging -> main
DR2 (drops): pr- -> staging -> main
```
Not `pr-` and `pr-` into `staging`, then one `staging` into `main`. That
collapses the add and the drop back into a single diff against `main`, and the
production deploy request is rejected as a rename. Each pass must be present in
`main` before the next one enters `staging`.
Add the new column and keep the old one. Make the new column nullable even
when the target schema declares it `NOT NULL`: currently deployed code
inserts without mentioning it, so `NOT NULL` with no default breaks those
writes as soon as it deploys. Backfill from the old column if the data
matters. Run steps 1 to 8, merging into `staging` and then into `main`.
Confirm the column exists in `main` before continuing.
Ship the code that reads and writes the new column, so nothing depends on the
old one.
Open a second GitHub pull request that updates the Drizzle schema to its
final shape. The diff now drops the old column and tightens the new one to
`NOT NULL`. A diff with no additions cannot be read as a rename. Run steps 1
to 8 again, `staging` first and then `main`.
Tightening to `NOT NULL` fails if the additive pass left NULLs behind, so
backfill or clear the table before the drop pass.
Route both passes through `staging`. Do not shortcut a branch straight into
`main`, and do not skip `main` at the end of the additive pass.
A deploy request needs the source branch's snapshot of its target to be
current. `staging` is long lived, so advancing `main` by any other route leaves
it behind, and every later `staging` into `main` deploy request fails with
``Table `x` can't be modified because it has changed since this branch was
created``. Recovering means refreshing the schema from the deploy request in
the PlanetScale UI, or recreating `staging`, which costs its data and its
credentials.
A deployed deploy request stays in `complete_pending_revert` until its revert
window closes. While it does, a second deploy request touching the same tables
fails to lint with `table_conflict`, which reads like a schema problem but is
not. This bites in the gap between the two passes, since both touch the same
table. Finalize the earlier one before opening the next:
```bash theme={"theme":"kanagawa-wave"}
pscale deploy-request skip-revert unkey --org unkey
```
### Transitional columns reach new databases
`pkg/mysql/schema/*.sql` is both the desired end state and the input to
`dev/Dockerfile.mysql`, which feeds it to `docker-entrypoint-initdb.d`. A column
kept only to satisfy the additive pass is therefore created in every new local
and test database until the drop pass lands. Keep the gap between passes short,
and comment the column in the Drizzle schema so the next reader knows it is
scheduled for removal.
### Collation statements in push output are not a diff
`drizzle-kit push` prints `ALTER TABLE ... MODIFY COLUMN ... COLLATE
utf8mb4_0900_as_cs` for many columns on every run. Its introspection does not
recognize the collation already applied, so it re-emits the statement. Executing
it changes nothing.
Do not remove the collation from the Drizzle schema to silence this. The case
sensitive collation on ids and hashes is deliberate, and dropping it makes key
lookups case insensitive.
These statements cannot fail the production schema gate, which asserts on
`pscale branch diff` rather than on push output. If the gate fails, read the
`Diff check` and `::error::` lines in the job log for the real diff.
# Create a key migration
Source: https://engineering.unkey.com/infra/runbooks/key-migration
How to set up a key migration when a customer wants to import existing API keys into Unkey
Customers with existing API keys can import them into Unkey through
[`/v2/keys.migrateKeys`](https://www.unkey.com/docs/api-reference/keys/migrate-api-keys).
Before they can call that endpoint, someone on our side has to create a
migration for their workspace. This runbook covers how to do that and what to
send back to the customer.
## How key migrations work
A migration is a row in the `key_migrations` table
(`pkg/mysql/schema/key_migrations.sql`) with three fields: the customer-facing
`id` (what they pass as `migrationId`), the `workspace_id` it is scoped to, and
the `algorithm` that describes how their existing keys are hashed.
When the customer calls `/v2/keys.migrateKeys`
(`svc/api/routes/v2_keys_migrate_keys/handler.go`), we store their hashes
verbatim and mark each key with `pending_migration_id`. Hashes that already
exist are returned in the `failed` array instead of failing the request.
Identities, roles, and permissions referenced by the imported keys are created
on the fly if they do not exist yet.
The algorithm decides what happens at verification time:
* `sha256`: the customer submits the base64 encoded (standard encoding, not
URL-safe) SHA-256 hash of the full key. This is exactly Unkey's native hash
format (`pkg/hash`), so migrated keys verify through the normal
`/v2/keys.verifyKey` lookup with no extra parameters. Hex encoded SHA-256
does not work.
* `github.com/seamapi/prefixed-api-key`: for keys in the Seam
`prefix_shortToken_longToken` format. The customer submits the hex encoded
SHA-256 hash of the long token only. These keys are found during
verification only when the customer includes `migrationId` in the
`/v2/keys.verifyKey` request. On the first successful verification we
re-hash the key to our native format, set the display prefix, and clear
`pending_migration_id` (`internal/services/keys/get_migrated.go`).
Any other hash scheme (bcrypt, hex SHA-256 of the full key, HMAC, etc.)
requires a code change: add a value to the `algorithm` enum in
`pkg/mysql/schema/key_migrations.sql` and a case to the switch in
`internal/services/keys/get_migrated.go`. Scope that with the team before
promising it to a customer.
## What to ask the customer
Before creating anything, you need their workspace ID, what system the keys
live in today, the exact hash algorithm and encoding, the key format
(prefixes, separators), and a rough key count. The
[public migration guide](https://www.unkey.com/docs/platform/apis/migrations/introduction)
asks them to include most of this in their first email.
If they still have plaintext keys, steer them to `sha256`: they hash the keys
themselves and everything works with zero special handling. Only use the
prefixed-api-key algorithm when they exclusively store hashes of a token
segment and cannot re-hash.
## Create the migration
Connect to the production MySQL database and insert the migration row. The
`id` is customer-facing and globally unique; use the `mig_`
convention, for example `mig_acme`:
```sql theme={"theme":"kanagawa-wave"}
INSERT INTO key_migrations (id, workspace_id, algorithm)
VALUES ('mig_acme', 'ws_XXX', 'sha256');
```
Double-check the workspace ID belongs to the requesting customer before
inserting. Both `/v2/keys.migrateKeys` and the verification lookup resolve the
migration scoped to the caller's workspace, so a wrong workspace ID surfaces
to the customer as `err:unkey:data:migration_not_found`.
Confirm the row:
```sql theme={"theme":"kanagawa-wave"}
SELECT id, workspace_id, algorithm FROM key_migrations WHERE id = 'mig_acme';
```
## Reply to the customer
Send them the `migrationId`, state the exact hash format the migration
expects, and link the endpoint docs. Fill in the placeholders, and for the
`sha256` algorithm state the format as "sha256 hashed and base64 encoded".
```text theme={"theme":"kanagawa-wave"}
Hey ,
I've created a new migration for you, the id is mig_ and
you'll need that in the api request below.
This migration expects your existing keys as . Let me know if that is going to be a problem.
To migrate your keys into unkey, all you need to do is call this endpoint:
https://www.unkey.com/docs/api-reference/keys/migrate-api-keys
For example:
curl -X POST https://api.unkey.com/v2/keys.migrateKeys \
-H "Content-Type: application/json" \
-H "Authorization: Bearer " \
-d '{
"migrationId": "mig_",
"apiId": "",
"keys": [
{
"hash": ""
}
]
}'
Each key can carry optional fields like externalId, meta, ratelimits, or
expiry, see the docs above. You can send keys in batches too of course.
To verify migrated keys, call our /v2/keys.verifyKey endpoint and include
the migrationId:
https://www.unkey.com/docs/api-reference/keys/verify-api-key
curl -X POST https://api.unkey.com/v2/keys.verifyKey \
-H "Content-Type: application/json" \
-H "Authorization: Bearer " \
-d '{
"key": "",
"migrationId": "mig_"
}'
Keep sending the migrationId until every active key has been verified at
least once, after that you can drop it.
There's also a step-by-step guide if you want more detail:
https://www.unkey.com/docs/platform/apis/migrations/keys
Let me know if you need help with anything.
```
Always telling the customer to pass `migrationId` during verification is
deliberate: for `sha256` migrations it is redundant (the native lookup finds
the key directly and the parameter is ignored), but for every other algorithm
it is required, and a template that works for both cannot be applied wrong.
## After the customer migrates
The endpoint returns HTTP 200 even on partial success. If the customer
reports entries in the `failed` array, those hashes already exist in the
system, usually from an earlier partial run. Look the hashes up in the `keys`
table to confirm whether they belong to the same workspace before advising
the customer to skip or clean them up.
Keys imported under a migration keep `pending_migration_id` set until their
first verification through the migration path. For `sha256` migrations the
column stays set (the native lookup never consults it), which is harmless.
# Create a legacy invoice manually
Source: https://engineering.unkey.com/infra/runbooks/legacy-billing
How to create and review a draft Stripe invoice for a workspace that still uses legacy billing
The legacy billing workflow creates a standalone Stripe draft invoice for one
workspace and one completed calendar month. Use it only for workspaces that
still store pricing in `workspaces.subscriptions` and don't have an API or
Compute Stripe subscription.
Running this workflow writes an invoice and invoice items to the Stripe
account configured on the worker. It doesn't have a dry-run mode. Confirm the
workspace, month, pricing, usage data, and worker environment before running it.
The workflow never finalizes, sends, pays, or attaches the invoice to a
subscription. It leaves the complete invoice in draft state for review.
## Understand what the workflow bills
The workflow reads pricing and usage from existing Unkey data before it writes to
Stripe:
* `plan` and `support` entries in `workspaces.subscriptions` become full fixed
charges with quantity 1.
* `verifications` and `ratelimits` entries use billable ClickHouse usage for
the requested workspace, year, and month.
* Free tiers consume usage without creating invoice lines. Each paid tier
creates one line for the quantity inside that tier.
* Fixed and usage lines have no proration. Every line belongs directly to the
standalone invoice, not to a subscription.
The selected month must be complete in UTC. The workflow uses the current legacy
subscriptions JSON, not a historical pricing snapshot. Before billing an older
month, confirm that the stored prices and tiers applied during that month.
This process doesn't replace subscription billing or Deploy billing. The workflow
stops if `billing_subscriptions` contains an API or Compute Stripe subscription
ID for the workspace.
## Complete the preflight checks
Complete every check before you invoke the workflow. It starts writing to
Stripe after the database and usage checks pass.
1. Confirm the workspace ID and customer-approved billing month.
2. Confirm the month has ended in UTC.
3. Confirm ClickHouse ingestion is complete for that month. A missing monthly
aggregate can appear as zero usage.
4. Inspect the workspace's `subscriptions` JSON and confirm each product ID,
fixed price, free tier, paid tier, and boundary against the agreement that
applied during the selected month.
5. Confirm `stripe_customer_id` belongs to the intended customer in the
intended Stripe environment.
6. Confirm both subscription IDs are empty. Don't bypass this check to bill a
subscription customer.
7. Search Stripe for an existing invoice for the workspace and billing month.
The workflow also performs an account-wide metadata scan and stops if it
finds multiple matches.
You can inspect the MySQL records with this read-only query:
```sql theme={"theme":"kanagawa-wave"}
SELECT
w.id,
w.name,
w.deleted_at_m,
w.subscriptions,
wb.stripe_customer_id,
wb.deleted_at_m AS billing_deleted_at_m,
bs_api.stripe_subscription_id AS api_subscription_id,
bs_compute.stripe_subscription_id AS compute_subscription_id
FROM workspaces AS w
LEFT JOIN workspace_billing AS wb
ON wb.workspace_id = w.id
LEFT JOIN billing_subscriptions AS bs_api
ON bs_api.workspace_id = w.id AND bs_api.product = 'api'
LEFT JOIN billing_subscriptions AS bs_compute
ON bs_compute.workspace_id = w.id AND bs_compute.product = 'compute'
WHERE w.id = 'ws_XXX';
```
Stop if the workspace or billing record is deleted, the legacy subscriptions
JSON is empty or unexpected, the customer ID is wrong, or either subscription
ID is present.
## Create the draft invoice
Invoke Restate ingress for workspace `ws_XXX` and July 2026:
```bash theme={"theme":"kanagawa-wave"}
curl --fail-with-body \
-X POST \
-H 'content-type: application/json' \
"${RESTATE_INGRESS_URL}/hydra.v1.LegacyBillingWorkflow/Run" \
-d '{"workspaceId":"ws_XXX","year":2026,"month":7}'
```
The worker exposes this workflow only when a real ClickHouse client and Stripe
secret are configured.
The workflow performs these operations in order:
1. Validates the request and completed month.
2. Loads the workspace, legacy pricing, Stripe customer, and subscription
state from MySQL.
3. Loads verification and ratelimit usage from ClickHouse when those products
exist in the legacy pricing.
4. Validates all prices and tiers and builds at most 250 invoice lines.
5. Scans Stripe for an invoice tagged with the tool source, workspace ID, and
billing period.
6. Creates a standalone invoice with `auto_advance=false` and excludes unrelated
pending customer invoice items, or resumes one matching draft.
7. Adds only missing invoice lines with deterministic idempotency keys.
8. Retrieves and validates the invoice and every line again before reporting
success.
MySQL, ClickHouse, and Stripe operations are durable Restate steps. External
provider retries are bounded. If an invocation fails, use the recovery
procedure below instead of changing the invoice manually.
A successful invocation returns JSON in this form:
```json theme={"theme":"kanagawa-wave"}
{"invoiceId":"in_XXX","itemCount":3,"verifications":"175000","ratelimits":"0"}
```
Copy the invoice ID into the operational ticket so another operator can audit
what was created.
## Review the invoice in Stripe
Open the returned invoice ID in the worker's Stripe account. Complete this
review before any separate finalization process:
1. Confirm the invoice status is `draft` and automatic advancement is off.
2. Confirm the Stripe customer and workspace metadata match the request.
3. Confirm the billing-period metadata and displayed custom field match the
requested UTC month.
4. Confirm there is no subscription association.
5. Compare every fixed charge with the legacy subscriptions JSON.
6. Compare every usage tier quantity with the approved ClickHouse totals and
tier boundaries.
7. Confirm unrelated pending customer invoice items aren't present.
8. Record the review and invoice URL in the operational ticket.
The workflow doesn't finalize the invoice. Use the separately approved billing
process after review if the draft must be finalized or sent.
The draft uses Stripe's `charge_automatically` collection method. Finalizing
it through another process can initiate collection from the customer's
default payment method. Don't finalize the draft until the invoice and the
collection action have both been approved.
## Recover from an interrupted run
MySQL, ClickHouse, and Stripe steps retry for up to 15 minutes. If a step
exhausts that retry window, the invocation completes with a terminal failure.
After fixing the cause, invoke the workflow again with the same request. Stripe
metadata, idempotency keys, and reconciliation ensure the existing draft is
resumed and only missing lines are created.
Don't delete lines, add manual lines, or change workflow metadata before retrying.
The workflow stops when an existing line differs from the expected product,
quantity, period, currency, or charge specification. This fail-closed behavior
prevents a retry from silently changing an invoice.
Investigate instead of bypassing these errors:
* Active subscription: The workspace is handled by subscription billing or
has inconsistent billing data. Don't create a legacy standalone invoice.
* Multiple invoices: More than one Stripe invoice has the same workspace
and period metadata. Review all matches before taking any action.
* Unexpected invoice state: The matching invoice has a different customer,
isn't a draft, has automatic advancement enabled, or has a subscription
parent. Don't rerun against another Stripe account to avoid the error.
* Unexpected or changed line: Someone or another process changed the draft,
or the legacy pricing changed between runs. Compare the draft with the source
data and resolve the discrepancy manually.
* Incomplete month: Wait until the UTC month has ended and usage ingestion
is complete.
* ClickHouse or MySQL error: Restore read access or service availability.
The workflow doesn't treat query failures as zero usage.
If a draft needs to be deleted, finalized, voided, or otherwise corrected,
handle that as a separate reviewed Stripe operation. The billing workflow performs
none of those actions.