Skip to main content
The logdrain service (svc/logdrain) streams audit logs from ClickHouse to customer-owned destinations: generic HTTP endpoints and Axiom. Under one valid lease, it sends events in cursor order. It provides at-least-once delivery. MySQL stores configuration and delivery progress. The service does not use an external queue or workflow engine.

​
How the service works

Each process runs a lease service and a delivery engine. The lease service assigns drains to the process and keeps those assignments valid. The delivery engine polls for assigned drains that are due, then gives them to a worker pool. A worker first confirms its assignment in MySQL. It then reads the drain’s configuration and cursor, queries the next audit log batch from ClickHouse, and sends that batch to the customer destination. After the destination acknowledges the batch, the worker advances the cursor in MySQL. A failed delivery leaves the cursor unchanged. MySQL stores the destination configuration as a serialized logdrain.v1.Config protobuf. Its provider oneof keeps HTTP and Axiom fields separate. Secret fields contain Vault ciphertext, so MySQL does not store plaintext credentials. The Vault ciphertext includes the key ID that the service needs for decryption. HTTP header names stay in plaintext. Each HTTP header value has separate Vault ciphertext. The worker does not hold a MySQL transaction while it calls the customer destination. A fencing token protects each state read and write instead.

​
Delivery cycle

On each poll, the delivery engine finds enabled, active drains assigned to its lease ID that have reached next_attempt_at. It passes the drain ID and fencing token to a worker. The worker reads the drain only if that token still identifies a valid lease. If the lease expired or another process acquired it, the worker stops. For a valid lease, the worker reads events between the committed cursor and a safe upper time boundary. It sends the events as one batch. After a successful request, a fenced write advances the cursor only if the lease is still valid. An unacknowledged response records its status and body in delivery telemetry. An unexpected delivery failure records its error in delivery telemetry. Both cases schedule a retry without advancing the cursor. If a full batch succeeds, the worker reads the next batch immediately. A short or empty batch means the worker reached the safe upper time boundary. The worker commits that boundary, schedules the next attempt after PollInterval, and returns to the pool.

​
Cursor and watermark

Each drain subscribes to one stream. The supported stream is audit_logs, backed by the ClickHouse table audit_logs_raw_v1. The cursor is the pair (inserted_at, event_id). MySQL stores it as committed_offset_inserted_at and committed_offset_event_id. MySQL and ClickHouse must order event_id values the same way. ClickHouse compares String values byte by byte, so the MySQL committed_offset_event_id column uses the binary utf8mb4_0900_bin collation. If the two databases used different ordering, they could disagree about which event follows the committed cursor and skip or repeat events. The event ID is necessary because multiple events can have the same inserted_at value. For example, assume a batch size of two and these events: The first query returns evt_a and evt_b, then commits the cursor (1000, evt_b). The next query starts after that pair, so it returns evt_c before evt_d. A cursor that stored only 1000 would start the next query after that millisecond and skip evt_c. A full batch advances the cursor to its last event. A short or empty batch advances the cursor to the safe upper time boundary with an empty event ID. The empty event ID keeps events at that exact boundary eligible for the next cycle. The upper boundary stays behind the process clock by WatermarkLag. This settling period gives late ClickHouse inserts time to arrive before the cursor passes their timestamp.

​
Leasing and fencing

Leases divide drains between service processes. Each process creates one startup-unique lease_id. The lease service acquires expired leases for that ID and refreshes them before they expire. Database time controls acquisition, refresh, and expiry. Each acquisition also creates a new fencing_token. The token identifies one specific ownership period, even if the same process loses and later reacquires the drain. The poller includes this token with each work item. Every delivery state read and write requires the same token and a lease that is still valid. As a result, stale workers cannot change the cursor, retry time, or failure state. Fencing protects durable state. It cannot cancel a customer request that started before the lease expired.

​
Disablement, retries, and pauses

The logdrains.enabled Boolean records whether the user enabled the drain. The logdrains.status field is active or paused_by_failure. The engine processes a drain only when it is enabled and active. A destination rejection, such as HTTP 400 or HTTP 500, leaves the cursor unchanged. An unexpected delivery error, such as a timeout or DNS failure, also leaves the cursor unchanged. The engine retries both cases. The engine calculates an exponential delay from consecutive_failures. The first retry waits 1 minute. Each later failure doubles the delay through 128 minutes. Later retries wait 4 hours. With the default failure threshold of 50, the 49 retry waits span 7 days and 15 minutes. A destination can request a longer delay. The HTTP sink reads the standard Retry-After response header. The Axiom sink reads Retry-After first. If that header is not valid, it reads X-RateLimit-Reset as a UTC Unix timestamp in seconds. The engine uses the longer of the local delay and the destination delay. It limits a destination delay to 24 hours. The failure update query increments consecutive_failures and writes an absolute retry time to next_attempt_at. It calculates that timestamp from MySQL time, not process time. The due-drain queries return the drain only after next_attempt_at has passed. A successful cursor update resets the failure count. A full batch remains due immediately. A short or empty batch becomes due after PollInterval. The engine pauses the drain only after it reaches the configured failure threshold. Re-enabling a drain or changing its destination makes it active. The change resets its failure count and retry time. It preserves the cursor. Delivery resumes from that cursor, subject to the 90-day ClickHouse retention period for audit logs.

​
Delivery guarantees

The cursor, lease, and fencing token define the delivery semantics:
  • Deliver-then-commit gives at-least-once delivery. If the process crashes after the destination acknowledges a batch but before the cursor update, the next cycle sends that batch again.
  • A failure never advances the cursor. The drain retries the same events until the destination accepts them or the drain pauses.
  • Fencing prevents stale workers from changing durable state after lease expiry, configuration changes, deletion, or reacquisition. It cannot cancel an external request after a worker completes its fenced read.
Customers must handle duplicate events. They can use the audit log event id as a deduplication key.

​
Health telemetry

Each destination attempt writes one row to the ClickHouse table logdrain_deliveries_raw_v1. The row contains the outcome (success or error), event count, webhook_duration_ms, request_body_bytes, and a truncated error message. request_body_bytes contains the uncompressed encoded body size. It does not include request headers. A rejected HTTP response also includes the response status and up to 4 KiB of the response body. The duration measures only the destination request. The dashboard reads this table for the drain health charts and delivery errors.

​
Prometheus metrics

The Prometheus endpoint reports scheduling, delivery, and failure health for the service.