Skip to main content
The logdrain service (svc/logdrain) streams audit logs from ClickHouse to customer-owned destinations: generic HTTP endpoints and Axiom. Under one valid lease, it sends events in cursor order. It provides at-least-once delivery. MySQL stores configuration and delivery progress. The service does not use an external queue or workflow engine.

​
How the service works

Each process runs a lease service and a delivery engine. The lease service assigns drains to the process and keeps those assignments valid. The delivery engine polls for assigned drains that are due, then gives them to a worker pool. A worker first confirms its assignment in MySQL. It then reads the drain’s configuration and cursor, queries the next audit log batch from ClickHouse, and sends that batch to the customer destination. After the destination acknowledges the batch, the worker advances the cursor in MySQL. A failed delivery leaves the cursor unchanged. MySQL stores the destination configuration as a serialized logdrain.v1.Config protobuf. Its provider oneof keeps HTTP and Axiom fields separate. Secret fields contain Vault ciphertext, so MySQL does not store plaintext credentials. The Vault ciphertext includes the key ID that the service needs for decryption. HTTP header names stay in plaintext. Each HTTP header value has separate Vault ciphertext. The stream oneof identifies the stream and contains its typed configuration, independently of the destination. audit_logs selects AuditLogStreamConfig. Its event_types list selects exact audit event names. An empty list means all audit events, including event types added later. Existing protobuf configs with no stream retain the legacy audit-log behavior without filters. The MySQL stream column remains the scheduling index and must agree with the config. Deploy workers that understand the stream config before enabling the dashboard selector; older workers ignore it. ClickHouse applies the filter before ordering and limiting each batch. Windows with no matching events still advance the cursor. Editing the filter expires the lease and resumes from the committed cursor with a new fencing token. It does not replay events skipped by the previous filter. A request already in flight can still reach the destination, but its old fence cannot commit. The worker does not hold a MySQL transaction while it calls the customer destination. A fencing token protects each state read and write instead.

​
Delivery cycle

On each poll, the delivery engine finds running drains assigned to its lease ID that have reached next_attempt_at. It passes the drain ID and fencing token to a worker. The worker reads the drain only if that token still identifies a valid lease. If the lease expired or another process acquired it, the worker stops. The engine owns leases, delivery, and durable progress. A batchReader owns source queries and adaptive windows. It returns a page containing events, a proposed next cursor, and a caughtUp flag. Reading a page does not commit its cursor. The engine commits only after the destination acknowledges all events in that page. Empty pages need no destination request. After each commit, the engine immediately reads another page unless caughtUp is true. A partial or empty page can exhaust one window without reaching the watermark. It does not cause a poll delay while a backlog remains. Cancellation stops the cycle. It does not mark a source cancellation as a retryable failure. If the failure-state write fails or loses its fence, the worker also stops. These exits never authorize a cursor advance.

​
Adaptive read windows

Each cycle starts with a 1-minute window and one fixed watermark. The reader sets windowEnd = min(cursor.Time + windowSize, watermark). Its query reads after the composite cursor and strictly before windowEnd, with at most BatchSize events. Empty windows grow through 1, 2, 4, 8, 16, 32, and 60 minutes. Further empty windows stay at 60 minutes. A full batch resets the next query to 1 minute from the last event’s cursor, including its event ID. A partial batch keeps the window size. Every query is capped at the fixed watermark, even when the chosen window is larger than the remaining interval. Window size lives only in the reader for that cycle. It is not stored in MySQL. A restart, retry, or later poll starts at 1 minute again, using the committed cursor. A larger window can still be expensive when an empty period is followed by a burst. Adaptation limits the time range, not query duration or rows scanned.

​
Cursor and watermark

Each drain subscribes to one stream. The supported stream is audit_logs, backed by the ClickHouse table audit_logs_raw_v1. The cursor is the pair (inserted_at, event_id). MySQL stores it as committed_offset_inserted_at and committed_offset_event_id. Creation sets the cursor to the drain’s creation time with an empty event ID. Historical backfill before creation is not supported. Pauses, failures, and outages resume from the committed cursor and still catch up normally. A zero cursor follows the same adaptive window path from the Unix epoch. It eventually catches up without a separate history lookup, but can require many empty-window queries. MySQL and ClickHouse must order event_id values the same way. ClickHouse compares String values byte by byte, so the MySQL committed_offset_event_id column uses the binary utf8mb4_0900_bin collation. If the two databases used different ordering, they could disagree about which event follows the committed cursor and skip or repeat events. The event ID is necessary because multiple events can have the same inserted_at value. For example, assume a batch size of two and these events: The first query returns evt_a and evt_b, then commits the cursor (1000, evt_b). The next query starts after that pair, so it returns evt_c before evt_d. A cursor that stored only 1000 would start the next query after that millisecond and skip evt_c. A full batch advances the cursor to its last event. A short or empty batch advances the cursor to that window’s exclusive end with an empty event ID. The empty event ID keeps events at that exact boundary eligible for the next read, which can happen in the same cycle. The watermark is the poll’s enqueue timestamp minus WatermarkLag. It stays fixed while the worker catches up, rather than moving forward with every batch. This settling period gives late ClickHouse inserts time to arrive before the cursor passes their timestamp.

​
Leasing and fencing

Leases divide drains between service processes. Each process creates one startup-unique lease_id. The lease service acquires expired leases for that ID and refreshes them before they expire. Database time controls acquisition, refresh, and expiry. Each acquisition also creates a new fencing_token. The token identifies one specific ownership period, even if the same process loses and later reacquires the drain. The poller includes this token with each work item. Every delivery state read and write requires the same token and a lease that is still valid. As a result, stale workers cannot change the cursor, retry time, or failure state. Fencing protects durable state. It cannot cancel a customer request that started before the lease expired.

​
Disablement, retries, and pauses

The logdrains.status field is running, paused_by_user, or paused_by_failure. The engine processes a drain only when it is running. A destination rejection, such as HTTP 400 or HTTP 500, leaves the cursor unchanged. An unexpected delivery error, such as a timeout or DNS failure, also leaves the cursor unchanged. The engine retries both cases. The engine calculates an exponential delay from consecutive_failures. The first retry waits 1 minute. Each later failure doubles the delay through 128 minutes. Later retries wait 4 hours. With the default failure threshold of 50, the 49 retry waits span 7 days and 15 minutes. A destination can request a longer delay. The HTTP sink reads the standard Retry-After response header. The Axiom sink reads Retry-After first. If that header is not valid, it reads X-RateLimit-Reset as a UTC Unix timestamp in seconds. The engine uses the longer of the local delay and the destination delay. It limits a destination delay to 24 hours. The failure update query increments consecutive_failures and writes an absolute retry time to next_attempt_at. It calculates that timestamp from MySQL time, not process time. The due-drain queries return the drain only after next_attempt_at has passed. A successful cursor update resets the failure count. A page remains due immediately while a backlog remains. Only a page that reaches the watermark schedules the next attempt after PollInterval. The engine pauses the drain only after it reaches the configured failure threshold. Re-enabling a drain or changing its destination makes it running. The change resets its failure count and retry time. It preserves the cursor. Delivery resumes from that cursor, subject to the 90-day ClickHouse retention period for audit logs.

​
Delivery guarantees

The cursor, lease, and fencing token define the delivery semantics:
  • Deliver-then-commit gives at-least-once delivery. If the process crashes after the destination acknowledges a batch but before the cursor update, the next cycle sends that batch again.
  • A failure never advances the cursor. The drain retries the same events until the destination accepts them or the drain pauses.
  • Fencing prevents stale workers from changing durable state after lease expiry, configuration changes, deletion, or reacquisition. It cannot cancel an external request after a worker completes its fenced read.
Customers must handle duplicate events. They can use the audit log event id as a deduplication key.

​
Health telemetry

Each destination attempt writes one row to the ClickHouse table logdrain_deliveries_raw_v1. The row contains the outcome (success or error), event count, webhook_duration_ms, request_body_bytes, and a truncated error message. request_body_bytes contains the uncompressed encoded body size. It does not include request headers. A rejected HTTP response also includes the response status and up to 4 KiB of the response body. The duration measures only the destination request. The dashboard reads this table for the drain health charts and delivery errors.

​
Prometheus metrics

The Prometheus endpoint reports scheduling, delivery, and failure health for the service.