Staging OTLP exports on a stream before we parse them

Our OTLP endpoint, POST /api/v1/otlp/v1/traces, is the path every stock OpenTelemetry exporter uses, and our own SDKs too. Until recently it did all its work inside the request: decompress, parse, translate, then write every trace and span to Postgres before answering. This post covers how we moved the expensive part out, and what we left for later.

The problem with persisting inline

When we counted, a single small export made dozens of sequential round trips to Postgres and Redis before it returned, and a large one made hundreds. Each round trip is cheap on its own. Together, they made response time depend on network latency between services, which we do not fully control, and every slow request held an API worker thread while it waited.

Our REST batch endpoint already had an accept-fast path: validate, put the batch on a Redis Stream, answer, and let a worker persist it. OTLP never got one. This change gives it one.

Accept fast, but still tell the truth

The obvious version of accept-fast is "write the bytes somewhere and say 200." We did not do that. Everything that can reject an export still runs in the request, in the same order as before:

• size limits and a bounded gunzip
• protobuf or JSON parsing, and the check for resourceSpans
• the span-count cap and translation into our internal model
• the usage-limit check

A malformed body still gets a synchronous 400, and a quota problem a 429. Only then do we stage the export, so a 200 never reports as accepted something the worker would later throw away. The response is unchanged; "accepted" now means "validated and queued", as it always has on our REST batch path.

Then the request does three more things:

1. Encrypts the raw request bytes with the same key we use for stored trace payloads. Exports contain prompts and completions, and we do not stage those as plaintext. If encryption is not available, the endpoint falls back to the synchronous path rather than staging unencrypted data.
2. Claim-checks large bodies. An export over 64 KB is written to object storage under an org-scoped key, and the stream entry carries only a reference. If that write fails, the client gets a 503 with Retry-After. We never fall back silently.
3. Appends one entry to a dedicated OTLP stream, separate from the REST batch stream, so each has its own backlog and lag numbers.

We stage the raw bytes, not the parsed result. The worker parses them again. That costs some CPU, but it means the worker runs the same decode, translate and persist code the synchronous path runs, rather than a second implementation that could drift.

What the drain does

A Celery task drains both streams in one run. Each enqueue schedules a drain, and a beat every five minutes acts as a safety net. In outline, one drain run looks like this:

# Illustrative outline, not the literal code.
while within_time_budget():
    entries = xreadgroup("ingest-workers", consumer_name, stream, count=batch_size)
    for entry in entries:
        body = decrypt(entry)                       # or fetch the claim-checked object
        events = translate(parse(gunzip(body)))     # same code as the sync path
        persist(events)                             # same persist call as well
        xack_and_xdel(stream, entry.id)             # only after persist returns

A few details matter more than the outline:

• One consumer group, a distinct name per process. Concurrent drains split entries between them and do not duplicate work.
• A time budget per run. OTLP entries are read in small batches because one entry can hold many spans. A drain that runs out of budget with work left hands off to a fresh task, so the budget limits how long a task runs, not how much gets processed.
• Bounded concurrency. A counting slot set caps how many drains run at once across the cluster, which is effectively our database write budget, however many worker replicas are consuming.
• Acknowledge and delete together. An entry is acked and deleted in one pipeline, and only after its persist and post-persist work have finished. The stream length is then the true backlog.

Failure modes

Most of the design is about what happens when something goes wrong.

• The worker dies mid-entry. The entry stays pending. After a minute idle, another consumer reclaims it. Persisting is idempotent on span id, so a replay does not create duplicates.
• An entry keeps failing. After three delivery attempts it goes to a dead-letter stream with its error and source. Some failures dead-letter at once: a body that cannot be decrypted or parsed, or a reference that points at another org's data.
• The org was deleted while its export sat in the queue. The entry is acked and dropped and its staged object deleted. It is not dead-lettered.
• The backlog grows. The accept path refuses with a 503 and Retry-After once a stream reaches its entry cap or its staged-bytes cap. Refusing is the point: a capped stream that keeps answering 200 eventually trims unprocessed entries. Exporters retry on 503.
• The task hits its time limit mid-query. We found that this could leave a database connection unusable and make later steps fail. The drain now stops cleanly, acks what it finished, leaves the in-flight entry pending for redelivery, and discards a connection left in that state before it is reused.
• Two exports touch the same trace. They can be persisted out of order or at the same time. A row lock around the trace's aggregate updates makes the result the same as if the spans had arrived in one export.

The honest costs: read-after-write is now "a few seconds after the 200" rather than immediate. Our target is an accept-to-persist lag under five seconds at the 95th percentile, and we report lag per stream after every drain. The point of durability also moves from the Postgres commit to Redis, so the durability bar we accepted is Redis with an append-only file synced every second.

What we chose not to build yet

• Object storage first for every request. Writing every export to object storage before acking is how some systems get durability. When we measured, a single write cost about as much as the round trips we were trying to remove.
• A presigned-upload SDK path. It changes the SDK contract, stock exporters cannot use it, and it only pays off for large payloads.
• Worker micro-batching and folding the request-side gates into one Redis script. Both would cut more round trips. Neither is needed yet.
• A raw-export archive for replay. Useful, but it waits for a real need.
• A "processing" badge in the trace list. We would rather hold the lag target.

OTLP metrics and logs endpoints stay synchronous. The new path sits behind a flag that is off by default; with it off, the endpoint behaves exactly as before.