Typed spans from day one

The first rule in our architecture notes is four words long: typed spans from day one. This post covers what it means in practice, how it fits with accepting telemetry over more than one protocol, and what it costs us.

What "typed" means

Every span Morse persists has a type drawn from a fixed set. The original six were:

• agent for an agent invocation or step
• llm for a model call
• tool for a tool execution
• http for an outbound or inbound HTTP call
• db for a database query
• custom for everything else

The set has grown since, always by adding: memory, retriever, guardrail, handoff and queue exist today. It is a closed enum in the data model, not a free-text tag. A span that arrives on our REST ingest path with a type outside the enum is rejected with a validation error that lists the valid values. We would rather fail loudly at the door than store something nothing downstream knows how to read.

Type also decides which fields mean anything. Model name, input and output tokens and cost_usd belong to LLM spans and are null elsewhere. Input and output payloads are type-dependent: a prompt and completion for an LLM span, arguments and a result for a tool span.

One model, more than one way in

Teams arrive with very different instrumentation. Some install our SDK. Many already emit OpenTelemetry and want to point an exporter at us without adding a vendor library. So ingestion is protocol-agnostic: each wire format has an adapter, and every adapter translates into the same internal event format before anything is written.

There are two entry points today:

• OTLP/HTTP at POST /api/v1/otlp/v1/traces, accepting JSON or protobuf from any stock OpenTelemetry exporter.
• A REST batch endpoint at POST /api/v1/traces/batch that takes our own JSON shape.

Both adapters hand their output to one persist function. Nothing below that boundary knows which protocol a span came in on.

The REST path is easy to type, because the caller states the type. OTLP is harder: a span carries a kind (server, client, internal and so on) and a bag of attributes, and we have to infer the type. The adapter checks signals in priority order:

1. An explicit morse.span_type attribute, if it names a valid type, always wins.
2. gen_ai.operation.name: invoke_agent and create_agent become agent, execute_tool becomes tool, and chat, generate_content and text_completion become llm.
3. Any other gen_ai.* attribute falls back to llm.
4. db.system becomes db, messaging.system becomes queue, and HTTP or RPC attributes become http. Both the older and the current semantic-convention names count.
5. Only then does the span kind decide, and anything unrecognized is custom.

If your spans already follow the OpenTelemetry GenAI conventions, you get typed spans without changing your code. If you want to be explicit, set the attribute yourself:

from opentelemetry import trace
 
tracer = trace.get_tracer("support-agent")
 
with tracer.start_as_current_span("lookup_balance") as span:
    span.set_attribute("morse.span_type", "tool")
    result = lookup_balance(customer_id)

The day typing quietly broke

This part is not hypothetical. When we first ran an agent framework that emits OTLP natively into our dev environment, every persisted span came back as llm, including the agent invocation and the tool calls. The cause was one line: "any gen_ai.* attribute means an LLM call." That was true when it was written. Current GenAI conventions put gen_ai.* attributes on agent and tool spans too, so the rule swallowed all three types.

Our test suite was green the whole time, because our OTLP fixtures were written by hand and said what we already believed. The fix was the gen_ai.operation.name lookup in step 2 above, placed after the explicit override and before the old blanket rule, so an emitter that sends no operation name behaves exactly as before. The tests for it check the persisted row rather than the adapter's return value. We now also capture real exporter output and replay it, because a hand-written fixture cannot tell you the wire disagrees with it.

The lesson: conventions move, and a type that collapses raises no error. It just makes every chart built on it wrong.

What typing buys downstream

Cost per model call. Because model, tokens and cost live on llm spans, cost is worked out per call from the model and token counts and then summed into the trace. A span we cannot price stays null and is reported as unpriced rather than shown as a silent zero. Per-agent and per-run spend roll up from there, and a trace's cost can be split by layer: model calls, tools, agent orchestration and infrastructure.

Tool failures you can read. A failed tool span carries its error, so "tool lookup_balance failed with a timeout" is a query, not a log search. The same goes for seeing which tools an agent was given against which ones it actually called in a run.

Infrastructure inside the agent trace. HTTP, database and queue spans that share the agent's trace id appear inside that trace, next to the model and tool calls they served. That is the whole scope of infrastructure in Morse: trace-anchored correlation, not a separate metrics or logs product. Typing is what makes it work. We count a trace as correlated when it has at least one agent, llm, tool or memory span and at least one http or db span. When infrastructure spans arrive with no AI spans beside them, we report a propagation gap and do not leave you looking at an empty tab. Infra spans also never change a trace's status on their own, and an org can turn infra correlation off.

The trade-offs

Typing at ingest is not free.

• Inference can be wrong. OTLP typing is a set of rules over attributes that other people define and keep revising. We keep adding mappings as frameworks drift, and the explicit override exists for the cases the rules miss.
• The enum is a contract. Adding a type means touching the model, the adapters and everything that branches on type. We once added a type to the model and forgot a hand-maintained copy of the list in the OTLP adapter, so SDK-declared spans of that type were stored as custom without any error. That list is now derived from the model.
• custom is a release valve, not a fix. A span stored as custom is kept, but it contributes little to cost or failure analysis. When we see a lot of them from a framework, we treat it as a mapping gap to close.

We still think it is the right call. Inferring the type at read time, on every query, would cost more, and the result would still be a guess.