> ## Documentation Index
> Fetch the complete documentation index at: https://docs.elementum.io/llms.txt
> Use this file to discover all available pages before exploring further.
# Trace Forwarding
> Configure OpenTelemetry (OTEL) endpoints to stream OTLP trace data from Elementum to your observability and AI evaluation platforms.
## Overview
Trace Forwarding lets your organization push OpenTelemetry (OTEL) trace data from Elementum environments to external platforms — observability tools, AI evaluation suites, or any system that accepts the OpenTelemetry Protocol (OTLP) over gRPC or HTTP. You can configure multiple endpoints per channel to route traces to different teams or platforms in parallel.
**OTEL vs. OTLP.** OpenTelemetry (OTEL) is the open standard for generating and collecting trace, metric, and log data. The OpenTelemetry Protocol (OTLP) is the wire format OTEL uses to transmit that data over gRPC or HTTP. Elementum produces OTEL trace data and exports it using OTLP, so your endpoints must accept OTLP.
Go to 
**Organization Settings** > **Platform** > **Trace Forwarding**. Only **Organization Administrators** can view and modify these settings.
## Trace channels
Two independent channels are available:
| Channel | What it forwards | Default protocol | Example destinations |
| ------------------ | ------------------------------------------------------- | ---------------- | ----------------------------------- |
| **General Traces** | Operational telemetry from all Elementum activities | gRPC | Datadog, Grafana, Jaeger, Honeycomb |
| **GenAI Traces** | Detailed LLM and agent spans from AI-powered operations | HTTP | Weave (W\&B), LangSmith, Arize |
## Quick start: add your first endpoint
1. Go to **Organization Settings** > **Platform** > **Trace Forwarding**.
2. Click **Add General Endpoint** or **Add GenAI Endpoint** depending on the type of traces you want to forward.
3. Enter a **Name**, your **Endpoint URL**, and confirm the **Protocol** matches what your destination platform expects.
4. Select an **Authorization Type** and fill in the required credentials.
5. Optionally, add **Resource Attributes** to tag every span with metadata such as `environment` or `team`.
6. Click **Save & Test**. A green *Message delivered* result confirms the endpoint is reachable.
7. Click **Enable** to activate the endpoint.
8. Go to **Organization Settings** > **Platform** > **Environments**, click **Configure** on an environment card, then click **Edit Environment**, select your endpoint under **Trace Forwarding**, and click **Save Changes**.
## Add or edit an endpoint
Click **Add General Endpoint** or **Add GenAI Endpoint** to open the endpoint dialog. To edit an existing endpoint, click the pencil icon in its row. Both actions open the same dialog.
### Connection
| Field | Required | Details |
| ---------------- | -------- | --------------------------------------------------------------------------------------------------------------------- |
| **Name** | Yes | A human-readable label used to identify the endpoint in the list. |
| **Endpoint URL** | Yes | The full URL of your OTLP receiver. Must be a valid HTTP or HTTPS URL. |
| **Protocol** | Yes | **gRPC** (binary, efficient) or **HTTP** (REST-based). Defaults to gRPC for General Traces and HTTP for GenAI Traces. |
### Authentication
Select one of four options from the **Authorization Type** dropdown:
| Type | Behavior |
| ------------------ | --------------------------------------------------------------------------------------------------------------------------------------- |
| **None** | No authentication headers are sent. Use only for internal or open endpoints. |
| **Bearer Token** | Sends an `Authorization: ` header. Requires a **Bearer Token**; **Bearer Scheme** is optional and defaults to `Bearer`. |
| **Basic Auth** | Sends an `Authorization: Basic ` header. Requires **Username** and **Password**. |
| **Custom Headers** | Sends one or more arbitrary request headers. Add as many **Header Key** / **Header Value** pairs as needed. |
Secret values (tokens, passwords, header values) are encrypted at rest. When you reopen an endpoint for editing, existing secrets are masked as `****`. Click **Overwrite** to replace a secret, or leave it masked to keep the current value.
### Optional fields
* **Additional headers** — Extra HTTP headers attached to every trace export request beyond those required for authentication. Click **+ Add Header** to add key-value pairs.
* **Resource attributes** — Key-value pairs added to the OTLP resource on every exported span. Use these to tag trace data with environment, team, or deployment metadata that your external platform can filter on (e.g. `environment = production`, `team = ai-platform`).
Click **Save & Test** to save the endpoint and verify connectivity immediately. General Trace endpoints also offer a **Save** option that skips the connection test. GenAI endpoints require **Save & Test** — connectivity must be verified before the endpoint is saved.
## Manage endpoints
### Test a connection
A connection test sends a sample OTLP trace to the configured endpoint. You can run a test two ways:
* Click **Save & Test** when creating or editing an endpoint.
* Click the **send icon** (Test Connection) in the endpoint list row.
The test times out after 30 seconds. A result modal shows latency, HTTP status code, trace ID, and span ID on success, or an error message and guidance on failure. From the modal you can **Edit Connection**, toggle **Enable** / **Disable**, or **Cancel** to close.
### Assign to an environment
After configuring and enabling an endpoint, assign it to one or more environments. Traces are only forwarded from environments where an endpoint is assigned.
1. Go to **Organization Settings** > **Platform** > **Environments**.
2. Find the environment you want to forward traces from and click **Configure** on its card, then click **Edit Environment**.
3. In the **Trace Forwarding** section, select your configured endpoint from the list.
4. Click **Save Changes**.
Repeat for each environment you want to forward traces from.
### Delete an endpoint
1. Click the **trash icon** in the endpoint list row.
2. Confirm the deletion in the dialog that appears.
## Continuing an existing trace
Include a W3C `traceparent` header on a request to signal trace propagation for the traces Elementum exports. The exported spans are tied back to the information you provide, placed under the same trace using the span ID you supply as their parent, so in your platform the request appears as a single connected trace with Elementum's spans nested under your calling span. This requires a Trace Forwarding endpoint that is enabled and [assigned to the environment](#assign-to-an-environment); without one, there is nowhere to export the spans. If no `traceparent` is sent, the exported spans are not tied to an external trace.
### Add the header
Set the `traceparent` header on the request your system sends to Elementum. If your calling application is instrumented with OpenTelemetry, its HTTP client adds this header automatically. Otherwise, add it manually, populated from the calling span in your tracing system.
```
traceparent: 00---01
```
| Field | Length | Value |
| ---------------- | ------ | ------------------------------------------------------------ |
| `trace-id` | 32 hex | The overall trace ID. Elementum's spans adopt this. |
| `parent-span-id` | 16 hex | The span in your system that Elementum's spans attach under. |
The leading `00` (version) and trailing `01` (flags) are fixed parts of the W3C format; you normally leave them as shown, and if your caller uses OpenTelemetry they are set automatically.
Example: `00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01`
You may also send a `tracestate` header; Elementum preserves it.
### Verify
Open the trace by the `trace-id` you sent. You should see your originating span with Elementum's spans nested beneath it as one connected trace.
Propagation is opt-in per request. Without a `traceparent`, exported spans are not tied to an external trace.
## Troubleshooting
The credentials are invalid or expired. Re-enter the token or password and ensure the API key has the correct scopes.
The endpoint URL is likely incorrect. Double-check the URL and, for HTTP endpoints, confirm the path includes `/v1/traces` or the equivalent path required by your platform.
A network or firewall issue is preventing the connection. Verify the endpoint host is reachable from the Elementum backend and check any applicable firewall rules.
Check that the endpoint is **Enabled** and assigned to the environment you are testing from. A passing test only confirms connectivity, not that traces are being produced — trigger an AI operation in Elementum to generate spans.
## GenAI attribute reference
### LLM call spans
These attributes appear on spans representing a single call to a language model. Span names follow the pattern `chat `.
| Attribute | Type | Description |
| -------------------------------- | ------------- | ------------------------------------------------------------- |
| `gen_ai.system` | string | The LLM provider (e.g. `aws.bedrock`, `openai`, `anthropic`). |
| `gen_ai.request.model` | string | The model name that was requested. |
| `gen_ai.response.model` | string | The model name that actually responded. |
| `gen_ai.usage.input_tokens` | integer | Number of tokens in the prompt. |
| `gen_ai.usage.output_tokens` | integer | Number of tokens in the completion. |
| `gen_ai.prompt` | string (JSON) | Input messages as a JSON array of `{role, content}` objects. |
| `gen_ai.completion` | string (JSON) | Output messages as a JSON array of `{role, content}` objects. |
| `gen_ai.operation.name` | string | Always `"chat"` for LLM call spans. |
| `gen_ai.response.finish_reasons` | string\[] | Why the model stopped generating (e.g. `["stop"]`). |
| `gen_ai.response.id` | string | The provider's unique response identifier. |
### Tool execution spans
These attributes appear on spans representing an agent executing a tool. Span names follow the pattern `execute_tool `.
| Attribute | Type | Description |
| ----------------------- | ------ | --------------------------------------------------------- |
| `gen_ai.tool.name` | string | The function name of the tool that was called. |
| `gen_ai.tool.call.id` | string | Links this execution back to the LLM's tool call request. |
| `gen_ai.operation.name` | string | Always `"execute_tool"` for tool execution spans. |
### Span tree structure
Traces from Elementum AI operations are organized into a span hierarchy:
| Interaction type | Span structure |
| ---------------------------------- | --------------------------------------------- |
| Single LLM call | `root → chat` |
| Tool-calling agent (one round) | `root → chat → tool → chat` |
| Multi-step agent (multiple rounds) | `root → chat → tool → chat → tool → chat → …` |