feat: export OTLP traces to a local file by SandyChapman · Pull Request #1042 · NVIDIA/NeMo-Relay

SandyChapman · 2026-09-11T16:59:16Z

Overview

Relay's OpenTelemetry plugin can only ship spans to a collector. ATIF and ATOF both write to disk, so a consumer that treats a trajectory as an artifact rather than as telemetry has no OTLP option at all. This adds [[components.config.opentelemetry.file_sinks]], which writes the same ExportTraceServiceRequest an endpoint would receive to a local file.

The argument for a sink here rather than a collector is that it removes a routable address and an egress rule for isolated sandboxes. Additionally, some users using Fabric+Relay for evaluation, may not have provision the infrastructure required to collect traces.

I confirm this contribution is my own work, or I have the right to submit it under this project's license.
I searched existing issues and open pull requests, and this does not duplicate existing work.

Note for reviewers: a companion change is needed in NeMo-Fabric

This change is necessary but not sufficient for the evaluation use case.

Callers that reach Relay through Fabric use nemo_fabric.RelayOpenTelemetryConfig, which has two fields (enabled, endpoints) and a hand-written validator in its own models.py:

if self.enabled and not self.endpoints:
    raise ValueError("enabled NeMo Relay OpenTelemetry requires at least one endpoint")

Checked against nemo-fabric 0.3.0b1:

Multiple endpoints already work and need nothing from anyone.
A file sink alongside at least one endpoint works with this PR alone: the model sets extra="allow", so file_sinks survives serialization and reaches Relay untouched.
A file sink as the only destination — the configuration an eval actually wants — is rejected by Fabric before Relay sees the config.

Fabric needs file_sinks on the model and the validator relaxed to accept a file sink as a destination, which is the same shape as the validate_opentelemetry_section change in the second commit here.

Details

The destination is a sum type. endpoint, transport, headers, header_env, and timeout live inside TraceDestination::Otlp, so a config holding both an endpoint and a file sink cannot be built:

pub enum TraceDestination {
    Otlp(OtlpEndpointSettings),   // endpoint, transport, headers, header_env, timeout
    File(OtlpFileSinkSettings),   // output_directory, path, format, append
}

Endpoint and header validation moved into that variant, which removed the conditional that gated it on the absence of a file sink. Configuration mirrors the existing ATOF file sink (output_directory, filename, mode) instead of overloading endpoint with a path.

No silent ignore. The builders stay infallible, so an endpoint-only option set on a file sink is recorded and refused at construction: endpoint, headers, transport do not apply to a file sink destination, deduplicated and sorted. Each binding enforces the same rule at its own boundary. Python turns the three endpoint-only attributes into setters that raise; Node rejects them alongside outputDirectory, which it had been applying to file sinks all along, since it passed timeoutMillis and the header map through unconditionally.

Everything above the exporter (projection, id generation, batching, resource attributes, shutdown) is shared, so a span means the same thing whichever destination it goes to. SpanExporter::export returns impl Future, so the trait is not dyn-compatible and dispatch is a concrete enum.

Conversion is upstream. group_spans_by_resource_and_scope and the SpanData → protobuf From impls come from opentelemetry-proto, the crate opentelemetry-otlp already uses to build its wire payload. Both opentelemetry-proto and prost were already in crates/core/Cargo.toml under [dev-dependencies]; they move to [dependencies] with the trace and with-serde features. No new crates and no version changes.

Durability and permissions. Each export is flushed before it is reported as delivered, so a run that exits between batches leaves a readable prefix rather than an empty file. Output is created with the same owner-only permissions and directory confinement as the ATOF and ATIF sinks, because a trajectory carries prompt and response content. Two sinks writing one path are rejected at activation.

Surfaces. nemo-relay configure lists file sinks beside trace endpoints; Python, Node, Go, and the FFI each gain the destination.

Where should the reviewer start?

crates/core/src/observability/otel_file.rs — the exporter is ~200 lines and the whole design is visible there. Then crates/core/tests/unit/observability/otel_file_tests.rs: the round-trips decode with prost and serde_json directly rather than through this module's own encoder, so a writer and reader that agree only with each other cannot pass. json_lines_encode_span_identifiers_as_hex is the OTLP/JSON conformance check.

endpoint_options_on_a_file_sink_are_refused_by_name and several_refused_options_are_reported_together cover the exclusivity rule.

The design decision worth arguing about is file_sinks as a separate array versus a transport = "file" variant on endpoints. I chose the former because endpoint is required and URL-validated, and conditional validation on a field that sometimes holds a path seemed worse than a second array.

Testing

cargo test --workspace — 1660 core lib tests pass. 46 new tests: exporter round-trip and framing, config validation, destination exclusivity, editor schema, and per-binding coverage.
- Pre-existing unrelated failure: native_plugin_integration needs just build-test-plugin-fixtures, which I could not run locally.
cargo clippy --workspace --all-targets and cargo fmt --all clean.
Node: node --test crates/node/tests/observability_plugin_tests.mjs — 10/10. One test drives a file-sink-only section through the plugin host and asserts the trace file appears.
Python: pytest python/tests/test_types.py — 63/63, ruff clean, ty clean (9 pre-existing unused-ignore warnings).
Go: go test on otel_test.go passes; gofmt and go vet clean.
Not run locally: cargo deny (not installed) and the fixture-dependent native plugin integration tests.

Breaking changes

None. file_sinks defaults to empty and every existing configuration behaves identically.

Related Issues:

Relates to #1089

coderabbitai · 2026-09-11T16:59:23Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

_{Comment @coderabbitai help to get the list of available commands.}

github-actions · 2026-09-11T17:01:58Z

License Diff

Compared against origin/main.

Lockfile license changes

Lockfile License Changes

Rust

Added

const-hex 1.19.1 (Apache-2.0)
proptest 1.11.0 (MIT OR Apache-2.0)
rand_xorshift 0.4.0 (MIT OR Apache-2.0)
unarray 0.1.4 (MIT OR Apache-2.0)

Removed

None

Updated/Changed

None

Node

Added

None

Removed

None

Updated/Changed

None

Python

Added

None

Removed

None

Updated/Changed

None

Status output

[license-diff] selected languages: rust, node, python
[license-diff] generating current inventory
[license-diff] current: generating Rust inventory
[license-diff] current: Rust inventory complete (455 packages)
[license-diff] current: generating Node inventory
[license-diff] current: Node inventory complete (424 packages)
[license-diff] current: generating Python inventory
[license-diff] current: Python inventory complete (115 packages)
[license-diff] current inventory complete
[license-diff] checking out base ref origin/main into a temporary worktree
[license-diff] base: generating Rust inventory
[license-diff] base: Rust inventory complete (451 packages)
[license-diff] base: generating Node inventory
[license-diff] base: Node inventory complete (424 packages)
[license-diff] base: generating Python inventory
[license-diff] base: Python inventory complete (115 packages)
[license-diff] base inventory complete
[license-diff] removing temporary base worktree
[license-diff] comparing inventories
[license-diff] rendering Markdown output
[license-diff] done

github-actions · 2026-09-11T17:08:51Z

Fern docs preview: https://nvidia-preview-pull-request-1042.docs.buildwithfern.com/nemo/relay/dev

willkill07

I feel like this needs a proper proposal rather than instructing Claude to implement the functionality loosely based on need.

I understand this is in draft form, but it isn't conducive to any significant review time without a corresponding proposal.

Ultimately, we need a configuration shape for the new functionality without breaking existing plugin configuration files.

Relay's OpenTelemetry plugin could only ship spans to a collector. ATIF and ATOF both write to disk, so a consumer that treats a trajectory as an artifact rather than as telemetry -- an evaluation harness, an offline replay, any environment with no collector to export to -- had no OTLP option at all. `[[components.config.opentelemetry.file_sinks]]` writes the same `ExportTraceServiceRequest` an endpoint would receive. The default `json_lines` format follows the OpenTelemetry Protocol File Exporter specification: one OTLP/JSON record per line. `proto` writes each record length-delimited, matching the Collector file exporter's layout, for consumers that would rather not pay JSON's size and parse cost. A file sink is modelled as a destination rather than as a third `OtlpTransport`: it has no endpoint, headers, or timeout, and endpoint validation does not apply to it. Everything above the exporter -- projection, id generation, batching, resource attributes -- is shared, so the two destination kinds cannot drift on what a span means. `SpanExporter::export` returns `impl Future` and the trait is therefore not dyn-compatible, so dispatch is a concrete enum rather than a boxed trait object. Records are flushed before each export returns, so a run that dies between batches still leaves a readable prefix. Output is created with the same owner-only permissions and directory confinement as the ATOF and ATIF sinks, because a trajectory carries prompt and response content. Signed-off-by: Sandy Chapman <schapman@nvidia.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>

`nemo-relay configure` lists file sinks beside trace endpoints, and the Python, Node, Go, and FFI surfaces each gain the destination so a programmatically built subscriber can write a trace without a collector. The Node suite caught a second validation site: `validate_opentelemetry_section` runs before activation and still required an endpoint, so a file-sink-only section was rejected before `register_opentelemetry` ever saw it. It now accepts a file sink as a destination and reports a malformed one per index, matching how it already reports endpoints. Headers and `header_env` are refused on a Python file sink rather than silently dropped: a caller who sets them has misunderstood the destination. Signed-off-by: Sandy Chapman <schapman@nvidia.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>

Signed-off-by: Sandy Chapman <schapman@nvidia.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>

`opentelemetry-proto`'s `with-serde` feature, which provides the OTLP/JSON serialization the file exporter's default format needs, adds `const-hex` to the graph and its dev dependencies to the lock. Signed-off-by: Sandy Chapman <schapman@nvidia.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>

`OpenTelemetryConfig` carried `endpoint`, `transport`, `headers`, `header_env`, and `timeout` alongside an optional file sink, so a config holding both was representable and the endpoint half was dropped without a word. `nemo-relay plugins edit` can modify a configuration programmatically, so that combination was reachable from a supported surface, and a config that reads as though it exports to a collector while it writes a file gives no signal that anything is wrong. Those five fields move into `TraceDestination::Otlp`, leaving the two destinations mutually exclusive by construction. Endpoint and header validation moves with them, which removes the conditional that gated it on the absence of a file sink. The builders stay infallible, so misuse is recorded and refused at construction rather than silently dropped: `endpoint, headers, transport do not apply to a file sink destination`, deduplicated and sorted. Each binding enforces the same rule at its own boundary. Python turns the three endpoint-only attributes into setters that raise, and Node rejects them alongside `outputDirectory` -- which it had been applying to file sinks all along, since it passed `timeoutMillis` and the header map through unconditionally. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>

The FFI entry point existed but nothing declared or called it from Go, so a Go consumer could configure a file sink through `plugins.toml` and had no way to build one programmatically. `NewOpenTelemetryFileSinkSubscriber` closes that, taking a config with no endpoint, transport, headers, or timeout, since none of them apply to a file. Coverage measures `crates/ffi/src` through the Go suite, so the entry point was also reported as entirely unexercised despite its Rust-side test. Also covers the two reachable error paths in the exporter: a parent path that is a file, so the output directory cannot be created, and a poisoned writer lock, which every entry point reports instead of unwrapping into a second panic. Signed-off-by: Sandy Chapman <schapman@nvidia.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

`TraceExporter` existed to dispatch between the two exporters because `SpanExporter::export` returns `impl Future`, which makes the trait not dyn-compatible. A boxed trait object is still impossible, but the enum was not the only way out: `DiagnosticBatchSpanProcessor` is already generic over its exporter, so the provider construction takes the exporter as a type parameter and each destination hands over its concrete one. Thirty lines of hand-written dispatch go away. The builders matched on `endpoint_settings_mut()`, which turned a two-variant enum back into an `Option` and matched on that. They now match the enum directly, and the accessor has no callers left. The comments claimed more than they were worth. `TraceDestination` explained that an enum is mutually exclusive and then restated its own variants; the exporter module opened with motivation that belongs in a design discussion. What is left is the part a reader cannot recover from the code: why an empty batch writes nothing, why the JSON is compact, why a flush after shutdown succeeds, and why the output is owner-only. Signed-off-by: Sandy Chapman <schapman@nvidia.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

willkill07

Just leaving a round of comments

willkill07 · 2026-09-17T12:28:13Z

+`output_directory`, the same as the ATOF and ATIF file sinks, because a
+trajectory carries prompt and response content.


no need to specify reason.

willkill07 · 2026-09-17T12:28:31Z

+Endpoint validation does not apply: a file sink has no endpoint, transport,
+headers, or request timeout, and configuring those fields on one is an error.


This is obvious simply through the configuration, is it not?

willkill07 · 2026-09-17T12:29:27Z

+A file sink writes the same OTLP `ExportTraceServiceRequest` an endpoint would
+receive to a local file. Use one when a trajectory is an artifact rather than
+telemetry: an evaluation harness that scores the trace it just produced, an
+offline replay, or any environment with no collector to export to. File sinks


I really don't like Claude-isms in documentation. Can this please be rephrased?

willkill07 · 2026-09-17T12:30:45Z

+`json_lines` is the serialization described by the OpenTelemetry Protocol File
+Exporter specification: one OTLP/JSON-encoded `ExportTraceServiceRequest` per
+line, with `traceId` and `spanId` hex-encoded. `proto` writes each request
+length-delimited -- a big-endian `u32` byte count followed by that many bytes of
+encoded request -- matching the OpenTelemetry Collector file exporter's
+`format: proto` layout.


Can we just concisely describe the two output formats (and when to use each) without diving into the details of its output. If it does match the specified OpenTelemetry spec, then we should directly reference it.

willkill07 · 2026-09-17T12:31:43Z

+Two file sinks cannot write the same path: appending sinks would interleave
+their records and overwriting sinks would race, so the configuration is
+rejected at activation.


This should be moved above the table.

willkill07 · 2026-09-17T12:35:04Z

+    /// Set by [`PyOpenTelemetryConfig::file_sink`]; when present the spans are
+    /// written to this file instead of exported to `endpoint`.
+    pub(crate) file_sink: Option<PyOtlpFileSink>,


If we are doing this for the file sink, we should probably hoist out the endpoint-specific options as their own struct to reduce future confusion.

Then it could be an enum instead of two Options

willkill07 · 2026-09-17T12:44:03Z

-    headers: HashMap<String, String>,
-    header_env: HashMap<String, String>,
-    header_file: HeaderFiles,
+    destination: TraceDestination,


For other language bindings we have a clear split between:

OpenTelemetry endpoint

OpenTelemetry file sink

Shouldn't we match that here?

To be clear: the plugin config is good! But the configuration options for a file sink are very distinct from the options for the endpoint, and we currently need a lot of extra logic (like inapplicable_options) gives a code smell.

github-actions Bot added size:XL PR is extra large Feature a new feature lang:go PR changes/introduces Go code lang:js PR changes/introduces Javascript/Typescript code lang:python PR changes/introduces Python code lang:rust PR changes/introduces Rust code labels Sep 11, 2026

copy-pr-bot Bot deployed to fern September 11, 2026 16:59 Active

copy-pr-bot Bot deployed to fern September 11, 2026 17:22 Active

willkill07 assigned willkill07 and unassigned willkill07 Sep 11, 2026

willkill07 requested changes Sep 12, 2026

View reviewed changes

Comment thread crates/core/src/observability/otel.rs Outdated

Comment thread crates/core/src/observability/otel.rs Outdated

Comment thread crates/core/src/observability/otel.rs

SandyChapman mentioned this pull request Sep 16, 2026

[Enhancement]: Export OTLP traces to a local file #1089

Open

copy-pr-bot Bot deployed to fern September 16, 2026 19:52 Active

SandyChapman and others added 5 commits September 17, 2026 08:38

docs: document OpenTelemetry trace file sinks

950b432

Signed-off-by: Sandy Chapman <schapman@nvidia.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Sandy Chapman <schapman@nvidia.com>

SandyChapman force-pushed the feat/otlp-file-exporter branch from dad6459 to a7621e8 Compare September 17, 2026 11:54

copy-pr-bot Bot deployed to fern September 17, 2026 11:54 Active

copy-pr-bot Bot deployed to fern September 17, 2026 12:19 Active

copy-pr-bot Bot deployed to fern September 17, 2026 12:43 Active

willkill07 reviewed Sep 17, 2026

View reviewed changes

		`output_directory`, the same as the ATOF and ATIF file sinks, because a
		trajectory carries prompt and response content.

		Endpoint validation does not apply: a file sink has no endpoint, transport,
		headers, or request timeout, and configuring those fields on one is an error.

Conversation

SandyChapman commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Overview

Note for reviewers: a companion change is needed in NeMo-Fabric

Details

Where should the reviewer start?

Testing

Breaking changes

Related Issues:

Uh oh!

coderabbitai Bot commented Sep 11, 2026

Draft PR not reviewed

Uh oh!

github-actions Bot commented Sep 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

License Diff

Lockfile License Changes

Rust

Added

Removed

Updated/Changed

Node

Added

Removed

Updated/Changed

Python

Added

Removed

Updated/Changed

Uh oh!

github-actions Bot commented Sep 11, 2026

Uh oh!

willkill07 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

willkill07 left a comment

Choose a reason for hiding this comment

Uh oh!

willkill07 Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

willkill07 Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

willkill07 Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

willkill07 Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

willkill07 Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

willkill07 Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

willkill07 Sep 17, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

SandyChapman commented Sep 11, 2026 •

edited

Loading

github-actions Bot commented Sep 11, 2026 •

edited

Loading