Introduce load-balanced channel for OpenTelemetry exporters by dkostyrev · Pull Request #2175 · TraceMachina/nativelink

dkostyrev · 2026-02-23T14:54:41Z

Summary

This PR introduces client-side load balancing for OpenTelemetry (OTLP) gRPC connections using the ginepro library. When the NL_OTEL_ENDPOINT (name to be discussed, maybe boolean flag?) environment variable is set, NativeLink will create a load-balanced channel for exporting logs, traces, and metrics, distributing requests across multiple backend endpoints resolved via DNS. This change allows to distribute OTLP traffic across multiple OTLP collector instances.

Changes

Load-balanced OTLP exports:

Added ginepro dependency to provide client-side load balancing for gRPC channels used by OpenTelemetry exporters
Introduced NL_OTEL_ENDPOINT environment variable to configure the OTLP endpoint for load-balanced connections
Changed init_tracing() from synchronous to async to support balanced channel initialization
Updated OpenTelemetry dependencies from v0.29 to v0.30 to support the new channel configuration
All three exporters (logs, traces, metrics) now share the same load-balanced channel when configured

This change is

MarcusSorealheis · 2026-03-05T15:28:39Z

@dkostyrev Than you for this awesome PR. I have finally had a bit of time to explore it and now I understand quite a bit.

This is fantastic.

MarcusSorealheis · 2026-03-05T15:29:06Z

@amankrx when you get a chance, please help with the merge conflict here.

MarcusSorealheis · 2026-03-05T18:44:24Z

        .block_on(async {
            // The OTLP exporters need to run in a Tokio context.
-            spawn!("init tracing", async { init_tracing() })
+            spawn!("init tracing", async { init_tracing().await })


oof! Good catch.

palfrey · 2026-03-06T12:31:37Z

That coverage issue is interesting. mold: error: undefined symbol: __memcpy_chk is the sort of problem we used to be having before the fixes in #2192 that removes the hardeningDisable bit that was a workaround to this before. Not quite sure why this is hitting here now, but might need to re-add the workaround.

vercel · 2026-05-30T13:37:21Z

@dkostyrev is attempting to deploy a commit to the native-link-web-assets Team on Vercel.

A member of the Team first needs to authorize it.

vercel · 2026-05-30T16:38:16Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
nativelink	Ready	Preview, Comment	May 30, 2026 4:40pm
nativelink-aidm	Ready	Preview, Comment	May 30, 2026 4:40pm

Add client-side load balancing to OTLP gRPC connections using ginepro. When NL_OTEL_ENDPOINT is set, the telemetry system creates a load-balanced channel shared across log, trace, and metric exporters. This enables better distribution of telemetry traffic across multiple OTLP collector instances and improves overall system resilience. - Add ginepro dependency for gRPC load balancing - Upgrade OpenTelemetry dependencies from 0.29 to 0.30 - Change init_tracing() to async to support channel initialization - Add NL_OTEL_ENDPOINT environment variable for configuration - Update all OTLP exporters to use shared load-balanced channel # Conflicts: # Cargo.lock

palfrey · 2026-06-04T10:59:36Z

+    // so concurrent writes would be a data race on Unix.  The guard is always
+    // scoped so it drops before any `.await`, which satisfies
+    // `clippy::await_holding_lock`.
+    static ENV_LOCK: Mutex<()> = Mutex::new(());


We're already using serial_test and using that instead would get the same effect. Only need to tag as #[serial] the env-editing test methods (or possibly #[serial(env)] just to mark them as a unified group).

palfrey · 2026-06-04T11:00:19Z

+        );
+    }
+
+    #[ignore = "requires DNS resolution (/etc/resolv.conf); not available in sandboxed builds"]


Shouldn't this get run in some cases? e.g. direct cargo runs?

palfrey · 2026-06-04T11:01:24Z

+        reason = "`tokio::test` uses `tokio::runtime::Builder::new_multi_thread` and \
+                  `tokio::runtime::Runtime::block_on` internally"
+    )]
+    #[tokio::test(flavor = "multi_thread", worker_threads = 2)]


Please use nativelink_test. It'll accept arbitrary arguments to tokio::test internally

palfrey · 2026-06-04T11:02:47Z

 }
+
+#[nativelink_test]
+async fn metrics_are_tracked() -> Result<(), Box<dyn core::error::Error>> {


This is an excellent start. Can you run it both with and without NL_OTEL_ENDPOINT so we can check both do the same thing?

palfrey · 2026-06-04T11:03:20Z

This is all now fixed in main

dkostyrev temporarily deployed to production February 23, 2026 14:54 — with GitHub Actions Inactive

dkostyrev force-pushed the feature/balanced-channel branch from eb9bfe9 to b86f6d1 Compare February 23, 2026 16:21

dkostyrev temporarily deployed to production February 23, 2026 16:21 — with GitHub Actions Inactive

dkostyrev force-pushed the feature/balanced-channel branch from b86f6d1 to 9533feb Compare February 23, 2026 16:27

dkostyrev temporarily deployed to production February 23, 2026 16:27 — with GitHub Actions Inactive

dkostyrev force-pushed the feature/balanced-channel branch from 9533feb to 6406230 Compare February 23, 2026 16:35

dkostyrev temporarily deployed to production February 23, 2026 16:35 — with GitHub Actions Inactive

MarcusSorealheis requested a review from palfrey March 5, 2026 15:28

dkostyrev temporarily deployed to production March 5, 2026 18:14 — with GitHub Actions Inactive

amankrx approved these changes Mar 5, 2026

View reviewed changes

MarcusSorealheis approved these changes Mar 5, 2026

View reviewed changes

dkostyrev force-pushed the feature/balanced-channel branch from 29cb3d7 to dff32cd Compare March 6, 2026 12:34

dkostyrev temporarily deployed to production March 6, 2026 12:34 — with GitHub Actions Inactive

dkostyrev temporarily deployed to production March 9, 2026 12:46 — with GitHub Actions Inactive

dkostyrev force-pushed the feature/balanced-channel branch from 31d41d4 to 892ecee Compare March 16, 2026 13:48

dkostyrev temporarily deployed to production March 16, 2026 13:48 — with GitHub Actions Inactive

dkostyrev force-pushed the feature/balanced-channel branch from 892ecee to 1eb7f46 Compare March 16, 2026 13:59

dkostyrev temporarily deployed to production March 16, 2026 13:59 — with GitHub Actions Inactive

dkostyrev force-pushed the feature/balanced-channel branch from d6a38f4 to 9e6ddec Compare May 14, 2026 14:34

dkostyrev temporarily deployed to production May 14, 2026 14:34 — with GitHub Actions Inactive

dkostyrev temporarily deployed to production May 14, 2026 20:05 — with GitHub Actions Inactive

dkostyrev force-pushed the feature/balanced-channel branch from 7a99137 to 300e6c5 Compare May 18, 2026 15:57

dkostyrev temporarily deployed to production May 18, 2026 15:57 — with GitHub Actions Inactive

dkostyrev force-pushed the feature/balanced-channel branch from 5927abc to 7005eab Compare May 19, 2026 14:53

dkostyrev temporarily deployed to production May 19, 2026 14:53 — with GitHub Actions Inactive

dkostyrev had a problem deploying to production May 19, 2026 14:53 — with GitHub Actions Failure

dkostyrev temporarily deployed to production May 19, 2026 15:23 — with GitHub Actions Inactive

dkostyrev force-pushed the feature/balanced-channel branch from 8b3d6c2 to 6f77c5a Compare May 19, 2026 15:31

dkostyrev temporarily deployed to production May 19, 2026 15:31 — with GitHub Actions Inactive

dkostyrev force-pushed the feature/balanced-channel branch from 6f77c5a to 35001bc Compare May 30, 2026 13:37

dkostyrev force-pushed the feature/balanced-channel branch 2 times, most recently from f4fc7d3 to 89f99fa Compare May 30, 2026 16:37

vercel Bot deployed to Preview – nativelink May 30, 2026 16:40 View deployment

vercel Bot deployed to Preview – nativelink-aidm May 30, 2026 16:40 View deployment

dkostyrev force-pushed the feature/balanced-channel branch 3 times, most recently from 0bd5066 to e9a95c0 Compare May 31, 2026 07:44

dkostyrev force-pushed the feature/balanced-channel branch from e9a95c0 to 6e02626 Compare May 31, 2026 07:58

dkostyrev requested a review from palfrey May 31, 2026 16:17

palfrey requested changes Jun 4, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Introduce load-balanced channel for OpenTelemetry exporters#2175

Introduce load-balanced channel for OpenTelemetry exporters#2175
dkostyrev wants to merge 1 commit into
TraceMachina:mainfrom
joomcode:feature/balanced-channel

dkostyrev commented Feb 23, 2026 •

edited by MarcusSorealheis

Loading

Uh oh!

MarcusSorealheis commented Mar 5, 2026

Uh oh!

MarcusSorealheis commented Mar 5, 2026

Uh oh!

MarcusSorealheis Mar 5, 2026

Uh oh!

palfrey commented Mar 6, 2026

Uh oh!

vercel Bot commented May 30, 2026

Uh oh!

vercel Bot commented May 30, 2026 •

edited

Loading

Uh oh!

palfrey Jun 4, 2026

Uh oh!

palfrey Jun 4, 2026

Uh oh!

palfrey Jun 4, 2026

Uh oh!

palfrey Jun 4, 2026

Uh oh!

palfrey Jun 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Conversation

dkostyrev commented Feb 23, 2026 • edited by MarcusSorealheis Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Uh oh!

MarcusSorealheis commented Mar 5, 2026

Uh oh!

MarcusSorealheis commented Mar 5, 2026

Uh oh!

MarcusSorealheis Mar 5, 2026

Choose a reason for hiding this comment

Uh oh!

palfrey commented Mar 6, 2026

Uh oh!

vercel Bot commented May 30, 2026

Uh oh!

vercel Bot commented May 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

palfrey Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

palfrey Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

palfrey Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

palfrey Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

palfrey Jun 4, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

dkostyrev commented Feb 23, 2026 •

edited by MarcusSorealheis

Loading

vercel Bot commented May 30, 2026 •

edited

Loading