Skip to content

Add tracing for colibri requests - #633

Open
JulianGCalderon wants to merge 6 commits into
jitsi:masterfrom
JulianGCalderon:tracing-jibri
Open

Add tracing for colibri requests#633
JulianGCalderon wants to merge 6 commits into
jitsi:masterfrom
JulianGCalderon:tracing-jibri

Conversation

@JulianGCalderon

@JulianGCalderon JulianGCalderon commented Aug 12, 2026

Copy link
Copy Markdown

This PR adds tracing for the handling of colibri requests.

Depends on jitsi/jicofo#1301.

To test this, the following configuration is required

tracing {
    enabled = true
    otlp-protocol = "grpc"
    otlp-endpoint = "http://localhost:4317"
}

Related to the tracing backend GSoC project: https://summerofcode.withgoogle.com/programs/2026/projects/GseggmSv

@jitsi-jenkins

Copy link
Copy Markdown

Hi, thanks for your contribution!
If you haven't already done so, could you please make sure you sign our CLA (https://jitsi.org/icla for individuals and https://jitsi.org/ccla for corporations)? We would unfortunately be unable to merge your patch unless we have that piece :(.

WIP: the context is not being read correctly for some reason

@aaronkvanmeerten aaronkvanmeerten left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this — the overall approach (remote parent from the IQ extension, mirroring the jicofo side) looks right to me. A few observations beyond the inline comments:

The WIP "context is not being read correctly" issue is most likely already fixed. JibriIqProvider used to drop all child extensions when parsing, so the TraceParent never made it onto the parsed JibriIq. That was fixed in jitsi/jitsi-xmpp-extensions#147 ("Fix Jibri IQ extension parsing"), and the version this PR pins (1.0-118-g64ecd07) is exactly that commit — the WIP commit predates the bump. Worth an end-to-end retest and squashing the WIP note out of the history.

Spans will be silently dropped at process exit — worst in single-use mode. TracingGlobal uses a BatchSpanProcessor (default 5s export delay) and never flushes; jibri exits via exitProcess() in cleanupAndExit (Main.kt). With jibri.single-use-mode = true the process exits right after the session ends — exactly when the most valuable spans were just created, so the jibri.stop span (and any session spans) will routinely be lost. Ideally jicoco-tracing would expose a shutdown()/forceFlush() (the sdk field is typed as the OpenTelemetry interface, so callers can't reach the SdkTracerProvider without a cast today); short of that, a JVM shutdown hook in Main that casts to OpenTelemetrySdk and closes it works.

On the jicofo side (jitsi/jicofo#1301): in the pending-timeout stop path, the TraceParent is added to startIq instead of stopRequest (copy-paste). Also both the timeout-stop and retry paths run on timer threads where Context.current() is root, so those stops will never carry a trace context — only user-initiated stops handled inside the jibri.request span will. Worth flagging over there.

Design note on the wire format (non-blocking): the extension carries trace_id/parent_id/trace_flags as separate attributes and drops tracestate entirely (TraceState.getDefault() on the receive side). Carrying the full W3C traceparent string (plus optional tracestate) in the extension and reusing W3CTraceContextPropagator with a trivial TextMapGetter over the IQ would eliminate the hand-rolled parse/build on both ends and stay spec-aligned. Might be worth settling before the format ossifies across jicofo/jibri/jvb. Relatedly, TracingGlobal builds the SDK with no propagators configured, which any future HTTP-header-based propagation (e.g. the ktor instrumentation on jibri's HTTP API) would silently trip over — adding W3CTraceContextPropagator there would future-proof it.

I'm putting together a follow-up branch with prototypes for the suggestions here (session-lifetime span with state-machine events, context propagation into the async service start, flush-on-exit, OTLP config defaults, and traceparent extraction on the HTTP API) and will link it from this PR.

Comment thread src/main/kotlin/org/jitsi/jibri/api/xmpp/XmppApi.kt
Comment thread src/main/kotlin/org/jitsi/jibri/api/xmpp/XmppApi.kt
Comment thread src/main/kotlin/org/jitsi/jibri/api/xmpp/XmppApi.kt
Comment thread pom.xml Outdated
Comment thread src/main/resources/reference.conf
@aaronkvanmeerten

Copy link
Copy Markdown
Member

As promised in my review, I've pushed a branch with prototypes for the suggestions: tracing-jibri...tracing-jibri-prototypes

It's based on this PR's head, with one commit per suggestion so they can be cherry-picked individually:

  • chore: Align OpenTelemetry artifacts with jicoco-tracing via the BOM — pins the OTel version once via opentelemetry-bom instead of a hardcoded version on the sdk dependency.
  • ref: Extract a withSpan tracing helper, enrich the XMPP IQ spans — new util/Tracing.kt with a Tracer.withSpan helper (makeCurrent + recordException + error status + guaranteed end); both IQ handlers use it with SpanKind.SERVER, and the stop span gains session.id.
  • feat: Add a session-lifetime span — a jibri.session span in JibriManager that lives from the start request until the service reaches a terminal state, records state transitions as span events, and is explicitly carried across the TaskPools.ioPool boundary with Context.wrap() so future child spans (selenium join, ffmpeg) can attach. The blocking service stop is also wrapped so stop traces show teardown/finalize duration.
  • fix: Flush pending trace spans on exit — shutdown hook closing the OpenTelemetrySdk; without it single-use-mode exits drop the end-of-session spans. (Longer-term this probably belongs on TracingGlobal itself as a shutdown(), which would also let jibri drop the direct sdk dependency.)
  • feat: Add OTLP defaults so tracing can be enabled with one settingotlp-protocol/otlp-endpoint defaults in reference.conf.
  • feat: Trace HTTP API start/stop requests — extracts a remote parent from the W3C traceparent header, mirroring the XMPP path (done manually rather than via the ktor OTel instrumentation since TracingGlobal currently configures no propagators).

Compile, ktlint, and the full test suite pass on the branch. Feel free to cherry-pick whatever's useful into this PR.

@JulianGCalderon
JulianGCalderon marked this pull request as ready for review August 18, 2026 09:15
@bgrozev

bgrozev commented Aug 18, 2026

Copy link
Copy Markdown
Member

jenkins, add to whitelist

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants