feat(profiling): Add Perfetto trace format support by markushi · Pull Request #5659 · getsentry/relay

markushi · 2026-02-25T08:17:21Z

This PR adds the ability to ingest binary Perfetto traces (.pftrace) and convert them into the existing Sample v2 JSON format used by Sentry's profiling pipeline. This targets Android
profiling, where Perfetto is the native tracing format.

Key Changes

New feature flag organizations:continuous-profiling-perfetto — gates all Perfetto processing.
Compound item format via meta_length — profile chunk items can now carry [JSON metadata][binary blob], split at meta_length. This is how Perfetto traces arrive: metadata describes the
profile, the binary blob is the raw .pftrace.
Perfetto → Sample v2 conversion — new relay-profiling/src/perfetto/ module (proto definitions + conversion logic) parses binary Perfetto traces and produces the existing Sample v2
JSON format.
Raw profile passthrough to Kafka — after expansion, the original Perfetto binary is preserved alongside the expanded JSON. Two new fields on ProfileChunkKafkaMessage: raw_profile and
raw_profile_content_type.
New fields on existing structs:
- Frame.package — library/container path for native/Java frames
- ProfileMetadata.content_type — carries "perfetto" through the pipeline
- DebugImage::native_image() constructor — creates debug images from Perfetto mapping data

This PR is part of a stack:

feat(profiling): Add Perfetto trace format support (this PR)
feat(profiling): Add pipeline workflow to perfetto profiling #5932
feat(profiling): Route raw profiles through objectstore service #6025

Add support for ingesting binary Perfetto traces (.pftrace) as profile chunks. The SDK sends an envelope with a ProfileChunk metadata item paired with a ProfileChunkData item containing the raw Perfetto protobuf. Relay decodes the Perfetto trace, extracts CPU profiling samples (PerfSample and StreamingProfilePacket), converts them to the internal Sample v2 format, and forwards both the expanded JSON and the original binary blob to Kafka for downstream processing. Key changes: - New `perfetto` module in relay-profiling for protobuf decoding and conversion to Sample v2 - New `ProfileChunkData` envelope item type for binary profile payloads - Pairing logic to associate ProfileChunk metadata with ProfileChunkData - Raw profile blob preserved through to Kafka for further processing Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

…tto-profiling-support

# Conflicts: # relay-server/src/envelope/item.rs

markushi · 2026-03-19T07:51:06Z

@sentry review

…fer main thread Separate the shared `strings` intern table into distinct `function_names`, `mapping_paths`, and `build_ids` tables matching the Perfetto spec where each InternedData field has its own ID namespace. Also infer the main thread name when tid equals pid and no explicit name is provided. Co-Authored-By: Claude <noreply@anthropic.com>

cursor

Cursor Bugbot has reviewed your changes and found 1 potential issue.

Autofix Details

Bugbot Autofix prepared a fix for the issue found in the latest run.

✅ Fixed: First timestamp delta skipped in StreamingProfilePacket processing
- Removed the 'i > 0' guard so that timestamp_delta_us[0] is now correctly applied to the first sample's timestamp.

Or push these changes by commenting:

@cursor push ab5f3ec796

Preview (ab5f3ec796)

diff --git a/relay-profiling/src/perfetto/mod.rs b/relay-profiling/src/perfetto/mod.rs
--- a/relay-profiling/src/perfetto/mod.rs
+++ b/relay-profiling/src/perfetto/mod.rs
@@ -229,9 +229,7 @@
             Some(Data::StreamingProfilePacket(spp)) => {
                 let mut ts = packet.timestamp.unwrap_or(0);
                 for (i, &cs_iid) in spp.callstack_iid.iter().enumerate() {
-                    if i > 0
-                        && let Some(&delta) = spp.timestamp_delta_us.get(i)
-                    {
+                    if let Some(&delta) = spp.timestamp_delta_us.get(i) {
                         // `delta` is i64 (can be negative for out-of-order samples).
                         // Casting to u64 wraps negative values, which is correct because
                         // `wrapping_add` of a wrapped negative value subtracts as expected.

_{This Bugbot Autofix run was free. To enable autofix for future PRs, go to the Cursor dashboard.}

Replace `get().is_none()` with `!contains_key()` to satisfy clippy and add a CHANGELOG entry for the Perfetto interning changes. Co-Authored-By: Claude <noreply@anthropic.com>

The first delta in timestamp_delta_us was skipped due to an i > 0 guard, but per the Perfetto spec the first sample's timestamp should be TracePacket.timestamp + timestamp_delta_us[0]. Update tests to use non-zero first deltas to verify the fix. Co-Authored-By: Claude <noreply@anthropic.com>

…kDescriptor StreamingProfilePacket samples were all assigned tid=0 because the code never resolved the trusted_packet_sequence_id → TrackDescriptor → ThreadDescriptor chain. This collapsed multi-thread profiles into a single thread. Now we build a seq_id→tid mapping from TrackDescriptor packets and use it when processing StreamingProfilePacket samples. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

…al state resets The two-pass architecture resolved all samples against the final state of the intern tables. If a trace contained an incremental state reset that reused an interning ID, samples collected before the reset would silently resolve to the wrong function names. This merges the two passes into a single pass that resolves callstacks immediately using the current intern table state, and extracts debug images inline. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

process_compound_item was parsing the entire metadata JSON into a serde_json::Value tree just to read the content_type field, and split_item_payload was doing the same on the expanded JSON (potentially hundreds of KB). Instead, surface content_type from the already- deserialized ProfileMetadata via ExpandedPerfettoChunk, and hardcode "perfetto" in split_item_payload since compound items are always validated as perfetto by process_compound_item. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

The previous commit moved expand_perfetto before the content_type check, which would waste work on non-perfetto payloads and produce confusing errors. Restore the early content_type check using a minimal serde struct that only deserializes the one field, and remove the unused content_type field from ExpandedPerfettoChunk. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Replace the hard-coded "perfetto" content type in split_item_payload with a lightweight deserialization that reads only the content_type field from the metadata JSON. This avoids a silent coupling that would produce incorrect values if compound items support other formats. Also remove a dead item.set_platform() call that wrote to an item immediately replaced by a new one. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Dav1dde

A few small things in the conversion code. I tried to mostly follow along logically but the interning and things are quite complicated according to the docs, so I mostly left like small improvements there.

For the actual processing pipeline, see my larger comment in the processor, we need to get some type safety in here. It will also help with code duplication of the duplicated split functions.
Let me know if you want me to help out with that.

Dav1dde · 2026-04-30T07:10:35Z

+    #[serde(rename = "organizations:continuous-profiling-perfetto")]
+    ContinuousProfilingPerfetto,


I assume the long-term plan is to roll this out to everyone as long as they have continous profiling enabled?

Yes, exactly! The flag is meant for the initial rollout, but should be removed once the feature is GA.

jjbayer · 2026-04-30T09:19:33Z

Fine with me!

jjbayer · 2026-04-30T09:22:40Z

+    /// For native frames this is the dynamic library path (e.g. `libc.so`).
+    /// For Java frames this is the container (e.g. `boot-framework.oat`).
+    #[serde(skip_serializing_if = "Option::is_none")]
+    pub package: Option<String>,


Do we need review from @getsentry/product-owners-profiling for these protocol changes?

I don't think so, as package is already well defined, see https://develop.sentry.dev/sdk/foundations/transport/event-payloads/stacktrace/#frame-attributes

jjbayer · 2026-05-04T12:24:39Z

+    function: Option<String>,
+    module: Option<String>,
+    package: Option<String>,
+    instruction_addr: Option<u64>,


Are all of the combinations of None / Some valid here? For example, a FrameKey with four Nones? If not, an enum might be more suitable.

When looking at the protocol spec all fields are optional, so in theory it could be valid, but yeah it has little to no information when all is blank.

I can imagine to have something like this, but I guess it doesn't really improve the overall situation either.

enum FrameKey { Java { function: Option<String>, module: Option<String>, package: String, // always present (is_java requires mapping_path) }, Native { function: Option<String>, package: Option<String>, instruction_addr: Option<u64>, }, }

cursor

Cursor Bugbot has reviewed your changes and found 2 potential issues.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit ac8dc49. Configure here.}

sentry · 2026-06-03T09:21:41Z

    Jvm,
 }

+/// A debug information image referenced by a profile.


Bug: The deduplication key (code_file, image_addr) is insufficient when a mapping's start address is missing, causing valid debug images to be incorrectly discarded.
_{Severity: MEDIUM}

Suggested Fix

The deduplication key should be made more specific to prevent collisions when the image_addr defaults to 0. Consider including the debug_id or another unique identifier from the mapping in the deduplication tuple to distinguish between different images that might otherwise appear identical.

Prompt for AI Agent

Review the code at the location below. A potential bug has been identified by an AI agent. Verify if this is a real issue. If it is, propose a fix; if not, explain why it's not valid. Location: relay-profiling/src/debug_image.rs#L20 Potential issue: In the `collect_debug_image` function, debug images are deduplicated using a `HashSet` with a key of `(code_file, image_addr)`. The `image_addr` is derived from `mapping.start`, which defaults to `0` if `mapping.start` is `None`. This creates a bug where two genuinely different debug images (e.g., with different `build_id`s) that share the same `code_file` and both lack a `start` address will be treated as duplicates. The first image is processed, but the second is incorrectly discarded, leading to loss of profiling information for that image.

markushi and others added 11 commits February 25, 2026 09:16

style(profiling): Format Perfetto module with cargo fmt

1687276

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

fix(profiling): Remove private intra-doc link in expand_perfetto

2cbf111

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

Merge remote-tracking branch 'origin/master' into feat/markushi/perfe…

9ee2661

…tto-profiling-support

Merge branch 'master' into feat/markushi/perfetto-profiling-support

9d4693c

# Conflicts: # relay-server/src/envelope/item.rs

Merge branch 'master' into feat/markushi/perfetto-profiling-support

4407d9e

Extend profile chunk to support blob attachment via meta_length

e67fa58

Improve code generation script and docs

c4fa16e

Guard perfetto processing by feature flag

1e6df1f

Address some more PR comments, add missing tests

0a58b24

Merge branch 'master' into feat/markushi/perfetto-profiling-support

f729460

sentry Bot reviewed Mar 19, 2026

View reviewed changes

Comment thread relay-profiling/src/perfetto/mod.rs Outdated

markushi and others added 2 commits March 23, 2026 09:01

Merge branch 'master' into feat/markushi/perfetto-profiling-support

685ec0a

markushi marked this pull request as ready for review March 23, 2026 08:57

markushi requested a review from a team as a code owner March 23, 2026 08:57

cursor Bot reviewed Mar 23, 2026

View reviewed changes

Comment thread relay-profiling/src/perfetto/mod.rs

markushi and others added 2 commits March 23, 2026 10:27

fix(profiling): Fix clippy lint and add changelog entry

5c6e457

Replace `get().is_none()` with `!contains_key()` to satisfy clippy and add a CHANGELOG entry for the Perfetto interning changes. Co-Authored-By: Claude <noreply@anthropic.com>

cursor Bot reviewed Mar 27, 2026

View reviewed changes

Comment thread relay-profiling/src/perfetto/mod.rs Outdated

markushi and others added 2 commits April 1, 2026 08:59

Merge branch 'master' into feat/markushi/perfetto-profiling-support

06467d1

sentry Bot reviewed Apr 1, 2026

View reviewed changes

Comment thread relay-profiling/src/perfetto/mod.rs Outdated

cursor Bot reviewed Apr 1, 2026

View reviewed changes

Comment thread relay-server/src/processing/profile_chunks/mod.rs Outdated

markushi and others added 2 commits April 1, 2026 10:36

cursor Bot reviewed Apr 1, 2026

View reviewed changes

Comment thread relay-server/src/processing/profile_chunks/mod.rs Outdated

Comment thread relay-server/src/processing/profile_chunks/process.rs Outdated

sentry Bot reviewed Apr 15, 2026

View reviewed changes

Comment thread relay-profiling/src/perfetto/mod.rs Outdated

cursor Bot reviewed Apr 15, 2026

View reviewed changes

Comment thread relay-profiling/src/perfetto/mod.rs Outdated

Dav1dde assigned markushi Apr 24, 2026

Add e2e tests, remove packets which weren't need for Android profiling

f3f0496

sentry Bot reviewed Apr 28, 2026

View reviewed changes

Comment thread relay-server/src/processing/profile_chunks/process.rs

cursor Bot reviewed Apr 28, 2026

View reviewed changes

Comment thread relay-profiling/src/lib.rs Outdated

Ensure profile meta data is not dropped

e1c0bac

sentry Bot reviewed Apr 28, 2026

View reviewed changes

Comment thread relay-server/src/services/store.rs

Comment thread relay-server/src/processing/profile_chunks/mod.rs

Comment thread relay-profiling/src/perfetto/mod.rs

cursor Bot reviewed Apr 28, 2026

View reviewed changes

Comment thread relay-profiling/src/sample/v2.rs Outdated

Fix lint errors

34acd48

sentry Bot reviewed Apr 28, 2026

View reviewed changes

Comment thread relay-profiling/src/lib.rs

markushi added 3 commits April 29, 2026 10:47

Address PR feedback

7238762

Merge branch 'master' into feat/markushi/perfetto-profiling-support

8899544

Update Changelog

db555e6

sentry Bot reviewed Apr 30, 2026

View reviewed changes

Comment thread relay-profiling/src/perfetto/mod.rs

Dav1dde requested changes Apr 30, 2026

View reviewed changes

jjbayer reviewed May 4, 2026

View reviewed changes

cursor Bot mentioned this pull request May 7, 2026

feat(profiling): Add pipeline workflow to perfetto profiling #5932

Open

markushi added 7 commits May 8, 2026 07:37

Simplify python tests, don't overwrite timeouts

75b8fee

Make comments doc friendly

ad44bf3

Improve ownership handling, introduce helper function for intern_strings

338c601

Use uuid::Uuid::from_bytes_le, avoid uncessary hex/string roundtrip

afb382f

Apply style suggestions

0990673

Add doc for new pub struct DebugImage

aecdb7c

Fix unneeded rename

ac8dc49

cursor Bot reviewed May 8, 2026

View reviewed changes

Comment thread relay-profiling/src/perfetto/mod.rs

Comment thread relay-server/src/processing/profile_chunks/process.rs

Merge branch 'master' into feat/markushi/perfetto-profiling-support

9945807

sentry Bot reviewed Jun 3, 2026

View reviewed changes

markushi mentioned this pull request Jun 3, 2026

feat(profiling): Route raw profiles through objectstore service #6025

Open

5 tasks

Fix Changelog

d057317

		#[serde(rename = "organizations:continuous-profiling-perfetto")]
		ContinuousProfilingPerfetto,

Conversation

markushi commented Feb 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

markushi commented Mar 19, 2026

Uh oh!

Uh oh!

cursor Bot left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Dav1dde left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Dav1dde Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

markushi May 8, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jjbayer Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

jjbayer Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

markushi May 11, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jjbayer May 4, 2026

Choose a reason for hiding this comment

Uh oh!

markushi May 8, 2026

Choose a reason for hiding this comment

Uh oh!

cursor Bot left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

sentry Bot Jun 3, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

markushi commented Feb 25, 2026 •

edited

Loading

cursor Bot left a comment •

edited

Loading

Dav1dde left a comment •

edited

Loading