Skip to content

fix(llmobs): make span data JSON-safe at span finish [backport 4.14] - #19813

Open
dd-octo-sts[bot] wants to merge 1 commit into
4.14from
backport-19758-to-4.14
Open

fix(llmobs): make span data JSON-safe at span finish [backport 4.14]#19813
dd-octo-sts[bot] wants to merge 1 commit into
4.14from
backport-19758-to-4.14

Conversation

@dd-octo-sts

@dd-octo-sts dd-octo-sts Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Backport #19758 to 4.14

## Description

Some integrations pass raw SDK request/config objects straight into the `metadata`/`config`
dicts handed to `_annotate_llmobs_span_data()` — for example, the Google GenAI integration puts
`SafetySetting` objects from the request config into span metadata. Those values ride along on
the APM span's `_llmobs` meta_struct, which gets JSON-encoded by the agentless APM exporter later.
That encoder has no fallback for non-JSON-serializable objects, so a `TypeError` there causes the
**entire trace payload to be silently dropped** — not just the offending span.

Observed in production as:
```
TypeError: Object of type SafetySetting is not JSON serializable
 when serializing list item 0
 when serializing dict item 'safety_settings'
 when serializing dict item 'metadata'
 when serializing dict item 'meta'
 when serializing dict item '_llmobs'
 when serializing dict item 'meta_struct'
 when serializing list item N
 when serializing dict item 'spans'
```

This has shown up per-integration so far (Google GenAI's `safety_settings`, LangGraph's
`tool_config` in #19711). Rather than chase it integration-by-integration — or field-by-field —
this enforces JSON-safety once, **at span finish**.

Claude session: `e0d1ad2b-5a78-4d3f-a480-4179abc30060`
Resume: `claude --resume e0d1ad2b-5a78-4d3f-a480-4179abc30060`

Co-authored-by: yun.kim <yun.kim@datadoghq.com>
(cherry picked from commit b16f02e)

Co-authored-by: Yun Kim <35776586+Yun-Kim@users.noreply.github.com>
@dd-octo-sts
dd-octo-sts Bot requested review from a team as code owners August 21, 2026 21:22
@dd-octo-sts
dd-octo-sts Bot requested review from Kyle-Verhoog and removed request for a team August 21, 2026 21:22
@cit-pr-commenter-54b7da

Copy link
Copy Markdown

Codeowners resolved as

Resolved from the full PR diff against 4.14 using the target branch CODEOWNERS file.
CODEOWNERS team requests not listed below are not required by the current file set.

ddtrace/llmobs/_llmobs.py                                               @DataDog/ml-observability
ddtrace/llmobs/_utils.py                                                @DataDog/ml-observability
releasenotes/notes/llmobs-metadata-json-safety-9c3f7a1e4b2d6f80.yaml    @DataDog/apm-python
tests/contrib/google_genai/utils.py                                     @DataDog/ml-observability
tests/contrib/litellm/utils.py                                          @DataDog/ml-observability
tests/llmobs/test_llmobs.py                                             @DataDog/ml-observability
tests/llmobs/test_utils.py                                              @DataDog/ml-observability

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

Circular import analysis

⚠️ Existing circular imports

There are 4 circular imports that already exist on the base branch and have not been changed by this PR.

ddtrace.contrib.internal.django.patch -> ddtrace.contrib.internal.django.response -> ddtrace.contrib.internal.django.patch
ddtrace.errortracking._handled_exceptions.bytecode_injector -> ddtrace.errortracking._handled_exceptions.callbacks -> ddtrace.errortracking._handled_exceptions.collector -> ddtrace.errortracking._handled_exceptions.bytecode_reporting -> ddtrace.errortracking._handled_exceptions.bytecode_injector
ddtrace.llmobs -> ddtrace.llmobs._evaluators -> ddtrace.llmobs._evaluators.format -> ddtrace.llmobs._experiment -> ddtrace.llmobs
ddtrace.appsec._asm_request_context -> ddtrace.appsec._iast._iast_request_context_base -> ddtrace.appsec._iast._iast_env -> ddtrace.appsec._iast.reporter -> ddtrace.appsec._exploit_prevention.stack_traces -> ddtrace.appsec._asm_request_context

@cit-pr-commenter-54b7da

Copy link
Copy Markdown

Dependency direction analysis

⚠️ Existing dependency direction violations

There are 257 dependency direction violations that already exist on the base branch and have not been changed by this PR.

Show existing violations (showing 5 of 257 highest severity)
ddtrace.internal.tracemethods -×-> ddtrace.trace  (internal-core -> product:tracing, score=135)
ddtrace.internal.ci_visibility.filters -×-> ddtrace.trace  (product:ci_visibility -> product:tracing, score=133)
ddtrace.llmobs._integrations.claude_agent_sdk -×-> ddtrace.trace  (product:llmobs -> product:tracing, score=133)
ddtrace.profiling.collector.pytorch -×-> ddtrace.trace  (product:profiling -> product:tracing, score=133)
ddtrace.profiling.collector.stack -×-> ddtrace.trace  (product:profiling -> product:tracing, score=133)

To see all violations, download the layers-base.json and layers-pr.json artifacts from this CI job and run:

uv run --script scripts/import-analysis/layers.py compare layers-base.json layers-pr.json

@datadog-datadog-prod-us1

datadog-datadog-prod-us1 Bot commented Aug 21, 2026

Copy link
Copy Markdown
Contributor

Pipelines  Tests

Unblock PR with BitsAI

⚠️ Warnings

🚦 24 Pipeline jobs failed

DataDog/apm-reliability/dd-trace-py | llmobs/pydantic_ai 1/16 — 🔧 Needs a code fix, caused by this PR

View in Datadog · View in GitLab

DataDog/apm-reliability/dd-trace-py | llmobs/pydantic_ai 10/16 — 🔧 Needs a code fix, caused by this PR

View in Datadog · View in GitLab

DataDog/apm-reliability/dd-trace-py | llmobs/pydantic_ai 11/16 — 🔧 Needs a code fix, caused by this PR

View in Datadog · View in GitLab

View all 24 failed jobs.

ℹ️ Info

No other issues found (see more)

🧪 All tests passed
❄️ No new flaky tests detected

Useful? React with 👍 / 👎

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 30e4c56 | Docs | View more details | Give us feedback!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants