Define dataflows for community-owned integrations - #3095
Conversation
CI note:
|
| PR | Files | Dirs already drifting on master |
run / Validate |
|---|---|---|---|
| #3094 | 4 | 2 (gatekeeper, kepler) |
fail |
| #3095 | 24 | 6 | fail |
| #3096 | 43 | 16 | fail |
| #3097 | 10 | 0 | pass |
#3097 is the natural control: same generator, same file shape, zero drifting directories, and run / Validate is green. That isolates the failure to pre-existing drift rather than to anything about dataflows.yaml.
Two further confirmations:
-
The string
dataflowdoes not appear anywhere in the validation output of the failing runs. (It appears 5 times in each log, all ingit fetchbranch-listing noise.) No validator names adataflows.yamlfile. The failing steps are only "Validate default configuration files" and "Validate configuration models". -
masteremits the identical errors. The lastValidate repositoryrun onmaster— run 30528431295 at9c197fe, the exact SHA these branches are based on — already reports, verbatim:##[error]File `conf.yaml.example` is not in sync, run "ddev validate config kepler -s" ##[error]File `defaults.py` is not in sync, run "ddev validate models gatekeeper -s" ##[error]File `instance.py` is not in sync, run "ddev validate models gatekeeper -s"That master run has 182
is not in syncerrors across 39 integration directories, and 10 of the last 10Validate repositoryruns onmasterfailed. Because PRs run withTARGET=changed, touching any of those 39 directories pulls the drift into scope.
This PR adds one new file per directory and changes nothing that ddev validate config/models inspects. Fixing the repo-wide codegen drift is out of scope here.
e176c9b to
e402083
Compare
philjlee
changed the title
e402083 to
8b71c6a
Compare
28c2cc3 to
44277f1
Compare
8b71c6a to
72f24da
Compare
44277f1 to
28c2cc3
Compare
72f24da to
1f3cede
Compare
1f3cede to
736433b
Compare
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 736433b2df
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| @@ -0,0 +1,6 @@ | |||
| provides: | |||
| - id: redpanda-metrics | |||
There was a problem hiding this comment.
When users enable the shipped Redpanda log configuration, the Agent collects journald logs with source: redpanda (redpanda/assets/configuration/spec.yaml, lines 38–41), and the manifest also declares that log source. Because this file lists only metrics, dataflow consumers will incorrectly treat Redpanda as not providing logs; add a redpanda-logs inbound entry.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed. Verified against tidb-logs — same shape (journald-only logs template in spec.yaml, no assets/logs/ pipeline) and already merged as precedent (#3100). Added redpanda-logs.
| @@ -0,0 +1,6 @@ | |||
| provides: | |||
| - id: gnatsd-streaming-metrics | |||
There was a problem hiding this comment.
When a fault-tolerant NATS Streaming server changes state, gnatsd_streaming.py submits a failover event, and test_failover_event verifies that behavior. Listing only metrics therefore hides an event type that the check actively provides from dataflow consumers; add a gnatsd-streaming-events inbound entry.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed. _failover_check submits a real event (covered by test_failover_event), matching the existing direction: inbound, data_type: events precedent (e.g. redis-sentinel-events, gatling-enterprise-events). Added gnatsd-streaming-events.
| @@ -0,0 +1,6 @@ | |||
| provides: | |||
| - id: launchdarkly-metrics | |||
There was a problem hiding this comment.
When customers configure the documented LaunchDarkly events integration, flag, project, and environment events are sent into Datadog (launchdarkly/README.md, lines 21–23 and 61–63), and the manifest sets creates_events to true. This metrics-only declaration consequently makes catalog consumers miss a supported data type; add a launchdarkly-events inbound entry.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed. Manifest sets creates_events: true and the README documents the events integration pushing flag/project/environment events into Datadog via API key (inbound, not an outbound notification tile). Added launchdarkly-events.
| @@ -0,0 +1,6 @@ | |||
| provides: | |||
| - id: rigor-metrics | |||
There was a problem hiding this comment.
When a Rigor check crosses its alert threshold or recovers, the integration pushes an event into Datadog (rigor/README.md, lines 117–124), consistent with the manifest's creates_events: true. Declaring only metrics causes dataflow consumers to omit this supported event stream; add a rigor-events inbound entry.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed. Manifest sets creates_events: true and the README documents Failed/Back-online events pushed into Datadog. Added rigor-events.
| @@ -0,0 +1,6 @@ | |||
| provides: | |||
| - id: sigsci-metrics | |||
There was a problem hiding this comment.
When Signal Sciences flags an IP address, the integration sends an event to the Datadog Event Stream (sigsci/README.md, lines 90–92), and its manifest explicitly sets creates_events to true. The metrics-only file therefore underreports the integration's provided data types; add a sigsci-events inbound entry.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Confirmed and fixed. Manifest sets creates_events: true and the README documents events sent to the Datadog Event Stream when an IP is flagged. Added sigsci-events.
Add assets/dataflows.yaml to 24 integrations owned by a vendor GitHub organization or a corporate support alias, with no @DataDog/... team CODEOWNER. Each entry is derived mechanically: a metadata.csv with at least one metric row yields a metrics dataflow, an assets/logs/ pipeline yields a logs dataflow. Dataflow IDs are <app_id from manifest.json>-<data_type>. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
- redpanda: adds redpanda-logs (journald log template in spec.yaml, same pattern as tidb's already-merged logs entry) - gnatsd_streaming: adds gnatsd-streaming-events (check submits a failover event, covered by test_failover_event) - launchdarkly, rigor, sigsci: add <app>-events (manifest creates_events: true, and README documents each integration pushing Datadog events)
36af90b to
5696964
Compare
2 participants
Jira: TXP-277
What
Adds
assets/dataflows.yamlto 24 integrations (31 dataflow entries: 22metrics, 5logs, 4events).This is batch 2 of the integrations-extras dataflows backfill. Batch 1 was #3093, which was closed and re-split by team into #3100–#3103 — all four are now merged.
Selection criteria
An integration is in scope for this PR if all of the following hold, computed from the repo at
9c197fe:manifest.json(the dataflows validator hard-requires one).assets/dataflows.yaml.metadata.csvwith at least one data row, and/or a log pipeline underassets/logs/. Nodata_typewas ever guessed..github/CODEOWNERSresolves<dir>/assets/dataflows.yaml(last-match-wins) to an owner set containing no@DataDog/...team.fluxcd/traefik(README-only, disabled in APW).Then, within that set of 81, this PR takes the cluster with vendor-organization owners: The CODEOWNERS entry for these paths resolves to a vendor GitHub organization/team (e.g.
@Mergifyio/oss-integrations,@symopsio/eng,@EMnify/development) and/or a corporate support alias (e.g.support@redpanda.com,field.engineers@redis.com).Content
Every file follows the uniform template already used by the pre-existing files in this repo:
idismanifest.json'sapp_idplus-<data_type>. That is the mechanical rule the existing entries follow — not the directory name. For all 81 integrations in batch 2 the two happen to coincide after slugification, butapp_idis the authority.metadata.csvrows produce ametricsentry, anassets/logs/pipeline (or, pertidb-logsprecedent, a journald-onlylogstemplate inspec.yaml) produces alogsentry, and a check/manifest that submits Datadog events (self.event(), orcreates_events: truebacked by a documented events integration) produces aneventsentry. Integrations with more than one get all of them (metrics, logs, events).Integrations in this PR
ambassadorbonsaiemnifygnatsdgnatsd_streamingtest_failover_event)k6launchdarklycreates_events)mergifymergify_oauthmodalocientperimeterxqdrantredis_cloudredis_enterpriseredis_enterprise_prometheusredpandalogstemplate (tidb-logsprecedent)reflectizrigorcreates_events)rundecksigscicreates_events)stardogsymvespaCoverage
As of 2026-08-21,
mastercarries 107assets/dataflows.yamlfiles (batch 1's #3100–#3103 and batch 2's #3094 have since merged). If this PR merges:masterBatch 1 + batch 2 together complete the entire mechanical bucket — every integration in the repo with an unambiguous local
data_typesignal.Validation
These files were validated by running the real upstream validator, not by static inspection.
DataflowsValidationHandlerfromddoghq/dd-source@main(domains/integrationscatalog/libs/catalogassetslib/dataflows_validation_handler.go) was executed viago test -overlay, with only the COBS-backed dataflows client and the deployment handler stubbed out. The validation code path itself is byte-for-byte upstream — the handler file was diffed againstddoghq/dd-source@mainand is identical. Note thatvalidDataTypeshas grown since this PR was opened and is now 16 values (rum_eventsandsecurity_investigation_api_callshave both landed since), so re-diff before relying on this claim again.The harness was negative-tested first, and correctly rejects all of: missing
always_on; adata_typeoutside the enum; anidviolating^[a-z0-9-]+$; anidshorter than 3 characters; a file with neitherprovidesnoruses; a.ymlextension; a missingmanifest.json; and the same dataflow ID provided by two apps. A known-good baseline file is accepted. So a pass here means something.APW also posts a
validate-dataflowsstatus check on extras PRs, so there will be pre-merge feedback from the pipeline itself as well.CI note —
run / Validateredness is pre-existingThe
run / Validatejob is still failing onmaster(confirmed 2026-08-21, most recent runs allfailure). Because the workflow runs withTARGET=changedon PRs, touching a directory pulls that pre-existing drift into this PR's scope even though this PR only adds a new file and changes nothing thatddev validate config/modelsinspects. 6 of the 24 directories in this PR still carry that pre-existing drift onmaster:mergify,ocient,qdrant,redis_cloud,redis_enterprise,redpanda. This is unrelated to dataflows and is not something this PR attempts to fix.Codex review round
Codex flagged 5 missed entries, all confirmed and fixed:
redpanda-logs(journaldlogstemplate inspec.yaml, matching the already-mergedtidb-logsprecedent), andeventsentries forgnatsd_streaming,launchdarkly,rigor, andsigsci(each verified against source/README to actually submit Datadog events inbound, not just an unratified outbound-tile guess — see replies on the individual review comments).Deferred, not in scope
Nothing was dropped from this PR for an ambiguous
data_type— every integration in the batch-2 set mapped cleanly tometricsand/orlogs. What remains unaddressed repo-wide after batch 1 + batch 2 is the research bucket, which needs schema decisions rather than scripting:cfssl,cybersixgill_actionable_alerts,go_pprof_scraper,isdown,reboot_required) — service checks map to nodata_type.events+direction: outboundis the least-bad fit but has not been ratified by anyone.manifest.json(aerospike_enterprise,scamalytics,warpstream, plusfluxcd/traefikwhich are README-only and explicitly disabled in APW) — the validator requires a manifest.The RUM SDK tile bucket that used to be listed here is done:
rum_eventslanded invalidDataTypes(dd-source#50640) and extras#3108 populated the 15 RUM SDK tiles, both merged.Related: #3093 (batch 1, closed/superseded by #3100–#3103), #2925 (the original dataflows PR for saas-integrations).
🤖 Generated with Claude Code