[Integration Proposal] Kafka Connect: Connector & Task Health Monitoring (REST API)

[Priyanshu-fusionpact]

Is your feature request related to a problem? Please describe.

Datadog has built-in JMX-based checks for Kafka brokers (kafka) and consumer
group lag (kafka_consumer), and Data Streams Monitoring covers Kafka Connect
worker throughput. However, there is currently no dedicated Agent check for
connector- and task-level operational state in Kafka Connect clusters.

This matters because Kafka Connect workers can stay "up" and healthy at the
process level while individual connectors or tasks silently fail — a failed
sink/source connector doesn't crash the worker, it just stops moving data,
often with no alert until someone notices downstream data is stale.

Describe the solution you'd like

We at Fusionpact Technologies would like to propose and maintain a new
kafka_connect integration in integrations-extras, polling the Kafka
Connect REST API (default port 8083: /connectors?expand=status,
/connectors?expand=info).

Metrics

• kafka_connect.connectors.total (gauge) — count of connectors by status (running/failed/paused/unassigned)
• kafka_connect.tasks.total (gauge) — count of tasks by state
• kafka_connect.connector.status (gauge, tagged connector:<name>, type:<sink|source>)
• kafka_connect.task.status (gauge, tagged connector:<name>, task_id:<id>, worker_id:<host>) — worker tag makes it possible to trace which worker a task landed on after a rebalance

Service checks

• kafka_connect.can_connect — REST API reachability
• kafka_connect.connector.state — OK/WARNING/CRITICAL by connector status
• kafka_connect.task.state — OK/CRITICAL by task status

Features

• Multi-cluster support (multiple Connect endpoints in conf.yaml.example)
• Basic Auth + TLS/SSL support, since Connect clusters in production are usually secured
• OOTB dashboard + recommended monitors for FAILED connector/task states

Tooling: Python 3, built via ddev, following standard integrations-extras structure (manifest.json, spec.yaml, metadata.csv, service_checks.json, unit + integration tests via docker-compose).

Describe alternatives you've considered

• Data Streams Monitoring: covers Connect worker throughput, not connector/task status — complementary, not overlapping.
• Generic jmx_check / OpenMetrics: possible stopgap, but lacks the structured service checks, dashboard, and Connect-specific taxonomy a dedicated check provides.
• Existing kafka / kafka_consumer checks: cover brokers and consumer lag respectively, not Connect's own worker/connector/task layer.

Additional context

• Maintainer: Fusionpact Technologies Inc.
• Maintainer contact alias: devops@fusionpact.com

Dev environment is set up and we've surveyed integrations-core/integrations-extras
to confirm no existing or in-flight work covers this. Happy to follow up with
a draft PR once scope is confirmed with a maintainer.