feat(observability): add configurable grafana.prometheusDatasourceName to agent chart

[mesutoezdil]

Closes #1891

Adds a grafana.prometheusDatasourceName field to the observability agent chart values.

The observability agent reaches Prometheus exclusively through the Grafana MCP tool server.

Tools like query_prometheus take a Grafana datasource name or UID, not a raw endpoint.
When grafana.prometheusDatasourceName is set, it is injected into the agent system message so the agent knows which Grafana datasource to use for Prometheus queries across restarts.

When left empty (default), nothing changes.

[Copilot]

Pull request overview

Note

Copilot was unable to run its full agentic suite in this review.

Adds a configurable Prometheus server URL to the observability agent Helm chart and injects it into the agent’s system message so the agent can reliably query the correct Prometheus endpoint.

Changes:

• Introduces prometheus.url in chart values as an optional configuration.
• Conditionally renders a “Prometheus Configuration” section in the agent system message when the URL is set.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File	Description
helm/agents/observability/values.yaml	Adds a new prometheus.url value (default empty) with inline documentation.
helm/agents/observability/templates/agent.yaml	Conditionally injects the configured Prometheus URL into the agent’s system prompt.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[iplay88keys]

I'm not sure that the URL will do much in practice here. The observability agent reaches Prometheus exclusively through the Grafana tool server. The query_prometheus tool takes a Grafana datasourceUid, not a raw endpoint and the grafana-mcp tool server is configured with only GRAFANA_URL and a service-account token. None of the tools attached to the agent take in a Prometheus URL.

I think the underlying issue is the agent not knowing which datasource to query, which would mean that the fix would be to inject the Grafana datasource name or UID rather than a raw url. Can you clarify a bit more about how the URL is used? Maybe I'm missing something here.

[mesutoezdil]

I'm not sure that the URL will do much in practice here. The observability agent reaches Prometheus exclusively through the Grafana tool server. The query_prometheus tool takes a Grafana datasourceUid, not a raw endpoint and the grafana-mcp tool server is configured with only GRAFANA_URL and a service-account token. None of the tools attached to the agent take in a Prometheus URL.

I think the underlying issue is the agent not knowing which datasource to query, which would mean that the fix would be to inject the Grafana datasource name or UID rather than a raw url. Can you clarify a bit more about how the URL is used? Maybe I'm missing something here.

thx for catching that. yeah, i v updated the code to use grafana.prometheusDatasourceName instead. when set, the agent is told which grafan datasource to use for query_prometheus and related tools. and this keeps the original goal (persistence across restarts) while giving the agent sth it can actually act on.

[mesutoezdil]

I'm not sure that the URL will do much in practice here. The observability agent reaches Prometheus exclusively through the Grafana tool server. The query_prometheus tool takes a Grafana datasourceUid, not a raw endpoint and the grafana-mcp tool server is configured with only GRAFANA_URL and a service-account token. None of the tools attached to the agent take in a Prometheus URL.
I think the underlying issue is the agent not knowing which datasource to query, which would mean that the fix would be to inject the Grafana datasource name or UID rather than a raw url. Can you clarify a bit more about how the URL is used? Maybe I'm missing something here.

thx for catching that. yeah, i v updated the code to use grafana.prometheusDatasourceName instead. when set, the agent is told which grafan datasource to use for query_prometheus and related tools. and this keeps the original goal (persistence across restarts) while giving the agent sth it can actually act on.

I have updated the PR title and the pr description.

[mesutoezdil]

dug into the grafana/mcp-grafana source to be more precise here.

query_prometheus (and all related tools like list_prometheus_metric_names, list_prometheus_label_names, etc.) strictly require a datasourceUid, not a name:

  type QueryPrometheusParams struct {
      DatasourceUID string `json:"datasourceUid" jsonschema:"required,description=The UID of the datasource to 
  query"`
      ...
  }

so grafana.prometheusDatasourceName is not directly usable in tool calls. the agent would need to call get_datasource_by_name first to resolve the name to a uid, then pass that uid to the prometheus tools.

2 options from here:

nr1: change to grafana.prometheusDatasourceUid so the agent can use the value directly without a lookup. downside: users need to know their datasource uid upfront.

nr2: keep prometheusDatasourceName and update the system message to explicitly tell the agent to resolve it via get_datasource_by_name before calling any prometheus tools. more user-friendly to configure, 1 extra tool call at runtime.

which one is the best @iplay88keys?