Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions cmd/main.go
Original file line number Diff line number Diff line change
Expand Up @@ -201,10 +201,12 @@ func main() {
metricsAddr string
enableLeaderElection bool
probeAddr string
pprofAddr string
err error
)
flag.StringVar(&metricsAddr, "metrics-bind-address", ":8080", "The address the metric endpoint binds to.")
flag.StringVar(&probeAddr, "health-probe-bind-address", ":8081", "The address the probe endpoint binds to.")
flag.StringVar(&pprofAddr, "pprof-bind-address", ":8082", "The address the pprof endpoint binds to.")
flag.BoolVar(&enableLeaderElection, "leader-elect", false,
"Enable leader election for controller manager. "+
"Enabling this will ensure there is only one active controller manager.")
Expand Down Expand Up @@ -239,6 +241,7 @@ func main() {
Metrics: metricsserver.Options{BindAddress: metricsAddr},
WebhookServer: webhook.NewServer(webhook.Options{Port: 9443}),
HealthProbeBindAddress: probeAddr,
PprofBindAddress: pprofAddr,
LeaderElection: enableLeaderElection,
LeaderElectionID: "f139389e.kuadrant.io",
}
Expand Down
107 changes: 107 additions & 0 deletions doc/observability/profiling.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,107 @@
# Profiling

All Go-based Kuadrant components support runtime profiling via Go's built-in [pprof](https://pkg.go.dev/net/http/pprof) tooling. This uses controller-runtime's `PprofBindAddress` option, enabled by default on port `8082`.

Profiling is useful for diagnosing performance issues such as slow reconciliation, high CPU usage, or excessive memory allocation at scale.

## Connecting to a component

Port-forward to the component you want to profile:

```bash
# Kuadrant Operator
kubectl port-forward -n kuadrant-system deploy/kuadrant-operator-controller-manager 8082:8082

# Authorino
kubectl port-forward -n <authorino-namespace> deploy/authorino 8083:8082

# Authorino Operator
kubectl port-forward -n authorino-operator deploy/authorino-operator-controller-manager 8084:8082

# Limitador Operator
kubectl port-forward -n limitador-operator-system deploy/limitador-operator-controller-manager 8085:8082

# DNS Operator
kubectl port-forward -n dns-operator-system deploy/dns-operator-controller-manager 8086:8082
```

When profiling multiple components simultaneously, use different local ports (8082-8086) as shown above.

## Capturing profiles

### CPU profile

Captures a CPU profile for a specified duration (default 30 seconds):

```bash
go tool pprof http://localhost:8082/debug/pprof/profile?seconds=30
```

### Heap profile

Captures a snapshot of current memory allocations:

```bash
go tool pprof http://localhost:8082/debug/pprof/heap
```

### Goroutine dump

Lists all goroutines and their stack traces, useful for diagnosing stuck reconciliation:

```bash
curl http://localhost:8082/debug/pprof/goroutine?debug=2
```

## Analysing profiles

### Interactive web UI

The most useful way to view profiles — opens a browser with flame graphs, call graphs, and source annotation:

```bash
go tool pprof -http=:8080 http://localhost:8082/debug/pprof/profile?seconds=30
```

### Save and view later

```bash
# Save to disk
curl -o cpu.prof "http://localhost:8082/debug/pprof/profile?seconds=30"
curl -o heap.prof "http://localhost:8082/debug/pprof/heap"

# View saved profiles
go tool pprof -http=:8080 cpu.prof
```

### Compare two profiles

Useful for validating that a change improved performance:

```bash
go tool pprof -http=:8080 -diff_base=before.prof after.prof
```

Functions that got faster appear in green, slower in red.

### Terminal-only (no browser)

```bash
go tool pprof -top cpu.prof # top functions by flat CPU
go tool pprof -top -cum cpu.prof # top functions by cumulative CPU
go tool pprof -top heap.prof # top memory allocators
```

## Configuration

Each component accepts a `--pprof-bind-address` flag:

| Component | Flag | Default |
|---|---|---|
| kuadrant-operator | `--pprof-bind-address` | `:8082` |
| authorino | `--pprof-bind-address` | `:8082` |
| authorino-operator | `--pprof-bind-address` | `:8082` |
| limitador-operator | `--pprof-bind-address` | `:8082` |
| dns-operator | `--pprof-bind-address` | `:8082` |

Set to an empty string to disable: `--pprof-bind-address=""`.
Loading