From 27e78e5bdd6b303c938b0534c8ad60256e9ef090 Mon Sep 17 00:00:00 2001 From: Anand Kumar Sankaran Date: Mon, 2 Mar 2026 15:39:04 -0800 Subject: [PATCH 01/15] Proposal for REST API to access events and metrics --- .../in-dev/unreleased/proposals/_index.md | 34 + .../proposals/observability-rest-api.md | 922 ++++++++++++++++++ 2 files changed, 956 insertions(+) create mode 100644 site/content/in-dev/unreleased/proposals/_index.md create mode 100644 site/content/in-dev/unreleased/proposals/observability-rest-api.md diff --git a/site/content/in-dev/unreleased/proposals/_index.md b/site/content/in-dev/unreleased/proposals/_index.md new file mode 100644 index 0000000000..302ff5fba2 --- /dev/null +++ b/site/content/in-dev/unreleased/proposals/_index.md @@ -0,0 +1,34 @@ +--- +title: Proposals +linkTitle: Proposals +weight: 900 +--- + + +# Proposals + +This section contains design proposals for new features and enhancements to Apache Polaris. + +## Active Proposals + +| Proposal | Status | Description | +|----------|--------|-------------| +| [Observability REST API](observability-rest-api.md) | Draft | REST API endpoints for querying table metrics and catalog events | + diff --git a/site/content/in-dev/unreleased/proposals/observability-rest-api.md b/site/content/in-dev/unreleased/proposals/observability-rest-api.md new file mode 100644 index 0000000000..0435687a5d --- /dev/null +++ b/site/content/in-dev/unreleased/proposals/observability-rest-api.md @@ -0,0 +1,922 @@ +--- +title: Observability REST API +linkTitle: Observability REST API +weight: 100 +--- + + +# Proposal: REST API for Querying Table Metrics and Events + +**Author:** Anand Sankaran +**Date:** 2026-03-02 +**Status:** Draft Proposal +**Target:** Apache Polaris + +--- + +## Abstract + +This proposal defines REST API endpoints for querying table metrics and catalog events from Apache Polaris. The endpoints expose data already being persisted via the existing JDBC persistence model (`events`, `scan_metrics_report`, `commit_metrics_report` tables) and follow established Polaris API patterns. + +--- + +## Table of Contents + +1. [Motivation](#1-motivation) +2. [Use Cases](#2-use-cases) +3. [Design Principles](#3-design-principles) +4. [API Specification](#4-api-specification) +5. [Authorization](#5-authorization) +6. [OpenAPI Schema](#6-openapi-schema) +7. [Implementation Notes](#7-implementation-notes) + +--- + +## 1. Motivation + +Apache Polaris currently persists table metrics (scan reports, commit reports) and catalog events to the database, but provides no REST API to query this data. Users must access the database directly to retrieve metrics or audit information. + +Adding read-only REST endpoints enables: +- Programmatic access to metrics without database credentials +- Integration with monitoring dashboards and alerting systems +- Consistent authorization via Polaris RBAC +- Pagination and filtering without writing SQL + +--- + +## 2. Use Cases + +### 2.1 Table Health Monitoring +- Track write patterns: files added/removed per commit, record counts, duration trends +- Identify tables with high commit frequency or unusually large commits +- Detect issues indicating need for compaction (many small files) or optimization + +### 2.2 Query Performance Analysis +- Understand read patterns: files scanned vs skipped, planning duration +- Identify inefficient queries with low manifest/file pruning ratios +- Correlate performance with filter expressions and projected columns + +### 2.3 Capacity Planning & Chargeback +- Aggregate metrics by table, namespace, or principal over time +- Track storage growth trends (`total_file_size_bytes`) +- Attribute usage to teams/users via `principal_name` + +### 2.4 Debugging & Troubleshooting +- Correlate metrics with distributed traces (`otel_trace_id`, `otel_span_id`) +- Investigate specific commits by `snapshot_id` +- Trace operations via `request_id` + +### 2.5 Audit & Compliance +- Track who created/dropped/modified catalog objects +- Monitor administrative actions (credential rotation, grant changes) +- Generate compliance reports for access patterns + +--- + +## 3. Design Principles + +| Principle | Rationale | +|-----------|-----------| +| **Management API namespace** | Use `/api/management/v1/...` to separate from Iceberg REST Catalog paths | +| **Read-only endpoints** | Only GET methods; metrics/events are written via existing flows | +| **Consistent pagination** | Follow existing `pageToken`/`nextPageToken` patterns | +| **Flexible filtering** | Time ranges, principal, snapshot - common query patterns | +| **RBAC integration** | Leverage existing Polaris authorization model | + +--- + +## 4. API Specification + +### 4.1 Endpoint Summary + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/management/v1/catalogs/{catalogName}/events` | List events for a catalog | +| GET | `/api/management/v1/catalogs/{catalogName}/events/{eventId}` | Get a specific event | +| GET | `/api/management/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/scan-metrics` | List scan metrics for a table | +| GET | `/api/management/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/commit-metrics` | List commit metrics for a table | + +### 4.2 Path Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `catalogName` | string | Name of the catalog | +| `namespace` | string | Namespace (URL-encoded, multi-level separated by `%1F`) | +| `table` | string | Table name | +| `eventId` | string | Unique event identifier | + +### 4.3 Query Parameters + +#### List Events (`/catalogs/{catalogName}/events`) + +| Parameter | Type | Required | Default | Description | +|-----------|------|----------|---------|-------------| +| `pageToken` | string | No | - | Cursor for pagination (from previous response) | +| `pageSize` | integer | No | 100 | Results per page (max: 1000) | +| `eventType` | string | No | - | Filter by event type (e.g., `AFTER_CREATE_TABLE`) | +| `resourceType` | string | No | - | Filter by resource: `CATALOG`, `NAMESPACE`, `TABLE`, `VIEW` | +| `resourceIdentifier` | string | No | - | Filter by resource identifier (exact match) | +| `principalName` | string | No | - | Filter by principal who triggered the event | +| `timestampFrom` | long | No | - | Start of time range (epoch milliseconds, inclusive) | +| `timestampTo` | long | No | - | End of time range (epoch milliseconds, exclusive) | + +#### List Scan Metrics (`/.../tables/{table}/scan-metrics`) + +| Parameter | Type | Required | Default | Description | +|-----------|------|----------|---------|-------------| +| `pageToken` | string | No | - | Cursor for pagination | +| `pageSize` | integer | No | 100 | Results per page (max: 1000) | +| `snapshotId` | long | No | - | Filter by snapshot ID | +| `principalName` | string | No | - | Filter by principal | +| `timestampFrom` | long | No | - | Start of time range (epoch ms) | +| `timestampTo` | long | No | - | End of time range (epoch ms) | + +#### List Commit Metrics (`/.../tables/{table}/commit-metrics`) + +| Parameter | Type | Required | Default | Description | +|-----------|------|----------|---------|-------------| +| `pageToken` | string | No | - | Cursor for pagination | +| `pageSize` | integer | No | 100 | Results per page (max: 1000) | +| `snapshotId` | long | No | - | Filter by snapshot ID | +| `operation` | string | No | - | Filter by operation: `append`, `overwrite`, `delete`, `replace` | +| `principalName` | string | No | - | Filter by principal | +| `timestampFrom` | long | No | - | Start of time range (epoch ms) | +| `timestampTo` | long | No | - | End of time range (epoch ms) | + +### 4.4 Example Requests and Responses + +#### List Events + +**Request:** +```http +GET /api/management/v1/catalogs/my-catalog/events?pageSize=2&eventType=AFTER_CREATE_TABLE×tampFrom=1709251200000 +Authorization: Bearer +``` + +**Response:** +```json +{ + "nextPageToken": "eyJ0cyI6MTcwOTMzNzYxMjM0NSwiaWQiOiI1NTBlODQwMCJ9", + "events": [ + { + "eventId": "550e8400-e29b-41d4-a716-446655440000", + "catalogId": "my-catalog", + "requestId": "req-12345", + "eventType": "AFTER_CREATE_TABLE", + "timestampMs": 1709337612345, + "principalName": "admin@example.com", + "resourceType": "TABLE", + "resourceIdentifier": "analytics.events.page_views", + "additionalProperties": { + "table-uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890" + } + }, + { + "eventId": "661f9511-f30c-52e5-b827-557766551111", + "catalogId": "my-catalog", + "requestId": "req-12346", + "eventType": "AFTER_CREATE_TABLE", + "timestampMs": 1709337500000, + "principalName": "etl-service@example.com", + "resourceType": "TABLE", + "resourceIdentifier": "analytics.events.user_actions", + "additionalProperties": { + "table-uuid": "b2c3d4e5-f6a7-8901-bcde-f23456789012" + } + } + ] +} +``` + +#### Get Single Event + +**Request:** +```http +GET /api/management/v1/catalogs/my-catalog/events/550e8400-e29b-41d4-a716-446655440000 +Authorization: Bearer +``` + +**Response:** +```json +{ + "eventId": "550e8400-e29b-41d4-a716-446655440000", + "catalogId": "my-catalog", + "requestId": "req-12345", + "eventType": "AFTER_CREATE_TABLE", + "timestampMs": 1709337612345, + "principalName": "admin@example.com", + "resourceType": "TABLE", + "resourceIdentifier": "analytics.events.page_views", + "additionalProperties": { + "table-uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "traceparent": "00-abc123def456789012345678901234-def456789012-01" + } +} +``` + +#### List Scan Metrics + +**Request:** +```http +GET /api/management/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views/scan-metrics?pageSize=2×tampFrom=1709251200000 +Authorization: Bearer +``` + +**Response:** +```json +{ + "nextPageToken": null, + "reports": [ + { + "reportId": "scan-001-abc123", + "catalogId": 12345, + "tableId": 67890, + "timestampMs": 1709337612345, + "principalName": "analyst@example.com", + "requestId": "req-scan-001", + "otelTraceId": "abc123def456789012345678901234", + "otelSpanId": "def456789012", + "snapshotId": 1234567890123, + "schemaId": 0, + "filterExpression": "event_date >= '2024-03-01'", + "projectedFieldIds": "1,2,3,5,8", + "projectedFieldNames": "event_id,user_id,event_type,timestamp,page_url", + "resultDataFiles": 150, + "resultDeleteFiles": 5, + "totalFileSizeBytes": 1073741824, + "totalDataManifests": 12, + "totalDeleteManifests": 2, + "scannedDataManifests": 8, + "scannedDeleteManifests": 2, + "skippedDataManifests": 4, + "skippedDeleteManifests": 0, + "skippedDataFiles": 45, + "skippedDeleteFiles": 0, + "totalPlanningDurationMs": 250, + "equalityDeleteFiles": 3, + "positionalDeleteFiles": 2, + "indexedDeleteFiles": 0, + "totalDeleteFileSizeBytes": 52428800 + } + ] +} +``` + +#### List Commit Metrics + +**Request:** +```http +GET /api/management/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views/commit-metrics?operation=append&pageSize=2 +Authorization: Bearer +``` + +**Response:** +```json +{ + "nextPageToken": "eyJ0cyI6MTcwOTMzNzcwMDAwMCwiaWQiOiJjb21taXQtMDAyIn0=", + "reports": [ + { + "reportId": "commit-001-xyz789", + "catalogId": 12345, + "tableId": 67890, + "timestampMs": 1709337800000, + "principalName": "etl-service@example.com", + "requestId": "req-commit-001", + "otelTraceId": "xyz789abc123456789012345678901", + "otelSpanId": "abc123456789", + "snapshotId": 1234567890124, + "sequenceNumber": 42, + "operation": "append", + "addedDataFiles": 10, + "removedDataFiles": 0, + "totalDataFiles": 160, + "addedDeleteFiles": 0, + "removedDeleteFiles": 0, + "totalDeleteFiles": 5, + "addedEqualityDeleteFiles": 0, + "removedEqualityDeleteFiles": 0, + "addedPositionalDeleteFiles": 0, + "removedPositionalDeleteFiles": 0, + "addedRecords": 100000, + "removedRecords": 0, + "totalRecords": 15000000, + "addedFileSizeBytes": 104857600, + "removedFileSizeBytes": 0, + "totalFileSizeBytes": 1178599424, + "totalDurationMs": 5000, + "attempts": 1 + } + ] +} +``` + +--- + +## 5. Authorization + +### 5.1 Required Privileges + +| Endpoint | Required Privilege | Scope | +|----------|-------------------|-------| +| List/Get Events | `CATALOG_MANAGE_METADATA` | Catalog | +| List Scan Metrics | `TABLE_READ_DATA` | Table | +| List Commit Metrics | `TABLE_READ_DATA` | Table | + +### 5.2 Rationale + +- **Events** contain catalog-wide audit information and should require catalog-level administrative access +- **Metrics** are table-specific and align with read access since they describe query patterns and commit history +- This follows the principle of least privilege while enabling common use cases + +### 5.3 Alternative: New Privileges + +If finer-grained control is desired, new privileges could be introduced: + +| New Privilege | Description | +|---------------|-------------| +| `CATALOG_READ_EVENTS` | Read-only access to catalog events | +| `TABLE_READ_METRICS` | Read-only access to table metrics | + +--- + +## 6. OpenAPI Schema + +Add the following to `spec/polaris-management-service.yml`: + +### 6.1 Paths + +```yaml +paths: + /catalogs/{catalogName}/events: + parameters: + - $ref: '#/components/parameters/catalogName' + get: + operationId: listCatalogEvents + summary: List events for a catalog + description: Returns a paginated list of events with optional filtering + tags: + - Observability + parameters: + - name: pageToken + in: query + schema: + type: string + - name: pageSize + in: query + schema: + type: integer + minimum: 1 + maximum: 1000 + default: 100 + - name: eventType + in: query + schema: + type: string + - name: resourceType + in: query + schema: + type: string + enum: [CATALOG, NAMESPACE, TABLE, VIEW] + - name: resourceIdentifier + in: query + schema: + type: string + - name: principalName + in: query + schema: + type: string + - name: timestampFrom + in: query + schema: + type: integer + format: int64 + - name: timestampTo + in: query + schema: + type: integer + format: int64 + responses: + '200': + description: Paginated list of events + content: + application/json: + schema: + $ref: '#/components/schemas/ListEventsResponse' + '403': + description: Insufficient privileges + '404': + description: Catalog not found + + /catalogs/{catalogName}/events/{eventId}: + parameters: + - $ref: '#/components/parameters/catalogName' + - name: eventId + in: path + required: true + schema: + type: string + get: + operationId: getEvent + summary: Get a specific event + tags: + - Observability + responses: + '200': + description: The requested event + content: + application/json: + schema: + $ref: '#/components/schemas/CatalogEvent' + '403': + description: Insufficient privileges + '404': + description: Event or catalog not found + + /catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/scan-metrics: + parameters: + - $ref: '#/components/parameters/catalogName' + - $ref: '#/components/parameters/namespace' + - $ref: '#/components/parameters/table' + get: + operationId: listTableScanMetrics + summary: List scan metrics for a table + tags: + - Observability + parameters: + - name: pageToken + in: query + schema: + type: string + - name: pageSize + in: query + schema: + type: integer + minimum: 1 + maximum: 1000 + default: 100 + - name: snapshotId + in: query + schema: + type: integer + format: int64 + - name: principalName + in: query + schema: + type: string + - name: timestampFrom + in: query + schema: + type: integer + format: int64 + - name: timestampTo + in: query + schema: + type: integer + format: int64 + responses: + '200': + description: Paginated list of scan metrics + content: + application/json: + schema: + $ref: '#/components/schemas/ListScanMetricsResponse' + '403': + description: Insufficient privileges + '404': + description: Table not found + + /catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/commit-metrics: + parameters: + - $ref: '#/components/parameters/catalogName' + - $ref: '#/components/parameters/namespace' + - $ref: '#/components/parameters/table' + get: + operationId: listTableCommitMetrics + summary: List commit metrics for a table + tags: + - Observability + parameters: + - name: pageToken + in: query + schema: + type: string + - name: pageSize + in: query + schema: + type: integer + minimum: 1 + maximum: 1000 + default: 100 + - name: snapshotId + in: query + schema: + type: integer + format: int64 + - name: operation + in: query + schema: + type: string + - name: principalName + in: query + schema: + type: string + - name: timestampFrom + in: query + schema: + type: integer + format: int64 + - name: timestampTo + in: query + schema: + type: integer + format: int64 + responses: + '200': + description: Paginated list of commit metrics + content: + application/json: + schema: + $ref: '#/components/schemas/ListCommitMetricsResponse' + '403': + description: Insufficient privileges + '404': + description: Table not found +``` + +### 6.2 Schemas + +```yaml +components: + schemas: + CatalogEvent: + type: object + required: + - eventId + - catalogId + - eventType + - timestampMs + - resourceType + - resourceIdentifier + properties: + eventId: + type: string + description: Unique event identifier + catalogId: + type: string + description: Catalog where the event occurred + requestId: + type: string + description: Request ID that triggered this event + eventType: + type: string + description: Event type (e.g., AFTER_CREATE_TABLE) + timestampMs: + type: integer + format: int64 + description: Event timestamp (epoch milliseconds) + principalName: + type: string + description: Principal who triggered the event + resourceType: + type: string + enum: [CATALOG, NAMESPACE, TABLE, VIEW] + resourceIdentifier: + type: string + description: Fully qualified resource identifier + additionalProperties: + type: object + additionalProperties: + type: string + description: Event-specific metadata + + ListEventsResponse: + type: object + properties: + nextPageToken: + type: string + description: Token for next page (null if no more results) + events: + type: array + items: + $ref: '#/components/schemas/CatalogEvent' + + ScanMetricsReport: + type: object + required: + - reportId + - catalogId + - tableId + - timestampMs + properties: + reportId: + type: string + catalogId: + type: integer + format: int64 + tableId: + type: integer + format: int64 + timestampMs: + type: integer + format: int64 + principalName: + type: string + requestId: + type: string + otelTraceId: + type: string + description: OpenTelemetry trace ID + otelSpanId: + type: string + description: OpenTelemetry span ID + snapshotId: + type: integer + format: int64 + schemaId: + type: integer + filterExpression: + type: string + projectedFieldIds: + type: string + projectedFieldNames: + type: string + resultDataFiles: + type: integer + format: int64 + resultDeleteFiles: + type: integer + format: int64 + totalFileSizeBytes: + type: integer + format: int64 + totalDataManifests: + type: integer + format: int64 + totalDeleteManifests: + type: integer + format: int64 + scannedDataManifests: + type: integer + format: int64 + scannedDeleteManifests: + type: integer + format: int64 + skippedDataManifests: + type: integer + format: int64 + skippedDeleteManifests: + type: integer + format: int64 + skippedDataFiles: + type: integer + format: int64 + skippedDeleteFiles: + type: integer + format: int64 + totalPlanningDurationMs: + type: integer + format: int64 + equalityDeleteFiles: + type: integer + format: int64 + positionalDeleteFiles: + type: integer + format: int64 + indexedDeleteFiles: + type: integer + format: int64 + totalDeleteFileSizeBytes: + type: integer + format: int64 + + ListScanMetricsResponse: + type: object + properties: + nextPageToken: + type: string + reports: + type: array + items: + $ref: '#/components/schemas/ScanMetricsReport' + + CommitMetricsReport: + type: object + required: + - reportId + - catalogId + - tableId + - timestampMs + - snapshotId + - operation + properties: + reportId: + type: string + catalogId: + type: integer + format: int64 + tableId: + type: integer + format: int64 + timestampMs: + type: integer + format: int64 + principalName: + type: string + requestId: + type: string + otelTraceId: + type: string + otelSpanId: + type: string + snapshotId: + type: integer + format: int64 + sequenceNumber: + type: integer + format: int64 + operation: + type: string + description: Commit operation (append, overwrite, delete, replace) + addedDataFiles: + type: integer + format: int64 + removedDataFiles: + type: integer + format: int64 + totalDataFiles: + type: integer + format: int64 + addedDeleteFiles: + type: integer + format: int64 + removedDeleteFiles: + type: integer + format: int64 + totalDeleteFiles: + type: integer + format: int64 + addedEqualityDeleteFiles: + type: integer + format: int64 + removedEqualityDeleteFiles: + type: integer + format: int64 + addedPositionalDeleteFiles: + type: integer + format: int64 + removedPositionalDeleteFiles: + type: integer + format: int64 + addedRecords: + type: integer + format: int64 + removedRecords: + type: integer + format: int64 + totalRecords: + type: integer + format: int64 + addedFileSizeBytes: + type: integer + format: int64 + removedFileSizeBytes: + type: integer + format: int64 + totalFileSizeBytes: + type: integer + format: int64 + totalDurationMs: + type: integer + format: int64 + attempts: + type: integer + + ListCommitMetricsResponse: + type: object + properties: + nextPageToken: + type: string + reports: + type: array + items: + $ref: '#/components/schemas/CommitMetricsReport' +``` + +--- + +## 7. Implementation Notes + +### 7.1 Database Queries + +The endpoints will query existing tables with appropriate filtering and pagination: + +```sql +-- List events with cursor-based pagination +SELECT * FROM events +WHERE realm_id = ? + AND catalog_id = ? + AND (timestamp_ms, event_id) < (?, ?) -- cursor + AND event_type = ? -- optional filter + AND timestamp_ms >= ? -- optional filter + AND timestamp_ms < ? -- optional filter +ORDER BY timestamp_ms DESC, event_id DESC +LIMIT ?; + +-- List scan metrics +SELECT * FROM scan_metrics_report +WHERE realm_id = ? + AND catalog_id = ? + AND table_id = ? + AND timestamp_ms >= ? + AND timestamp_ms < ? +ORDER BY timestamp_ms DESC, report_id DESC +LIMIT ?; +``` + +### 7.2 Recommended Indexes + +```sql +-- Events indexes +CREATE INDEX IF NOT EXISTS idx_events_catalog_ts + ON events(realm_id, catalog_id, timestamp_ms DESC, event_id DESC); +CREATE INDEX IF NOT EXISTS idx_events_type + ON events(realm_id, catalog_id, event_type, timestamp_ms DESC); + +-- Metrics indexes (may already exist) +CREATE INDEX IF NOT EXISTS idx_scan_report_lookup + ON scan_metrics_report(realm_id, catalog_id, table_id, timestamp_ms DESC); +CREATE INDEX IF NOT EXISTS idx_commit_report_lookup + ON commit_metrics_report(realm_id, catalog_id, table_id, timestamp_ms DESC); +``` + +### 7.3 Files to Modify + +| File | Changes | +|------|---------| +| `spec/polaris-management-service.yml` | Add paths and schemas | +| `api/management-service/` | Generated API interfaces | +| `runtime/service/.../admin/` | Service implementation | +| `polaris-core/.../persistence/BasePersistence.java` | Add read methods | +| `persistence/relational-jdbc/.../JdbcBasePersistenceImpl.java` | Query implementations | + +### 7.4 Pagination Token Format + +Internal format (base64-encoded JSON, opaque to clients): + +```json +{ + "ts": 1709337612345, + "id": "550e8400-e29b-41d4-a716-446655440000" +} +``` + +--- + +## Appendix: Event Types Reference + +Events are categorized by code ranges: + +| Range | Category | Examples | +|-------|----------|----------| +| 100-109 | Catalog | `AFTER_CREATE_CATALOG`, `AFTER_DELETE_CATALOG` | +| 200-217 | Catalog Role | `AFTER_CREATE_CATALOG_ROLE`, `AFTER_ADD_GRANT_TO_CATALOG_ROLE` | +| 300-319 | Principal | `AFTER_CREATE_PRINCIPAL`, `AFTER_ROTATE_CREDENTIALS` | +| 400-417 | Principal Role | `AFTER_CREATE_PRINCIPAL_ROLE`, `AFTER_ASSIGN_CATALOG_ROLE_TO_PRINCIPAL_ROLE` | +| 500-511 | Namespace | `AFTER_CREATE_NAMESPACE`, `AFTER_DROP_NAMESPACE` | +| 600-617 | Table | `AFTER_CREATE_TABLE`, `AFTER_UPDATE_TABLE`, `AFTER_DROP_TABLE` | +| 700-715 | View | `AFTER_CREATE_VIEW`, `AFTER_REPLACE_VIEW` | +| 1200-1215 | Policy | `AFTER_CREATE_POLICY`, `AFTER_ATTACH_POLICY` | +| 1300-1307 | Generic Table | `AFTER_CREATE_GENERIC_TABLE` | + +--- + +## Open Questions + +1. **Aggregations**: Are aggregated metrics views needed (e.g., daily summaries)? +2. **Privileges**: Use existing privileges or introduce new `READ_EVENTS`/`READ_METRICS`? + +--- + +## References + +- Database schema: `persistence/relational-jdbc/src/main/resources/postgres/schema-v4.sql` +- Event types: `runtime/service/src/main/java/org/apache/polaris/service/events/PolarisEventType.java` +- Metrics persistence: `runtime/service/src/main/java/org/apache/polaris/service/reporting/PersistingMetricsReporter.java` + From 23cae654aecfac1656bf4e62ed89424c59012f92 Mon Sep 17 00:00:00 2001 From: Anand Kumar Sankaran Date: Tue, 3 Mar 2026 13:07:34 -0800 Subject: [PATCH 02/15] Align Events API with Iceberg Events API specification Updates the Observability REST API proposal to align with the emerging Iceberg Events API specification (apache/iceberg#12584) per review feedback. Key changes: - Changed Events endpoint from GET to POST with request body - Moved Events API to Iceberg REST Catalog path (/api/catalog/v1/{prefix}/events) - Adopted Iceberg event structure (event-id, request-id, timestamp-ms, operation) - Added standard operation types (create-table, update-table, drop-table, etc.) - Added Polaris custom operation types with x-polaris-* prefix convention - Updated OpenAPI schemas for Iceberg compatibility - Added Section 8 documenting Iceberg alignment rationale - Added mapping table from Polaris internal events to Iceberg operations References: - Iceberg Events API PR: https://github.com/apache/iceberg/pull/12584 - Iceberg Events API Doc: https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8 --- .../proposals/observability-rest-api.md | 680 +++++++++++++----- 1 file changed, 492 insertions(+), 188 deletions(-) diff --git a/site/content/in-dev/unreleased/proposals/observability-rest-api.md b/site/content/in-dev/unreleased/proposals/observability-rest-api.md index 0435687a5d..42b7fca2a1 100644 --- a/site/content/in-dev/unreleased/proposals/observability-rest-api.md +++ b/site/content/in-dev/unreleased/proposals/observability-rest-api.md @@ -35,6 +35,8 @@ weight: 100 This proposal defines REST API endpoints for querying table metrics and catalog events from Apache Polaris. The endpoints expose data already being persisted via the existing JDBC persistence model (`events`, `scan_metrics_report`, `commit_metrics_report` tables) and follow established Polaris API patterns. +**Note:** The Events API in this proposal is designed to align with the emerging [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584), which is nearing consensus in the Apache Iceberg community. This ensures forward compatibility and consistency with the broader Iceberg ecosystem. + --- ## Table of Contents @@ -46,6 +48,7 @@ This proposal defines REST API endpoints for querying table metrics and catalog 5. [Authorization](#5-authorization) 6. [OpenAPI Schema](#6-openapi-schema) 7. [Implementation Notes](#7-implementation-notes) +8. [Iceberg Events API Alignment](#8-iceberg-events-api-alignment) --- @@ -94,10 +97,12 @@ Adding read-only REST endpoints enables: | Principle | Rationale | |-----------|-----------| -| **Management API namespace** | Use `/api/management/v1/...` to separate from Iceberg REST Catalog paths | -| **Read-only endpoints** | Only GET methods; metrics/events are written via existing flows | -| **Consistent pagination** | Follow existing `pageToken`/`nextPageToken` patterns | -| **Flexible filtering** | Time ranges, principal, snapshot - common query patterns | +| **Iceberg Events API alignment** | Events API follows the [Iceberg Events API spec](https://github.com/apache/iceberg/pull/12584) for ecosystem compatibility | +| **Management API namespace** | Metrics APIs use `/api/management/v1/...` to separate from Iceberg REST Catalog paths | +| **POST for complex filtering** | Events API uses POST with request body (per Iceberg spec) to support complex filters (arrays, nested objects) | +| **Read-only semantics** | All endpoints are read-only; metrics/events are written via existing flows | +| **Consistent pagination** | Follow `continuation-token` pattern (Iceberg) and `pageToken` pattern (Polaris management APIs) | +| **Flexible filtering** | Time ranges, operation types, catalog objects - common query patterns | | **RBAC integration** | Leverage existing Polaris authorization model | --- @@ -108,34 +113,73 @@ Adding read-only REST endpoints enables: | Method | Path | Description | |--------|------|-------------| -| GET | `/api/management/v1/catalogs/{catalogName}/events` | List events for a catalog | -| GET | `/api/management/v1/catalogs/{catalogName}/events/{eventId}` | Get a specific event | +| POST | `/api/catalog/v1/{prefix}/events` | Query events for a catalog (Iceberg-compatible) | | GET | `/api/management/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/scan-metrics` | List scan metrics for a table | | GET | `/api/management/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/commit-metrics` | List commit metrics for a table | +> **Note:** The Events API uses POST (not GET) and follows the Iceberg REST Catalog path structure (`/api/catalog/v1/{prefix}/events`) for compatibility with the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584). The metrics APIs remain under the Polaris Management API namespace since they are Polaris-specific extensions. + ### 4.2 Path Parameters | Parameter | Type | Description | |-----------|------|-------------| -| `catalogName` | string | Name of the catalog | +| `prefix` | string | Catalog prefix (typically the catalog name) | +| `catalogName` | string | Name of the catalog (for management APIs) | | `namespace` | string | Namespace (URL-encoded, multi-level separated by `%1F`) | | `table` | string | Table name | -| `eventId` | string | Unique event identifier | - -### 4.3 Query Parameters -#### List Events (`/catalogs/{catalogName}/events`) - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `pageToken` | string | No | - | Cursor for pagination (from previous response) | -| `pageSize` | integer | No | 100 | Results per page (max: 1000) | -| `eventType` | string | No | - | Filter by event type (e.g., `AFTER_CREATE_TABLE`) | -| `resourceType` | string | No | - | Filter by resource: `CATALOG`, `NAMESPACE`, `TABLE`, `VIEW` | -| `resourceIdentifier` | string | No | - | Filter by resource identifier (exact match) | -| `principalName` | string | No | - | Filter by principal who triggered the event | -| `timestampFrom` | long | No | - | Start of time range (epoch milliseconds, inclusive) | -| `timestampTo` | long | No | - | End of time range (epoch milliseconds, exclusive) | +### 4.3 Events API (Iceberg-Compatible) + +The Events API follows the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584) for ecosystem compatibility. Key design decisions from the Iceberg spec: + +- **POST method**: Allows complex filtering with arrays and nested objects in the request body +- **Continuation token**: Opaque cursor for resumable pagination +- **Operation-centric model**: Events are structured around operations (create-table, update-table, etc.) +- **Custom extensions**: Support for `x-` prefixed custom operation types for Polaris-specific events + +#### Request Body (`QueryEventsRequest`) + +| Property | Type | Required | Description | +|----------|------|----------|-------------| +| `continuation-token` | string | No | Opaque cursor to resume fetching from previous request | +| `page-size` | integer | No | Maximum events per page (server may return fewer) | +| `after-timestamp-ms` | long | No | Filter: events after this timestamp (inclusive) | +| `operation-types` | array[string] | No | Filter by operation types (see below) | +| `catalog-objects-by-name` | array[array[string]] | No | Filter by namespace/table/view names | +| `catalog-objects-by-id` | array[object] | No | Filter by table/view UUIDs | +| `object-types` | array[string] | No | Filter by object type: `namespace`, `table`, `view` | +| `custom-filters` | object | No | Implementation-specific filter extensions | + +#### Standard Operation Types + +| Operation Type | Description | +|----------------|-------------| +| `create-table` | Table created and committed | +| `register-table` | Existing table registered in catalog | +| `drop-table` | Table dropped | +| `update-table` | Table metadata updated | +| `rename-table` | Table renamed | +| `create-view` | View created | +| `drop-view` | View dropped | +| `update-view` | View updated | +| `rename-view` | View renamed | +| `create-namespace` | Namespace created | +| `update-namespace-properties` | Namespace properties updated | +| `drop-namespace` | Namespace dropped | + +#### Polaris Custom Operation Types + +For Polaris-specific events not covered by the Iceberg spec, use the `x-` prefix convention: + +| Custom Operation Type | Description | +|----------------------|-------------| +| `x-polaris-create-catalog-role` | Catalog role created | +| `x-polaris-grant-privilege` | Privilege granted | +| `x-polaris-rotate-credentials` | Principal credentials rotated | +| `x-polaris-create-policy` | Policy created | +| `x-polaris-attach-policy` | Policy attached to resource | + +### 4.4 Query Parameters (Metrics APIs) #### List Scan Metrics (`/.../tables/{table}/scan-metrics`) @@ -160,74 +204,124 @@ Adding read-only REST endpoints enables: | `timestampFrom` | long | No | - | Start of time range (epoch ms) | | `timestampTo` | long | No | - | End of time range (epoch ms) | -### 4.4 Example Requests and Responses +### 4.5 Example Requests and Responses -#### List Events +#### Query Events (Iceberg-Compatible) **Request:** ```http -GET /api/management/v1/catalogs/my-catalog/events?pageSize=2&eventType=AFTER_CREATE_TABLE×tampFrom=1709251200000 +POST /api/catalog/v1/my-catalog/events Authorization: Bearer +Content-Type: application/json + +{ + "page-size": 2, + "operation-types": ["create-table", "update-table"], + "after-timestamp-ms": 1709251200000, + "catalog-objects-by-name": [ + ["analytics", "events"] + ], + "object-types": ["table"] +} ``` **Response:** ```json { - "nextPageToken": "eyJ0cyI6MTcwOTMzNzYxMjM0NSwiaWQiOiI1NTBlODQwMCJ9", + "next-page-token": "eyJ0cyI6MTcwOTMzNzYxMjM0NSwiaWQiOiI1NTBlODQwMCJ9", + "highest-processed-timestamp-ms": 1709337612345, "events": [ { - "eventId": "550e8400-e29b-41d4-a716-446655440000", - "catalogId": "my-catalog", - "requestId": "req-12345", - "eventType": "AFTER_CREATE_TABLE", - "timestampMs": 1709337612345, - "principalName": "admin@example.com", - "resourceType": "TABLE", - "resourceIdentifier": "analytics.events.page_views", - "additionalProperties": { - "table-uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890" + "event-id": "550e8400-e29b-41d4-a716-446655440000", + "request-id": "req-12345", + "request-event-count": 1, + "timestamp-ms": 1709337612345, + "actor": { + "principal": "admin@example.com", + "client-ip": "192.168.1.100" + }, + "operation": { + "operation-type": "create-table", + "identifier": { + "namespace": ["analytics", "events"], + "name": "page_views" + }, + "table-uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "updates": [ + {"action": "assign-uuid", "uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"}, + {"action": "set-current-schema", "schema-id": 0}, + {"action": "set-default-spec", "spec-id": 0} + ] } }, { - "eventId": "661f9511-f30c-52e5-b827-557766551111", - "catalogId": "my-catalog", - "requestId": "req-12346", - "eventType": "AFTER_CREATE_TABLE", - "timestampMs": 1709337500000, - "principalName": "etl-service@example.com", - "resourceType": "TABLE", - "resourceIdentifier": "analytics.events.user_actions", - "additionalProperties": { - "table-uuid": "b2c3d4e5-f6a7-8901-bcde-f23456789012" + "event-id": "661f9511-f30c-52e5-b827-557766551111", + "request-id": "req-12346", + "request-event-count": 1, + "timestamp-ms": 1709337500000, + "actor": { + "principal": "etl-service@example.com" + }, + "operation": { + "operation-type": "update-table", + "identifier": { + "namespace": ["analytics", "events"], + "name": "user_actions" + }, + "table-uuid": "b2c3d4e5-f6a7-8901-bcde-f23456789012", + "updates": [ + {"action": "add-snapshot", "snapshot-id": 123456789} + ], + "requirements": [ + {"type": "assert-table-uuid", "uuid": "b2c3d4e5-f6a7-8901-bcde-f23456789012"} + ] } } ] } ``` -#### Get Single Event +#### Query Events with Custom Polaris Operations **Request:** ```http -GET /api/management/v1/catalogs/my-catalog/events/550e8400-e29b-41d4-a716-446655440000 +POST /api/catalog/v1/my-catalog/events Authorization: Bearer +Content-Type: application/json + +{ + "page-size": 10, + "operation-types": ["x-polaris-grant-privilege", "x-polaris-rotate-credentials"] +} ``` **Response:** ```json { - "eventId": "550e8400-e29b-41d4-a716-446655440000", - "catalogId": "my-catalog", - "requestId": "req-12345", - "eventType": "AFTER_CREATE_TABLE", - "timestampMs": 1709337612345, - "principalName": "admin@example.com", - "resourceType": "TABLE", - "resourceIdentifier": "analytics.events.page_views", - "additionalProperties": { - "table-uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", - "traceparent": "00-abc123def456789012345678901234-def456789012-01" - } + "next-page-token": "eyJ0cyI6MTcwOTMzODAwMDAwMH0=", + "highest-processed-timestamp-ms": 1709338000000, + "events": [ + { + "event-id": "772f0622-g41d-63f6-c938-668877662222", + "request-id": "req-admin-001", + "request-event-count": 1, + "timestamp-ms": 1709338000000, + "actor": { + "principal": "security-admin@example.com" + }, + "operation": { + "operation-type": "custom", + "custom-type": "x-polaris-grant-privilege", + "identifier": { + "namespace": ["analytics", "events"], + "name": "page_views" + }, + "table-uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "privilege": "TABLE_READ_DATA", + "grantee": "data-analyst-role" + } + } + ] } ``` @@ -358,97 +452,69 @@ If finer-grained control is desired, new privileges could be introduced: ## 6. OpenAPI Schema -Add the following to `spec/polaris-management-service.yml`: +### 6.1 Events API (Iceberg REST Catalog Extension) -### 6.1 Paths +Add the following to `spec/rest-catalog-open-api.yaml` (aligned with Iceberg Events API spec): ```yaml paths: - /catalogs/{catalogName}/events: + /v1/{prefix}/events: parameters: - - $ref: '#/components/parameters/catalogName' - get: - operationId: listCatalogEvents - summary: List events for a catalog - description: Returns a paginated list of events with optional filtering + - $ref: '#/components/parameters/prefix' + post: tags: - - Observability - parameters: - - name: pageToken - in: query - schema: - type: string - - name: pageSize - in: query - schema: - type: integer - minimum: 1 - maximum: 1000 - default: 100 - - name: eventType - in: query - schema: - type: string - - name: resourceType - in: query - schema: - type: string - enum: [CATALOG, NAMESPACE, TABLE, VIEW] - - name: resourceIdentifier - in: query - schema: - type: string - - name: principalName - in: query - schema: - type: string - - name: timestampFrom - in: query - schema: - type: integer - format: int64 - - name: timestampTo - in: query - schema: - type: integer - format: int64 + - Catalog API + summary: Get events for changes to catalog objects + description: > + Returns a sequence of changes to catalog objects (tables, namespaces, views) + that allows clients to efficiently track metadata modifications without polling + individual resources. Consumers track their progress through a continuation-token, + enabling resumable synchronization after downtime or errors. + + This endpoint primarily supports use cases like catalog federation, workflow + triggering, and basic audit capabilities. + + Consumers should be prepared to handle 410 Gone responses when requested sequences + are outside the server's retention window. Consumers should also de-duplicate + received events based on the event's `event-id`. + operationId: getEvents + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/QueryEventsRequest' responses: '200': - description: Paginated list of events + description: A sequence of change events to catalog objects content: application/json: schema: - $ref: '#/components/schemas/ListEventsResponse' + $ref: '#/components/schemas/QueryEventsResponse' + '400': + $ref: '#/components/responses/BadRequestErrorResponse' + '401': + $ref: '#/components/responses/UnauthorizedResponse' '403': - description: Insufficient privileges - '404': - description: Catalog not found - - /catalogs/{catalogName}/events/{eventId}: - parameters: - - $ref: '#/components/parameters/catalogName' - - name: eventId - in: path - required: true - schema: - type: string - get: - operationId: getEvent - summary: Get a specific event - tags: - - Observability - responses: - '200': - description: The requested event + $ref: '#/components/responses/ForbiddenResponse' + '410': + description: Gone - The requested offset is no longer available content: application/json: schema: - $ref: '#/components/schemas/CatalogEvent' - '403': - description: Insufficient privileges - '404': - description: Event or catalog not found + $ref: '#/components/schemas/ErrorModel' + '503': + $ref: '#/components/responses/ServiceUnavailableResponse' + '5XX': + $ref: '#/components/responses/ServerErrorResponse' +``` + +### 6.2 Metrics APIs (Polaris Management Service) + +Add the following to `spec/polaris-management-service.yml`: +```yaml +paths: /catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/scan-metrics: parameters: - $ref: '#/components/parameters/catalogName' @@ -560,63 +626,215 @@ paths: description: Table not found ``` -### 6.2 Schemas +### 6.3 Events API Schemas (Iceberg-Compatible) + +Add these schemas to `spec/rest-catalog-open-api.yaml`: ```yaml components: schemas: - CatalogEvent: + QueryEventsRequest: type: object - required: - - eventId - - catalogId - - eventType - - timestampMs - - resourceType - - resourceIdentifier properties: - eventId: + continuation-token: type: string - description: Unique event identifier - catalogId: + description: > + A continuation token to resume fetching events from a previous request. + If not provided, events are fetched from the beginning of the event log + subject to other filters. + page-size: + type: integer + format: int32 + description: > + The maximum number of events to return in a single response. + Servers may return less results than requested. + after-timestamp-ms: + type: integer + format: int64 + description: > + The timestamp in milliseconds to start consuming events from (inclusive). + operation-types: + type: array + items: + $ref: "#/components/schemas/OperationType" + description: Filter events by operation type. + catalog-objects-by-name: + type: array + items: + $ref: "#/components/schemas/CatalogObjectIdentifier" + description: > + Filter events by catalog objects referenced by name (namespaces, tables, views). + For namespaces, events for all containing objects are returned recursively. + catalog-objects-by-id: + type: array + items: + $ref: "#/components/schemas/CatalogObjectUuid" + description: Filter events by table/view UUIDs. + object-types: + type: array + items: + type: string + enum: [namespace, table, view] + description: Filter events by catalog object type. + custom-filters: + type: object + additionalProperties: true + description: Implementation-specific filter extensions. + + QueryEventsResponse: + type: object + required: + - next-page-token + - highest-processed-timestamp-ms + - events + properties: + next-page-token: type: string - description: Catalog where the event occurred - requestId: + description: > + An opaque continuation token to fetch the next page of events. + highest-processed-timestamp-ms: + type: integer + format: int64 + description: > + The highest event timestamp processed when generating this response. + events: + type: array + items: + $ref: "#/components/schemas/Event" + + Event: + type: object + required: + - event-id + - request-id + - request-event-count + - timestamp-ms + - operation + properties: + event-id: type: string - description: Request ID that triggered this event - eventType: + description: Unique ID of this event. Clients should deduplicate based on this ID. + request-id: type: string - description: Event type (e.g., AFTER_CREATE_TABLE) - timestampMs: + description: > + Opaque ID of the request this event belongs to. Events from the same + request share this ID. + request-event-count: + type: integer + description: > + Total number of events generated by this request. + timestamp-ms: type: integer format: int64 - description: Event timestamp (epoch milliseconds) - principalName: + description: Timestamp when this event occurred (epoch milliseconds). + actor: + type: object + additionalProperties: true + description: > + The actor who performed the operation (e.g., user, service account). + Content is implementation-specific. + operation: + type: object + description: The operation that was performed. + discriminator: + propertyName: operation-type + mapping: + create-table: "#/components/schemas/CreateTableOperation" + register-table: "#/components/schemas/RegisterTableOperation" + drop-table: "#/components/schemas/DropTableOperation" + update-table: "#/components/schemas/UpdateTableOperation" + rename-table: "#/components/schemas/RenameTableOperation" + create-view: "#/components/schemas/CreateViewOperation" + drop-view: "#/components/schemas/DropViewOperation" + update-view: "#/components/schemas/UpdateViewOperation" + rename-view: "#/components/schemas/RenameViewOperation" + create-namespace: "#/components/schemas/CreateNamespaceOperation" + update-namespace-properties: "#/components/schemas/UpdateNamespacePropertiesOperation" + drop-namespace: "#/components/schemas/DropNamespaceOperation" + custom: "#/components/schemas/CustomOperation" + + OperationType: + type: string + description: > + Defines the type of operation. Clients should ignore unknown operation types. + anyOf: + - type: string + enum: + - create-table + - register-table + - drop-table + - update-table + - rename-table + - create-view + - drop-view + - update-view + - rename-view + - create-namespace + - update-namespace-properties + - drop-namespace + - $ref: '#/components/schemas/CustomOperationType' + + CustomOperationType: + type: string + description: > + Custom operation type for catalog-specific extensions. + Must start with 'x-' followed by an implementation-specific identifier. + pattern: '^x-[a-zA-Z0-9-_.]+$' + + CustomOperation: + type: object + description: Extension point for catalog-specific operations (e.g., Polaris privileges). + required: + - operation-type + - custom-type + properties: + operation-type: type: string - description: Principal who triggered the event - resourceType: + const: "custom" + custom-type: + $ref: '#/components/schemas/CustomOperationType' + identifier: + $ref: "#/components/schemas/TableIdentifier" + description: Table or view identifier this operation applies to, if applicable. + namespace: + $ref: "#/components/schemas/Namespace" + description: Namespace this operation applies to, if applicable. + table-uuid: type: string - enum: [CATALOG, NAMESPACE, TABLE, VIEW] - resourceIdentifier: + format: uuid + view-uuid: type: string - description: Fully qualified resource identifier - additionalProperties: - type: object - additionalProperties: - type: string - description: Event-specific metadata + format: uuid + additionalProperties: true - ListEventsResponse: + CatalogObjectIdentifier: + type: array + items: + type: string + description: Reference to a named object in the catalog (namespace, table, or view). + example: ["accounting", "tax"] + + CatalogObjectUuid: type: object + required: + - uuid + - type properties: - nextPageToken: + uuid: type: string - description: Token for next page (null if no more results) - events: - type: array - items: - $ref: '#/components/schemas/CatalogEvent' + description: The UUID of the catalog object. + type: + type: string + enum: [table, view] +``` + +### 6.4 Metrics API Schemas (Polaris Management Service) + +Add these schemas to `spec/polaris-management-service.yml`: +```yaml +components: + schemas: ScanMetricsReport: type: object required: @@ -870,9 +1088,12 @@ CREATE INDEX IF NOT EXISTS idx_commit_report_lookup | File | Changes | |------|---------| -| `spec/polaris-management-service.yml` | Add paths and schemas | -| `api/management-service/` | Generated API interfaces | -| `runtime/service/.../admin/` | Service implementation | +| `spec/rest-catalog-open-api.yaml` | Add Events API paths and schemas (Iceberg-compatible) | +| `spec/polaris-management-service.yml` | Add Metrics API paths and schemas | +| `api/iceberg-service/` | Generated Events API interfaces | +| `api/management-service/` | Generated Metrics API interfaces | +| `runtime/service/.../catalog/` | Events service implementation | +| `runtime/service/.../admin/` | Metrics service implementation | | `polaris-core/.../persistence/BasePersistence.java` | Add read methods | | `persistence/relational-jdbc/.../JdbcBasePersistenceImpl.java` | Query implementations | @@ -889,21 +1110,100 @@ Internal format (base64-encoded JSON, opaque to clients): --- -## Appendix: Event Types Reference +## Appendix A: Polaris Internal Event Types Reference + +Polaris internal event types are categorized by code ranges. These are mapped to Iceberg-compatible operation types when exposed via the Events API: + +| Range | Category | Internal Event Type | Iceberg Operation Type | +|-------|----------|---------------------|------------------------| +| 100-109 | Catalog | `AFTER_CREATE_CATALOG` | `custom` (`x-polaris-create-catalog`) | +| | | `AFTER_DELETE_CATALOG` | `custom` (`x-polaris-delete-catalog`) | +| 200-217 | Catalog Role | `AFTER_CREATE_CATALOG_ROLE` | `custom` (`x-polaris-create-catalog-role`) | +| | | `AFTER_ADD_GRANT_TO_CATALOG_ROLE` | `custom` (`x-polaris-grant-privilege`) | +| 300-319 | Principal | `AFTER_CREATE_PRINCIPAL` | `custom` (`x-polaris-create-principal`) | +| | | `AFTER_ROTATE_CREDENTIALS` | `custom` (`x-polaris-rotate-credentials`) | +| 400-417 | Principal Role | `AFTER_CREATE_PRINCIPAL_ROLE` | `custom` (`x-polaris-create-principal-role`) | +| 500-511 | Namespace | `AFTER_CREATE_NAMESPACE` | `create-namespace` | +| | | `AFTER_UPDATE_NAMESPACE_PROPERTIES` | `update-namespace-properties` | +| | | `AFTER_DROP_NAMESPACE` | `drop-namespace` | +| 600-617 | Table | `AFTER_CREATE_TABLE` | `create-table` | +| | | `AFTER_UPDATE_TABLE` | `update-table` | +| | | `AFTER_DROP_TABLE` | `drop-table` | +| | | `AFTER_RENAME_TABLE` | `rename-table` | +| | | `AFTER_REGISTER_TABLE` | `register-table` | +| 700-715 | View | `AFTER_CREATE_VIEW` | `create-view` | +| | | `AFTER_UPDATE_VIEW` | `update-view` | +| | | `AFTER_DROP_VIEW` | `drop-view` | +| | | `AFTER_RENAME_VIEW` | `rename-view` | +| 1200-1215 | Policy | `AFTER_CREATE_POLICY` | `custom` (`x-polaris-create-policy`) | +| | | `AFTER_ATTACH_POLICY` | `custom` (`x-polaris-attach-policy`) | +| 1300-1307 | Generic Table | `AFTER_CREATE_GENERIC_TABLE` | `custom` (`x-polaris-create-generic-table`) | -Events are categorized by code ranges: +--- -| Range | Category | Examples | -|-------|----------|----------| -| 100-109 | Catalog | `AFTER_CREATE_CATALOG`, `AFTER_DELETE_CATALOG` | -| 200-217 | Catalog Role | `AFTER_CREATE_CATALOG_ROLE`, `AFTER_ADD_GRANT_TO_CATALOG_ROLE` | -| 300-319 | Principal | `AFTER_CREATE_PRINCIPAL`, `AFTER_ROTATE_CREDENTIALS` | -| 400-417 | Principal Role | `AFTER_CREATE_PRINCIPAL_ROLE`, `AFTER_ASSIGN_CATALOG_ROLE_TO_PRINCIPAL_ROLE` | -| 500-511 | Namespace | `AFTER_CREATE_NAMESPACE`, `AFTER_DROP_NAMESPACE` | -| 600-617 | Table | `AFTER_CREATE_TABLE`, `AFTER_UPDATE_TABLE`, `AFTER_DROP_TABLE` | -| 700-715 | View | `AFTER_CREATE_VIEW`, `AFTER_REPLACE_VIEW` | -| 1200-1215 | Policy | `AFTER_CREATE_POLICY`, `AFTER_ATTACH_POLICY` | -| 1300-1307 | Generic Table | `AFTER_CREATE_GENERIC_TABLE` | +## 8. Iceberg Events API Alignment + +This section documents the alignment with the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584) and explains the rationale for design decisions. + +### 8.1 Why Align with Iceberg Events API? + +The Iceberg Events API is an emerging specification that is nearing consensus in the Apache Iceberg community. Aligning Polaris with this specification provides: + +1. **Ecosystem Compatibility**: Clients built for the Iceberg Events API will work with Polaris without modification +2. **Future-Proofing**: Avoids breaking changes when the Iceberg spec is finalized +3. **Tooling Interoperability**: Monitoring tools, federation services, and workflow triggers can work across Iceberg-compatible catalogs +4. **Reduced Cognitive Load**: Developers familiar with Iceberg don't need to learn a new API + +### 8.2 Key Design Decisions from Iceberg Spec + +| Decision | Iceberg Spec Approach | Rationale | +|----------|----------------------|-----------| +| **HTTP Method** | `POST` (not `GET`) | Allows complex filtering with arrays and nested objects in request body | +| **API Path** | `/v1/{prefix}/events` | Part of Iceberg REST Catalog, not a separate management API | +| **Pagination** | `continuation-token` | Opaque cursor that encodes server state; resumable after downtime | +| **Event Structure** | Operation-centric with discriminator | Each event contains a typed `operation` with operation-specific fields | +| **Operation Types** | Standardized enum + `x-` prefix extensions | Standard types for Iceberg operations; custom prefix for catalog-specific extensions | +| **Actor Field** | Generic object (implementation-specific) | Flexibility for different auth models (users, service accounts, etc.) | +| **Error Handling** | `410 Gone` for expired offsets | Explicit signal when continuation token is outside retention window | + +### 8.3 Polaris-Specific Extensions + +Polaris extends the Iceberg Events API using the `custom` operation type with `x-polaris-*` prefixed custom types: + +| Custom Type | Polaris Event | Description | +|-------------|---------------|-------------| +| `x-polaris-create-catalog` | `AFTER_CREATE_CATALOG` | Catalog created | +| `x-polaris-create-catalog-role` | `AFTER_CREATE_CATALOG_ROLE` | Catalog role created | +| `x-polaris-grant-privilege` | `AFTER_ADD_GRANT_TO_CATALOG_ROLE` | Privilege granted to role | +| `x-polaris-revoke-privilege` | `AFTER_REMOVE_GRANT_FROM_CATALOG_ROLE` | Privilege revoked | +| `x-polaris-create-principal` | `AFTER_CREATE_PRINCIPAL` | Principal created | +| `x-polaris-rotate-credentials` | `AFTER_ROTATE_CREDENTIALS` | Credentials rotated | +| `x-polaris-create-policy` | `AFTER_CREATE_POLICY` | Policy created | +| `x-polaris-attach-policy` | `AFTER_ATTACH_POLICY` | Policy attached to resource | + +### 8.4 Mapping Polaris Internal Events to Iceberg Operations + +| Polaris Event Type | Iceberg Operation Type | +|-------------------|------------------------| +| `AFTER_CREATE_TABLE` | `create-table` | +| `AFTER_UPDATE_TABLE` | `update-table` | +| `AFTER_DROP_TABLE` | `drop-table` | +| `AFTER_RENAME_TABLE` | `rename-table` | +| `AFTER_REGISTER_TABLE` | `register-table` | +| `AFTER_CREATE_VIEW` | `create-view` | +| `AFTER_UPDATE_VIEW` / `AFTER_REPLACE_VIEW` | `update-view` | +| `AFTER_DROP_VIEW` | `drop-view` | +| `AFTER_RENAME_VIEW` | `rename-view` | +| `AFTER_CREATE_NAMESPACE` | `create-namespace` | +| `AFTER_UPDATE_NAMESPACE_PROPERTIES` | `update-namespace-properties` | +| `AFTER_DROP_NAMESPACE` | `drop-namespace` | +| Other Polaris events | `custom` with `x-polaris-*` type | + +### 8.5 References + +- **Iceberg Events API Proposal**: [Google Doc](https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit) +- **Iceberg Events API PR**: [apache/iceberg#12584](https://github.com/apache/iceberg/pull/12584) +- **Iceberg REST Catalog Spec**: [rest-catalog-open-api.yaml](https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml) --- @@ -911,6 +1211,8 @@ Events are categorized by code ranges: 1. **Aggregations**: Are aggregated metrics views needed (e.g., daily summaries)? 2. **Privileges**: Use existing privileges or introduce new `READ_EVENTS`/`READ_METRICS`? +3. **Event Retention**: What is the default retention period for events? Should it be configurable? +4. **Consistency Guarantees**: What ordering and delivery guarantees should Polaris provide for the Events API? --- @@ -919,4 +1221,6 @@ Events are categorized by code ranges: - Database schema: `persistence/relational-jdbc/src/main/resources/postgres/schema-v4.sql` - Event types: `runtime/service/src/main/java/org/apache/polaris/service/events/PolarisEventType.java` - Metrics persistence: `runtime/service/src/main/java/org/apache/polaris/service/reporting/PersistingMetricsReporter.java` +- Iceberg Events API PR: https://github.com/apache/iceberg/pull/12584 +- Iceberg Events API Design Doc: https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit From 843da3e79f5d518d5612032cc0e2ff57961b1133 Mon Sep 17 00:00:00 2001 From: Anand Kumar Sankaran Date: Tue, 3 Mar 2026 13:16:56 -0800 Subject: [PATCH 03/15] Address authorization review feedback: introduce dedicated privileges Updates the Authorization section to address separation of duties concerns: - Replace CATALOG_MANAGE_METADATA with new CATALOG_READ_EVENTS privilege - Replace TABLE_READ_DATA with new TABLE_READ_METRICS privilege - Add detailed rationale for separation of duties - Document privilege hierarchy (higher privileges imply lower ones) - Add implementation notes for new privileges This ensures: - Read-only audit access does not require management permissions - Monitoring tools can access metrics without requiring data read access - Fine-grained access control is possible for different operational roles --- .../proposals/observability-rest-api.md | 73 ++++++++++++++----- 1 file changed, 55 insertions(+), 18 deletions(-) diff --git a/site/content/in-dev/unreleased/proposals/observability-rest-api.md b/site/content/in-dev/unreleased/proposals/observability-rest-api.md index 42b7fca2a1..4d140a96ef 100644 --- a/site/content/in-dev/unreleased/proposals/observability-rest-api.md +++ b/site/content/in-dev/unreleased/proposals/observability-rest-api.md @@ -427,26 +427,59 @@ Authorization: Bearer ### 5.1 Required Privileges -| Endpoint | Required Privilege | Scope | -|----------|-------------------|-------| -| List/Get Events | `CATALOG_MANAGE_METADATA` | Catalog | -| List Scan Metrics | `TABLE_READ_DATA` | Table | -| List Commit Metrics | `TABLE_READ_DATA` | Table | +This proposal introduces **new dedicated privileges** for reading observability data, following the principle of **separation of duties**. This ensures that: -### 5.2 Rationale +- Read-only audit/monitoring access does not require management permissions +- Monitoring tools can access metrics without requiring data read access +- Fine-grained access control is possible for different operational roles -- **Events** contain catalog-wide audit information and should require catalog-level administrative access -- **Metrics** are table-specific and align with read access since they describe query patterns and commit history -- This follows the principle of least privilege while enabling common use cases +| Endpoint | Required Privilege | Scope | New Privilege? | +|----------|-------------------|-------|----------------| +| Query Events | `CATALOG_READ_EVENTS` | Catalog | **Yes** | +| List Scan Metrics | `TABLE_READ_METRICS` | Table | **Yes** | +| List Commit Metrics | `TABLE_READ_METRICS` | Table | **Yes** | -### 5.3 Alternative: New Privileges +### 5.2 New Privilege Definitions -If finer-grained control is desired, new privileges could be introduced: +| Privilege | Scope | Description | +|-----------|-------|-------------| +| `CATALOG_READ_EVENTS` | Catalog | Read-only access to catalog events (audit log). Does not grant any management capabilities. | +| `TABLE_READ_METRICS` | Table | Read-only access to table scan and commit metrics. Does not grant access to table data. | -| New Privilege | Description | -|---------------|-------------| -| `CATALOG_READ_EVENTS` | Read-only access to catalog events | -| `TABLE_READ_METRICS` | Read-only access to table metrics | +### 5.3 Rationale: Separation of Duties + +Introducing dedicated read-only privileges enables proper **separation of duties**: + +| Use Case | Required Privilege | Why Not Reuse Existing? | +|----------|-------------------|------------------------| +| Security auditor reviewing catalog changes | `CATALOG_READ_EVENTS` | Should not require `CATALOG_MANAGE_METADATA` (management access) | +| Monitoring tool collecting table metrics | `TABLE_READ_METRICS` | Should not require `TABLE_READ_DATA` (data access) | +| Data analyst with table access | `TABLE_READ_DATA` implies `TABLE_READ_METRICS` | Users who can read data can also see metrics about their queries | +| Catalog admin | `CATALOG_MANAGE_METADATA` implies `CATALOG_READ_EVENTS` | Admins can see all events | + +### 5.4 Privilege Hierarchy + +The new privileges fit into the existing hierarchy as follows: + +``` +CATALOG_MANAGE_METADATA + └── CATALOG_READ_EVENTS (implied) + +TABLE_FULL_METADATA / TABLE_READ_DATA + └── TABLE_READ_METRICS (implied) +``` + +This means: +- Users with `CATALOG_MANAGE_METADATA` automatically have `CATALOG_READ_EVENTS` +- Users with `TABLE_READ_DATA` automatically have `TABLE_READ_METRICS` +- But the reverse is **not** true: `CATALOG_READ_EVENTS` does not grant management access, and `TABLE_READ_METRICS` does not grant data access + +### 5.5 Implementation Notes + +New privileges require: +1. Adding entries to `PolarisPrivilege` enum +2. Updating the privilege hierarchy in the authorizer +3. Adding privilege checks in the new API endpoints --- @@ -1210,9 +1243,13 @@ Polaris extends the Iceberg Events API using the `custom` operation type with `x ## Open Questions 1. **Aggregations**: Are aggregated metrics views needed (e.g., daily summaries)? -2. **Privileges**: Use existing privileges or introduce new `READ_EVENTS`/`READ_METRICS`? -3. **Event Retention**: What is the default retention period for events? Should it be configurable? -4. **Consistency Guarantees**: What ordering and delivery guarantees should Polaris provide for the Events API? +2. **Event Retention**: What is the default retention period for events? Should it be configurable? +3. **Consistency Guarantees**: What ordering and delivery guarantees should Polaris provide for the Events API? + +## Resolved Questions + +1. ~~**Privileges**: Use existing privileges or introduce new `READ_EVENTS`/`READ_METRICS`?~~ + - **Resolution**: Introduce new dedicated privileges (`CATALOG_READ_EVENTS`, `TABLE_READ_METRICS`) to support separation of duties. See [Section 5](#5-authorization) for details. --- From 155ac3b6a6bd7a77428bd8113b4e5809b366d30a Mon Sep 17 00:00:00 2001 From: Anand Kumar Sankaran Date: Tue, 3 Mar 2026 13:21:17 -0800 Subject: [PATCH 04/15] Add note about extracting OpenAPI specs to separate files Address reviewer feedback to clarify that OpenAPI YAML should be in separate files upon approval for ease of processing. --- .../in-dev/unreleased/proposals/observability-rest-api.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/site/content/in-dev/unreleased/proposals/observability-rest-api.md b/site/content/in-dev/unreleased/proposals/observability-rest-api.md index 4d140a96ef..2e9833665d 100644 --- a/site/content/in-dev/unreleased/proposals/observability-rest-api.md +++ b/site/content/in-dev/unreleased/proposals/observability-rest-api.md @@ -485,6 +485,10 @@ New privileges require: ## 6. OpenAPI Schema +> **Note:** The OpenAPI specifications below are embedded in this proposal for review context. Upon approval, these should be extracted into separate files for ease of processing and proper integration: +> - **Events API** → `spec/rest-catalog-open-api.yaml` (extending Iceberg REST Catalog spec) +> - **Metrics API** → `spec/polaris-management-service.yml` (Polaris Management Service spec) + ### 6.1 Events API (Iceberg REST Catalog Extension) Add the following to `spec/rest-catalog-open-api.yaml` (aligned with Iceberg Events API spec): From d790fbf45707ed7734444aeeec3fe255fe4d515e Mon Sep 17 00:00:00 2001 From: Anand Kumar Sankaran Date: Tue, 3 Mar 2026 13:41:44 -0800 Subject: [PATCH 05/15] Change metrics API path from /api/management/ to /api/metrics-reports/ Address reviewer feedback that metrics reports are read-only access to pre-populated data, not catalog management operations: - Change base path from /api/management/v1/ to /api/metrics-reports/v1/ - Update all example requests with new path - Specify new spec file: spec/metrics-reports-service.yml - Update Files to Modify section with new service locations - Add rationale explaining separation from management API This allows servers that don't support catalog management to still expose metrics reports for monitoring and observability. --- .../proposals/observability-rest-api.md | 30 ++++++++++--------- 1 file changed, 16 insertions(+), 14 deletions(-) diff --git a/site/content/in-dev/unreleased/proposals/observability-rest-api.md b/site/content/in-dev/unreleased/proposals/observability-rest-api.md index 2e9833665d..3e89487bf6 100644 --- a/site/content/in-dev/unreleased/proposals/observability-rest-api.md +++ b/site/content/in-dev/unreleased/proposals/observability-rest-api.md @@ -98,10 +98,10 @@ Adding read-only REST endpoints enables: | Principle | Rationale | |-----------|-----------| | **Iceberg Events API alignment** | Events API follows the [Iceberg Events API spec](https://github.com/apache/iceberg/pull/12584) for ecosystem compatibility | -| **Management API namespace** | Metrics APIs use `/api/management/v1/...` to separate from Iceberg REST Catalog paths | +| **Dedicated metrics-reports namespace** | Metrics APIs use `/api/metrics-reports/v1/...` to separate from management and catalog APIs | | **POST for complex filtering** | Events API uses POST with request body (per Iceberg spec) to support complex filters (arrays, nested objects) | | **Read-only semantics** | All endpoints are read-only; metrics/events are written via existing flows | -| **Consistent pagination** | Follow `continuation-token` pattern (Iceberg) and `pageToken` pattern (Polaris management APIs) | +| **Consistent pagination** | Follow `continuation-token` pattern (Iceberg) and `pageToken` pattern (Polaris APIs) | | **Flexible filtering** | Time ranges, operation types, catalog objects - common query patterns | | **RBAC integration** | Leverage existing Polaris authorization model | @@ -114,17 +114,17 @@ Adding read-only REST endpoints enables: | Method | Path | Description | |--------|------|-------------| | POST | `/api/catalog/v1/{prefix}/events` | Query events for a catalog (Iceberg-compatible) | -| GET | `/api/management/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/scan-metrics` | List scan metrics for a table | -| GET | `/api/management/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/commit-metrics` | List commit metrics for a table | +| GET | `/api/metrics-reports/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/scan-metrics` | List scan metrics for a table | +| GET | `/api/metrics-reports/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/commit-metrics` | List commit metrics for a table | -> **Note:** The Events API uses POST (not GET) and follows the Iceberg REST Catalog path structure (`/api/catalog/v1/{prefix}/events`) for compatibility with the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584). The metrics APIs remain under the Polaris Management API namespace since they are Polaris-specific extensions. +> **Note:** The Events API uses POST (not GET) and follows the Iceberg REST Catalog path structure (`/api/catalog/v1/{prefix}/events`) for compatibility with the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584). The metrics APIs use a dedicated `/api/metrics-reports/v1/` namespace since they expose pre-populated records rather than managing catalog state - a server that doesn't support catalog management may still expose metrics reports. ### 4.2 Path Parameters | Parameter | Type | Description | |-----------|------|-------------| | `prefix` | string | Catalog prefix (typically the catalog name) | -| `catalogName` | string | Name of the catalog (for management APIs) | +| `catalogName` | string | Name of the catalog | | `namespace` | string | Namespace (URL-encoded, multi-level separated by `%1F`) | | `table` | string | Table name | @@ -329,7 +329,7 @@ Content-Type: application/json **Request:** ```http -GET /api/management/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views/scan-metrics?pageSize=2×tampFrom=1709251200000 +GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views/scan-metrics?pageSize=2×tampFrom=1709251200000 Authorization: Bearer ``` @@ -377,7 +377,7 @@ Authorization: Bearer **Request:** ```http -GET /api/management/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views/commit-metrics?operation=append&pageSize=2 +GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views/commit-metrics?operation=append&pageSize=2 Authorization: Bearer ``` @@ -487,7 +487,7 @@ New privileges require: > **Note:** The OpenAPI specifications below are embedded in this proposal for review context. Upon approval, these should be extracted into separate files for ease of processing and proper integration: > - **Events API** → `spec/rest-catalog-open-api.yaml` (extending Iceberg REST Catalog spec) -> - **Metrics API** → `spec/polaris-management-service.yml` (Polaris Management Service spec) +> - **Metrics Reports API** → `spec/metrics-reports-service.yml` (new dedicated service spec with base path `/api/metrics-reports/v1/`) ### 6.1 Events API (Iceberg REST Catalog Extension) @@ -546,9 +546,11 @@ paths: $ref: '#/components/responses/ServerErrorResponse' ``` -### 6.2 Metrics APIs (Polaris Management Service) +### 6.2 Metrics APIs (New Metrics Reports Service) -Add the following to `spec/polaris-management-service.yml`: +Add the following to a new `spec/metrics-reports-service.yml` (or extend existing management service): + +> **Note:** The metrics APIs use `/api/metrics-reports/v1/` as the base path, separate from the management API. This reflects that metrics reports are read-only access to pre-populated data, not catalog management operations. ```yaml paths: @@ -1126,11 +1128,11 @@ CREATE INDEX IF NOT EXISTS idx_commit_report_lookup | File | Changes | |------|---------| | `spec/rest-catalog-open-api.yaml` | Add Events API paths and schemas (Iceberg-compatible) | -| `spec/polaris-management-service.yml` | Add Metrics API paths and schemas | +| `spec/metrics-reports-service.yml` | **New file** - Metrics Reports API paths and schemas | | `api/iceberg-service/` | Generated Events API interfaces | -| `api/management-service/` | Generated Metrics API interfaces | +| `api/metrics-reports-service/` | **New** - Generated Metrics Reports API interfaces | | `runtime/service/.../catalog/` | Events service implementation | -| `runtime/service/.../admin/` | Metrics service implementation | +| `runtime/service/.../metrics/` | **New** - Metrics reports service implementation | | `polaris-core/.../persistence/BasePersistence.java` | Add read methods | | `persistence/relational-jdbc/.../JdbcBasePersistenceImpl.java` | Query implementations | From a654d979a0ebc484f0ec8c82311ab157a7871943 Mon Sep 17 00:00:00 2001 From: Anand Kumar Sankaran Date: Tue, 3 Mar 2026 14:29:34 -0800 Subject: [PATCH 06/15] docs: Add PolarisEventType to Iceberg Events API mapping section Address review comment from nandorKollar regarding how Polaris internal event types (BEFORE_*/AFTER_*) map to Iceberg Events API operation-types. New section 7.5 covers: - Design decision: Only AFTER_* events exposed (completed operations) - Complete mapping table from PolarisEventType to operation-type - Standard Iceberg ops map directly, Polaris-specific use x-polaris-* prefix - Read-only operations (GET/LIST) are excluded - Implementation guidance with event filtering code sample --- .../proposals/observability-rest-api.md | 100 ++++++++++++++++++ 1 file changed, 100 insertions(+) diff --git a/site/content/in-dev/unreleased/proposals/observability-rest-api.md b/site/content/in-dev/unreleased/proposals/observability-rest-api.md index 3e89487bf6..d545715d66 100644 --- a/site/content/in-dev/unreleased/proposals/observability-rest-api.md +++ b/site/content/in-dev/unreleased/proposals/observability-rest-api.md @@ -1147,6 +1147,106 @@ Internal format (base64-encoded JSON, opaque to clients): } ``` +### 7.5 Mapping PolarisEventType to Iceberg Events API + +Polaris internally uses a `PolarisEventType` enum that distinguishes between `BEFORE_*` and `AFTER_*` events for each operation (e.g., `BEFORE_CREATE_TABLE` and `AFTER_CREATE_TABLE`). This section explains how these internal events map to the Iceberg Events API. + +#### 7.5.1 Design Decision: Only AFTER Events are Exposed + +The Iceberg Events API represents **completed operations** that have been committed to the catalog. Therefore: + +| Internal Event Pattern | Exposed via API? | Rationale | +|------------------------|------------------|-----------| +| `AFTER_*` events | **Yes** | Represent successful, committed operations | +| `BEFORE_*` events | **No** | Represent intent, not outcome; may fail after firing | + +**Why not expose BEFORE events?** + +1. **Semantic mismatch**: The Iceberg Events API is designed for change data capture (CDC) and audit logs of *completed* changes. `BEFORE_*` events fire before validation and persistence, so they may represent operations that ultimately fail. + +2. **Consistency**: Exposing `BEFORE_*` events could lead to consumers seeing "phantom" operations that never actually occurred. + +3. **Use case alignment**: The primary use cases (audit, federation, workflow triggers) all require knowing what *actually happened*, not what was *attempted*. + +4. **Internal vs external**: `BEFORE_*` events serve internal purposes (request filtering, rate limiting, pre-validation hooks) and are not meaningful to external consumers. + +#### 7.5.2 Mapping AFTER Events to Operation Types + +The mapping follows a straightforward pattern - the `AFTER_` prefix is stripped and the remaining name maps to the Iceberg operation type: + +| Polaris `PolarisEventType` | Iceberg `operation-type` | Notes | +|----------------------------|--------------------------|-------| +| **Standard Iceberg Operations** | | | +| `AFTER_CREATE_TABLE` | `create-table` | Direct mapping | +| `AFTER_REGISTER_TABLE` | `register-table` | Direct mapping | +| `AFTER_DROP_TABLE` | `drop-table` | Direct mapping | +| `AFTER_UPDATE_TABLE` | `update-table` | Includes schema evolution, property changes | +| `AFTER_RENAME_TABLE` | `rename-table` | Direct mapping | +| `AFTER_CREATE_VIEW` | `create-view` | Direct mapping | +| `AFTER_DROP_VIEW` | `drop-view` | Direct mapping | +| `AFTER_REPLACE_VIEW` | `update-view` | View replacement maps to update | +| `AFTER_RENAME_VIEW` | `rename-view` | Direct mapping | +| `AFTER_CREATE_NAMESPACE` | `create-namespace` | Direct mapping | +| `AFTER_UPDATE_NAMESPACE_PROPERTIES` | `update-namespace-properties` | Direct mapping | +| `AFTER_DROP_NAMESPACE` | `drop-namespace` | Direct mapping | +| **Polaris Custom Operations** | | Use `custom` type with `x-polaris-*` | +| `AFTER_CREATE_CATALOG` | `custom` (`x-polaris-create-catalog`) | Catalog-level, not in Iceberg spec | +| `AFTER_DELETE_CATALOG` | `custom` (`x-polaris-delete-catalog`) | Catalog-level | +| `AFTER_CREATE_PRINCIPAL` | `custom` (`x-polaris-create-principal`) | Access management | +| `AFTER_DELETE_PRINCIPAL` | `custom` (`x-polaris-delete-principal`) | Access management | +| `AFTER_ROTATE_CREDENTIALS` | `custom` (`x-polaris-rotate-credentials`) | Security operation | +| `AFTER_CREATE_PRINCIPAL_ROLE` | `custom` (`x-polaris-create-principal-role`) | RBAC | +| `AFTER_CREATE_CATALOG_ROLE` | `custom` (`x-polaris-create-catalog-role`) | RBAC | +| `AFTER_ADD_GRANT_TO_CATALOG_ROLE` | `custom` (`x-polaris-grant-privilege`) | RBAC | +| `AFTER_REVOKE_GRANT_FROM_CATALOG_ROLE` | `custom` (`x-polaris-revoke-privilege`) | RBAC | +| `AFTER_CREATE_POLICY` | `custom` (`x-polaris-create-policy`) | Policy management | +| `AFTER_ATTACH_POLICY` | `custom` (`x-polaris-attach-policy`) | Policy management | +| `AFTER_CREATE_GENERIC_TABLE` | `custom` (`x-polaris-create-generic-table`) | Generic table support | + +#### 7.5.3 Read-Only Operations: Not Exposed + +Events for read-only operations are **not exposed** via the Events API because they do not represent catalog mutations: + +| Excluded Event Types | Reason | +|----------------------|--------| +| `AFTER_GET_CATALOG`, `AFTER_LIST_CATALOGS` | Read-only, no state change | +| `AFTER_LOAD_TABLE`, `AFTER_LIST_TABLES`, `AFTER_CHECK_EXISTS_TABLE` | Read-only | +| `AFTER_LOAD_NAMESPACE_METADATA`, `AFTER_LIST_NAMESPACES` | Read-only | +| `AFTER_LOAD_VIEW`, `AFTER_LIST_VIEWS` | Read-only | +| `AFTER_GET_CONFIG`, `AFTER_LOAD_CREDENTIALS` | Configuration/credential reads | +| `AFTER_LIST_*` (all list operations) | Read-only enumeration | + +#### 7.5.4 Implementation: Event Filtering + +The event persistence layer should filter events before storing them for the Events API: + +```java +// Events eligible for the REST API (completed mutations only) +private static final Set EXPOSED_EVENT_TYPES = Set.of( + // Standard Iceberg operations + AFTER_CREATE_TABLE, AFTER_UPDATE_TABLE, AFTER_DROP_TABLE, + AFTER_RENAME_TABLE, AFTER_REGISTER_TABLE, + AFTER_CREATE_VIEW, AFTER_DROP_VIEW, AFTER_REPLACE_VIEW, AFTER_RENAME_VIEW, + AFTER_CREATE_NAMESPACE, AFTER_UPDATE_NAMESPACE_PROPERTIES, AFTER_DROP_NAMESPACE, + // Polaris custom operations + AFTER_CREATE_CATALOG, AFTER_DELETE_CATALOG, AFTER_UPDATE_CATALOG, + AFTER_CREATE_PRINCIPAL, AFTER_DELETE_PRINCIPAL, AFTER_UPDATE_PRINCIPAL, + AFTER_ROTATE_CREDENTIALS, AFTER_RESET_CREDENTIALS, + AFTER_CREATE_PRINCIPAL_ROLE, AFTER_DELETE_PRINCIPAL_ROLE, + AFTER_CREATE_CATALOG_ROLE, AFTER_DELETE_CATALOG_ROLE, + AFTER_ADD_GRANT_TO_CATALOG_ROLE, AFTER_REVOKE_GRANT_FROM_CATALOG_ROLE, + AFTER_ASSIGN_PRINCIPAL_ROLE, AFTER_REVOKE_PRINCIPAL_ROLE, + AFTER_ASSIGN_CATALOG_ROLE_TO_PRINCIPAL_ROLE, AFTER_REVOKE_CATALOG_ROLE_FROM_PRINCIPAL_ROLE, + AFTER_CREATE_POLICY, AFTER_UPDATE_POLICY, AFTER_DROP_POLICY, + AFTER_ATTACH_POLICY, AFTER_DETACH_POLICY, + AFTER_CREATE_GENERIC_TABLE, AFTER_DROP_GENERIC_TABLE +); + +public boolean shouldPersistForEventsApi(PolarisEventType eventType) { + return EXPOSED_EVENT_TYPES.contains(eventType); +} +``` + --- ## Appendix A: Polaris Internal Event Types Reference From 69bf1eef0bf286521aa9fa712af033f6fec65c92 Mon Sep 17 00:00:00 2001 From: Anand Kumar Sankaran Date: Tue, 3 Mar 2026 17:03:10 -0800 Subject: [PATCH 07/15] refactor: Consolidate metrics endpoints into single /metrics endpoint Address feedback from adnanhemani to use a single metrics endpoint with metricType as a query parameter instead of separate /scan-metrics and /commit-metrics endpoints. Changes: - Single endpoint: /catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/metrics - Required query parameter: metricType (enum: scan, commit) - Unified response schema: ListMetricsResponse with metricType discriminator - Future-extensible: new metric types can be added without new endpoints - operation parameter only applicable when metricType=commit --- .../proposals/observability-rest-api.md | 144 +++++++----------- 1 file changed, 52 insertions(+), 92 deletions(-) diff --git a/site/content/in-dev/unreleased/proposals/observability-rest-api.md b/site/content/in-dev/unreleased/proposals/observability-rest-api.md index d545715d66..229df6c59d 100644 --- a/site/content/in-dev/unreleased/proposals/observability-rest-api.md +++ b/site/content/in-dev/unreleased/proposals/observability-rest-api.md @@ -114,10 +114,9 @@ Adding read-only REST endpoints enables: | Method | Path | Description | |--------|------|-------------| | POST | `/api/catalog/v1/{prefix}/events` | Query events for a catalog (Iceberg-compatible) | -| GET | `/api/metrics-reports/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/scan-metrics` | List scan metrics for a table | -| GET | `/api/metrics-reports/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/commit-metrics` | List commit metrics for a table | +| GET | `/api/metrics-reports/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/metrics` | List metrics for a table (type specified via query parameter) | -> **Note:** The Events API uses POST (not GET) and follows the Iceberg REST Catalog path structure (`/api/catalog/v1/{prefix}/events`) for compatibility with the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584). The metrics APIs use a dedicated `/api/metrics-reports/v1/` namespace since they expose pre-populated records rather than managing catalog state - a server that doesn't support catalog management may still expose metrics reports. +> **Note:** The Events API uses POST (not GET) and follows the Iceberg REST Catalog path structure (`/api/catalog/v1/{prefix}/events`) for compatibility with the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584). The metrics API uses a dedicated `/api/metrics-reports/v1/` namespace since it exposes pre-populated records rather than managing catalog state - a server that doesn't support catalog management may still expose metrics reports. ### 4.2 Path Parameters @@ -179,30 +178,22 @@ For Polaris-specific events not covered by the Iceberg spec, use the `x-` prefix | `x-polaris-create-policy` | Policy created | | `x-polaris-attach-policy` | Policy attached to resource | -### 4.4 Query Parameters (Metrics APIs) +### 4.4 Query Parameters (Metrics API) -#### List Scan Metrics (`/.../tables/{table}/scan-metrics`) +#### List Table Metrics (`/.../tables/{table}/metrics`) | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| +| `metricType` | string | **Yes** | - | Type of metrics to retrieve: `scan` or `commit` | | `pageToken` | string | No | - | Cursor for pagination | | `pageSize` | integer | No | 100 | Results per page (max: 1000) | | `snapshotId` | long | No | - | Filter by snapshot ID | | `principalName` | string | No | - | Filter by principal | | `timestampFrom` | long | No | - | Start of time range (epoch ms) | | `timestampTo` | long | No | - | End of time range (epoch ms) | +| `operation` | string | No | - | Filter by commit operation (only applicable when `metricType=commit`): `append`, `overwrite`, `delete`, `replace` | -#### List Commit Metrics (`/.../tables/{table}/commit-metrics`) - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `pageToken` | string | No | - | Cursor for pagination | -| `pageSize` | integer | No | 100 | Results per page (max: 1000) | -| `snapshotId` | long | No | - | Filter by snapshot ID | -| `operation` | string | No | - | Filter by operation: `append`, `overwrite`, `delete`, `replace` | -| `principalName` | string | No | - | Filter by principal | -| `timestampFrom` | long | No | - | Start of time range (epoch ms) | -| `timestampTo` | long | No | - | End of time range (epoch ms) | +> **Note:** The `metricType` parameter is required. This design allows for future extensibility as new metric types are added (e.g., compaction metrics, maintenance metrics) without requiring new endpoints. ### 4.5 Example Requests and Responses @@ -325,11 +316,11 @@ Content-Type: application/json } ``` -#### List Scan Metrics +#### List Metrics (Scan) **Request:** ```http -GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views/scan-metrics?pageSize=2×tampFrom=1709251200000 +GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views/metrics?metricType=scan&pageSize=2×tampFrom=1709251200000 Authorization: Bearer ``` @@ -337,6 +328,7 @@ Authorization: Bearer ```json { "nextPageToken": null, + "metricType": "scan", "reports": [ { "reportId": "scan-001-abc123", @@ -373,11 +365,11 @@ Authorization: Bearer } ``` -#### List Commit Metrics +#### List Metrics (Commit) **Request:** ```http -GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views/commit-metrics?operation=append&pageSize=2 +GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views/metrics?metricType=commit&operation=append&pageSize=2 Authorization: Bearer ``` @@ -385,6 +377,7 @@ Authorization: Bearer ```json { "nextPageToken": "eyJ0cyI6MTcwOTMzNzcwMDAwMCwiaWQiOiJjb21taXQtMDAyIn0=", + "metricType": "commit", "reports": [ { "reportId": "commit-001-xyz789", @@ -546,25 +539,37 @@ paths: $ref: '#/components/responses/ServerErrorResponse' ``` -### 6.2 Metrics APIs (New Metrics Reports Service) +### 6.2 Metrics API (New Metrics Reports Service) Add the following to a new `spec/metrics-reports-service.yml` (or extend existing management service): -> **Note:** The metrics APIs use `/api/metrics-reports/v1/` as the base path, separate from the management API. This reflects that metrics reports are read-only access to pre-populated data, not catalog management operations. +> **Note:** The metrics API uses `/api/metrics-reports/v1/` as the base path, separate from the management API. This reflects that metrics reports are read-only access to pre-populated data, not catalog management operations. ```yaml paths: - /catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/scan-metrics: + /catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/metrics: parameters: - $ref: '#/components/parameters/catalogName' - $ref: '#/components/parameters/namespace' - $ref: '#/components/parameters/table' get: - operationId: listTableScanMetrics - summary: List scan metrics for a table + operationId: listTableMetrics + summary: List metrics for a table + description: > + Returns metrics reports for the specified table. The type of metrics + (scan or commit) must be specified via the required metricType parameter. + This unified endpoint supports future extensibility as new metric types + are added. tags: - Observability parameters: + - name: metricType + in: query + required: true + description: Type of metrics to retrieve + schema: + type: string + enum: [scan, commit] - name: pageToken in: query schema: @@ -595,70 +600,21 @@ paths: schema: type: integer format: int64 - responses: - '200': - description: Paginated list of scan metrics - content: - application/json: - schema: - $ref: '#/components/schemas/ListScanMetricsResponse' - '403': - description: Insufficient privileges - '404': - description: Table not found - - /catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/commit-metrics: - parameters: - - $ref: '#/components/parameters/catalogName' - - $ref: '#/components/parameters/namespace' - - $ref: '#/components/parameters/table' - get: - operationId: listTableCommitMetrics - summary: List commit metrics for a table - tags: - - Observability - parameters: - - name: pageToken - in: query - schema: - type: string - - name: pageSize - in: query - schema: - type: integer - minimum: 1 - maximum: 1000 - default: 100 - - name: snapshotId - in: query - schema: - type: integer - format: int64 - name: operation in: query + description: Filter by commit operation (only applicable when metricType=commit) schema: type: string - - name: principalName - in: query - schema: - type: string - - name: timestampFrom - in: query - schema: - type: integer - format: int64 - - name: timestampTo - in: query - schema: - type: integer - format: int64 + enum: [append, overwrite, delete, replace] responses: '200': - description: Paginated list of commit metrics + description: Paginated list of metrics reports content: application/json: schema: - $ref: '#/components/schemas/ListCommitMetricsResponse' + $ref: '#/components/schemas/ListMetricsResponse' + '400': + description: Bad request (e.g., missing metricType, invalid parameter combination) '403': description: Insufficient privileges '404': @@ -963,15 +919,29 @@ components: type: integer format: int64 - ListScanMetricsResponse: + ListMetricsResponse: type: object + required: + - metricType + - reports properties: nextPageToken: type: string + description: Cursor for fetching the next page of results + metricType: + type: string + enum: [scan, commit] + description: The type of metrics in this response reports: type: array + description: > + Array of metrics reports. The schema of each report depends on metricType: + - For metricType=scan: ScanMetricsReport objects + - For metricType=commit: CommitMetricsReport objects items: - $ref: '#/components/schemas/ScanMetricsReport' + oneOf: + - $ref: '#/components/schemas/ScanMetricsReport' + - $ref: '#/components/schemas/CommitMetricsReport' CommitMetricsReport: type: object @@ -1064,16 +1034,6 @@ components: format: int64 attempts: type: integer - - ListCommitMetricsResponse: - type: object - properties: - nextPageToken: - type: string - reports: - type: array - items: - $ref: '#/components/schemas/CommitMetricsReport' ``` --- From 7d53e0d0c23f0b9ed7a4dac25eb509e6f4e3ef4b Mon Sep 17 00:00:00 2001 From: Anand Kumar Sankaran Date: Tue, 3 Mar 2026 17:06:28 -0800 Subject: [PATCH 08/15] refactor: Remove redundant /metrics suffix from endpoint path The base path /api/metrics-reports/v1/ already establishes the context, so the trailing /metrics is redundant. Before: /api/metrics-reports/v1/catalogs/{catalog}/namespaces/{ns}/tables/{table}/metrics After: /api/metrics-reports/v1/catalogs/{catalog}/namespaces/{ns}/tables/{table} --- .../unreleased/proposals/observability-rest-api.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/site/content/in-dev/unreleased/proposals/observability-rest-api.md b/site/content/in-dev/unreleased/proposals/observability-rest-api.md index 229df6c59d..d8fc198b84 100644 --- a/site/content/in-dev/unreleased/proposals/observability-rest-api.md +++ b/site/content/in-dev/unreleased/proposals/observability-rest-api.md @@ -114,7 +114,7 @@ Adding read-only REST endpoints enables: | Method | Path | Description | |--------|------|-------------| | POST | `/api/catalog/v1/{prefix}/events` | Query events for a catalog (Iceberg-compatible) | -| GET | `/api/metrics-reports/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/metrics` | List metrics for a table (type specified via query parameter) | +| GET | `/api/metrics-reports/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}` | List metrics for a table (type specified via query parameter) | > **Note:** The Events API uses POST (not GET) and follows the Iceberg REST Catalog path structure (`/api/catalog/v1/{prefix}/events`) for compatibility with the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584). The metrics API uses a dedicated `/api/metrics-reports/v1/` namespace since it exposes pre-populated records rather than managing catalog state - a server that doesn't support catalog management may still expose metrics reports. @@ -180,7 +180,7 @@ For Polaris-specific events not covered by the Iceberg spec, use the `x-` prefix ### 4.4 Query Parameters (Metrics API) -#### List Table Metrics (`/.../tables/{table}/metrics`) +#### List Table Metrics (`/.../tables/{table}`) | Parameter | Type | Required | Default | Description | |-----------|------|----------|---------|-------------| @@ -320,7 +320,7 @@ Content-Type: application/json **Request:** ```http -GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views/metrics?metricType=scan&pageSize=2×tampFrom=1709251200000 +GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views?metricType=scan&pageSize=2×tampFrom=1709251200000 Authorization: Bearer ``` @@ -369,7 +369,7 @@ Authorization: Bearer **Request:** ```http -GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views/metrics?metricType=commit&operation=append&pageSize=2 +GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views?metricType=commit&operation=append&pageSize=2 Authorization: Bearer ``` @@ -547,7 +547,7 @@ Add the following to a new `spec/metrics-reports-service.yml` (or extend existin ```yaml paths: - /catalogs/{catalogName}/namespaces/{namespace}/tables/{table}/metrics: + /catalogs/{catalogName}/namespaces/{namespace}/tables/{table}: parameters: - $ref: '#/components/parameters/catalogName' - $ref: '#/components/parameters/namespace' From 3a1c49ea3e9d455d9990638224e3d0b8c70214d2 Mon Sep 17 00:00:00 2001 From: Anand Kumar Sankaran Date: Wed, 4 Mar 2026 10:50:01 -0800 Subject: [PATCH 09/15] docs: Address review feedback on Events API design Changes based on reviewer comments: 1. Use /api/events/v1/{prefix} instead of /api/catalog/v1/{prefix}/events to avoid URI path clashes with Iceberg REST Catalog events API until that spec is approved (dimas-b) 2. Change x-polaris-* to polaris-* for custom operation types since the x- prefix is obsolete (dimas-b) 3. Add note confirming QueryEventsRequest matches Iceberg Events API specification for ecosystem compatibility (dimas-b) 4. Update spec file references from rest-catalog-open-api.yaml to new events-service.yml --- .../proposals/observability-rest-api.md | 114 +++++++++--------- 1 file changed, 59 insertions(+), 55 deletions(-) diff --git a/site/content/in-dev/unreleased/proposals/observability-rest-api.md b/site/content/in-dev/unreleased/proposals/observability-rest-api.md index d8fc198b84..f73cb03014 100644 --- a/site/content/in-dev/unreleased/proposals/observability-rest-api.md +++ b/site/content/in-dev/unreleased/proposals/observability-rest-api.md @@ -113,10 +113,10 @@ Adding read-only REST endpoints enables: | Method | Path | Description | |--------|------|-------------| -| POST | `/api/catalog/v1/{prefix}/events` | Query events for a catalog (Iceberg-compatible) | +| POST | `/api/events/v1/{prefix}` | Query events for a catalog | | GET | `/api/metrics-reports/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}` | List metrics for a table (type specified via query parameter) | -> **Note:** The Events API uses POST (not GET) and follows the Iceberg REST Catalog path structure (`/api/catalog/v1/{prefix}/events`) for compatibility with the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584). The metrics API uses a dedicated `/api/metrics-reports/v1/` namespace since it exposes pre-populated records rather than managing catalog state - a server that doesn't support catalog management may still expose metrics reports. +> **Note:** The Events API uses a dedicated `/api/events/v1/` namespace to avoid URI path clashes with the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584) until that spec is approved. The API design follows Iceberg Events API patterns for future compatibility. The metrics API uses a dedicated `/api/metrics-reports/v1/` namespace since it exposes pre-populated records rather than managing catalog state. ### 4.2 Path Parameters @@ -134,10 +134,12 @@ The Events API follows the [Iceberg Events API specification](https://github.com - **POST method**: Allows complex filtering with arrays and nested objects in the request body - **Continuation token**: Opaque cursor for resumable pagination - **Operation-centric model**: Events are structured around operations (create-table, update-table, etc.) -- **Custom extensions**: Support for `x-` prefixed custom operation types for Polaris-specific events +- **Custom extensions**: Support for `polaris-` prefixed custom operation types for Polaris-specific events #### Request Body (`QueryEventsRequest`) +> **Note:** This request schema matches the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584) for ecosystem compatibility. + | Property | Type | Required | Description | |----------|------|----------|-------------| | `continuation-token` | string | No | Opaque cursor to resume fetching from previous request | @@ -168,15 +170,15 @@ The Events API follows the [Iceberg Events API specification](https://github.com #### Polaris Custom Operation Types -For Polaris-specific events not covered by the Iceberg spec, use the `x-` prefix convention: +For Polaris-specific events not covered by the Iceberg spec, use the `polaris-` prefix: | Custom Operation Type | Description | |----------------------|-------------| -| `x-polaris-create-catalog-role` | Catalog role created | -| `x-polaris-grant-privilege` | Privilege granted | -| `x-polaris-rotate-credentials` | Principal credentials rotated | -| `x-polaris-create-policy` | Policy created | -| `x-polaris-attach-policy` | Policy attached to resource | +| `polaris-create-catalog-role` | Catalog role created | +| `polaris-grant-privilege` | Privilege granted | +| `polaris-rotate-credentials` | Principal credentials rotated | +| `polaris-create-policy` | Policy created | +| `polaris-attach-policy` | Policy attached to resource | ### 4.4 Query Parameters (Metrics API) @@ -201,7 +203,7 @@ For Polaris-specific events not covered by the Iceberg spec, use the `x-` prefix **Request:** ```http -POST /api/catalog/v1/my-catalog/events +POST /api/events/v1/my-catalog Authorization: Bearer Content-Type: application/json @@ -276,13 +278,13 @@ Content-Type: application/json **Request:** ```http -POST /api/catalog/v1/my-catalog/events +POST /api/events/v1/my-catalog Authorization: Bearer Content-Type: application/json { "page-size": 10, - "operation-types": ["x-polaris-grant-privilege", "x-polaris-rotate-credentials"] + "operation-types": ["polaris-grant-privilege", "polaris-rotate-credentials"] } ``` @@ -302,7 +304,7 @@ Content-Type: application/json }, "operation": { "operation-type": "custom", - "custom-type": "x-polaris-grant-privilege", + "custom-type": "polaris-grant-privilege", "identifier": { "namespace": ["analytics", "events"], "name": "page_views" @@ -479,16 +481,18 @@ New privileges require: ## 6. OpenAPI Schema > **Note:** The OpenAPI specifications below are embedded in this proposal for review context. Upon approval, these should be extracted into separate files for ease of processing and proper integration: -> - **Events API** → `spec/rest-catalog-open-api.yaml` (extending Iceberg REST Catalog spec) +> - **Events API** → `spec/events-service.yml` (new dedicated service spec with base path `/api/events/v1/`) > - **Metrics Reports API** → `spec/metrics-reports-service.yml` (new dedicated service spec with base path `/api/metrics-reports/v1/`) -### 6.1 Events API (Iceberg REST Catalog Extension) +### 6.1 Events API + +Add the following to a new `spec/events-service.yml`: -Add the following to `spec/rest-catalog-open-api.yaml` (aligned with Iceberg Events API spec): +> **Note:** The Events API uses a dedicated `/api/events/v1/` namespace to avoid URI path clashes with the Iceberg REST Catalog events API until that spec is approved. The request/response schemas follow Iceberg Events API patterns for future compatibility. ```yaml paths: - /v1/{prefix}/events: + /v1/{prefix}: parameters: - $ref: '#/components/parameters/prefix' post: @@ -623,7 +627,7 @@ paths: ### 6.3 Events API Schemas (Iceberg-Compatible) -Add these schemas to `spec/rest-catalog-open-api.yaml`: +Add these schemas to `spec/events-service.yml`: ```yaml components: @@ -1087,11 +1091,11 @@ CREATE INDEX IF NOT EXISTS idx_commit_report_lookup | File | Changes | |------|---------| -| `spec/rest-catalog-open-api.yaml` | Add Events API paths and schemas (Iceberg-compatible) | +| `spec/events-service.yml` | **New file** - Events API paths and schemas | | `spec/metrics-reports-service.yml` | **New file** - Metrics Reports API paths and schemas | -| `api/iceberg-service/` | Generated Events API interfaces | +| `api/events-service/` | **New** - Generated Events API interfaces | | `api/metrics-reports-service/` | **New** - Generated Metrics Reports API interfaces | -| `runtime/service/.../catalog/` | Events service implementation | +| `runtime/service/.../events/` | **New** - Events service implementation | | `runtime/service/.../metrics/` | **New** - Metrics reports service implementation | | `polaris-core/.../persistence/BasePersistence.java` | Add read methods | | `persistence/relational-jdbc/.../JdbcBasePersistenceImpl.java` | Query implementations | @@ -1149,19 +1153,19 @@ The mapping follows a straightforward pattern - the `AFTER_` prefix is stripped | `AFTER_CREATE_NAMESPACE` | `create-namespace` | Direct mapping | | `AFTER_UPDATE_NAMESPACE_PROPERTIES` | `update-namespace-properties` | Direct mapping | | `AFTER_DROP_NAMESPACE` | `drop-namespace` | Direct mapping | -| **Polaris Custom Operations** | | Use `custom` type with `x-polaris-*` | -| `AFTER_CREATE_CATALOG` | `custom` (`x-polaris-create-catalog`) | Catalog-level, not in Iceberg spec | -| `AFTER_DELETE_CATALOG` | `custom` (`x-polaris-delete-catalog`) | Catalog-level | -| `AFTER_CREATE_PRINCIPAL` | `custom` (`x-polaris-create-principal`) | Access management | -| `AFTER_DELETE_PRINCIPAL` | `custom` (`x-polaris-delete-principal`) | Access management | -| `AFTER_ROTATE_CREDENTIALS` | `custom` (`x-polaris-rotate-credentials`) | Security operation | -| `AFTER_CREATE_PRINCIPAL_ROLE` | `custom` (`x-polaris-create-principal-role`) | RBAC | -| `AFTER_CREATE_CATALOG_ROLE` | `custom` (`x-polaris-create-catalog-role`) | RBAC | -| `AFTER_ADD_GRANT_TO_CATALOG_ROLE` | `custom` (`x-polaris-grant-privilege`) | RBAC | -| `AFTER_REVOKE_GRANT_FROM_CATALOG_ROLE` | `custom` (`x-polaris-revoke-privilege`) | RBAC | -| `AFTER_CREATE_POLICY` | `custom` (`x-polaris-create-policy`) | Policy management | -| `AFTER_ATTACH_POLICY` | `custom` (`x-polaris-attach-policy`) | Policy management | -| `AFTER_CREATE_GENERIC_TABLE` | `custom` (`x-polaris-create-generic-table`) | Generic table support | +| **Polaris Custom Operations** | | Use `custom` type with `polaris-*` | +| `AFTER_CREATE_CATALOG` | `custom` (`polaris-create-catalog`) | Catalog-level, not in Iceberg spec | +| `AFTER_DELETE_CATALOG` | `custom` (`polaris-delete-catalog`) | Catalog-level | +| `AFTER_CREATE_PRINCIPAL` | `custom` (`polaris-create-principal`) | Access management | +| `AFTER_DELETE_PRINCIPAL` | `custom` (`polaris-delete-principal`) | Access management | +| `AFTER_ROTATE_CREDENTIALS` | `custom` (`polaris-rotate-credentials`) | Security operation | +| `AFTER_CREATE_PRINCIPAL_ROLE` | `custom` (`polaris-create-principal-role`) | RBAC | +| `AFTER_CREATE_CATALOG_ROLE` | `custom` (`polaris-create-catalog-role`) | RBAC | +| `AFTER_ADD_GRANT_TO_CATALOG_ROLE` | `custom` (`polaris-grant-privilege`) | RBAC | +| `AFTER_REVOKE_GRANT_FROM_CATALOG_ROLE` | `custom` (`polaris-revoke-privilege`) | RBAC | +| `AFTER_CREATE_POLICY` | `custom` (`polaris-create-policy`) | Policy management | +| `AFTER_ATTACH_POLICY` | `custom` (`polaris-attach-policy`) | Policy management | +| `AFTER_CREATE_GENERIC_TABLE` | `custom` (`polaris-create-generic-table`) | Generic table support | #### 7.5.3 Read-Only Operations: Not Exposed @@ -1215,13 +1219,13 @@ Polaris internal event types are categorized by code ranges. These are mapped to | Range | Category | Internal Event Type | Iceberg Operation Type | |-------|----------|---------------------|------------------------| -| 100-109 | Catalog | `AFTER_CREATE_CATALOG` | `custom` (`x-polaris-create-catalog`) | -| | | `AFTER_DELETE_CATALOG` | `custom` (`x-polaris-delete-catalog`) | -| 200-217 | Catalog Role | `AFTER_CREATE_CATALOG_ROLE` | `custom` (`x-polaris-create-catalog-role`) | -| | | `AFTER_ADD_GRANT_TO_CATALOG_ROLE` | `custom` (`x-polaris-grant-privilege`) | -| 300-319 | Principal | `AFTER_CREATE_PRINCIPAL` | `custom` (`x-polaris-create-principal`) | -| | | `AFTER_ROTATE_CREDENTIALS` | `custom` (`x-polaris-rotate-credentials`) | -| 400-417 | Principal Role | `AFTER_CREATE_PRINCIPAL_ROLE` | `custom` (`x-polaris-create-principal-role`) | +| 100-109 | Catalog | `AFTER_CREATE_CATALOG` | `custom` (`polaris-create-catalog`) | +| | | `AFTER_DELETE_CATALOG` | `custom` (`polaris-delete-catalog`) | +| 200-217 | Catalog Role | `AFTER_CREATE_CATALOG_ROLE` | `custom` (`polaris-create-catalog-role`) | +| | | `AFTER_ADD_GRANT_TO_CATALOG_ROLE` | `custom` (`polaris-grant-privilege`) | +| 300-319 | Principal | `AFTER_CREATE_PRINCIPAL` | `custom` (`polaris-create-principal`) | +| | | `AFTER_ROTATE_CREDENTIALS` | `custom` (`polaris-rotate-credentials`) | +| 400-417 | Principal Role | `AFTER_CREATE_PRINCIPAL_ROLE` | `custom` (`polaris-create-principal-role`) | | 500-511 | Namespace | `AFTER_CREATE_NAMESPACE` | `create-namespace` | | | | `AFTER_UPDATE_NAMESPACE_PROPERTIES` | `update-namespace-properties` | | | | `AFTER_DROP_NAMESPACE` | `drop-namespace` | @@ -1234,9 +1238,9 @@ Polaris internal event types are categorized by code ranges. These are mapped to | | | `AFTER_UPDATE_VIEW` | `update-view` | | | | `AFTER_DROP_VIEW` | `drop-view` | | | | `AFTER_RENAME_VIEW` | `rename-view` | -| 1200-1215 | Policy | `AFTER_CREATE_POLICY` | `custom` (`x-polaris-create-policy`) | -| | | `AFTER_ATTACH_POLICY` | `custom` (`x-polaris-attach-policy`) | -| 1300-1307 | Generic Table | `AFTER_CREATE_GENERIC_TABLE` | `custom` (`x-polaris-create-generic-table`) | +| 1200-1215 | Policy | `AFTER_CREATE_POLICY` | `custom` (`polaris-create-policy`) | +| | | `AFTER_ATTACH_POLICY` | `custom` (`polaris-attach-policy`) | +| 1300-1307 | Generic Table | `AFTER_CREATE_GENERIC_TABLE` | `custom` (`polaris-create-generic-table`) | --- @@ -1261,24 +1265,24 @@ The Iceberg Events API is an emerging specification that is nearing consensus in | **API Path** | `/v1/{prefix}/events` | Part of Iceberg REST Catalog, not a separate management API | | **Pagination** | `continuation-token` | Opaque cursor that encodes server state; resumable after downtime | | **Event Structure** | Operation-centric with discriminator | Each event contains a typed `operation` with operation-specific fields | -| **Operation Types** | Standardized enum + `x-` prefix extensions | Standard types for Iceberg operations; custom prefix for catalog-specific extensions | +| **Operation Types** | Standardized enum + custom prefix extensions | Standard types for Iceberg operations; custom prefix for catalog-specific extensions | | **Actor Field** | Generic object (implementation-specific) | Flexibility for different auth models (users, service accounts, etc.) | | **Error Handling** | `410 Gone` for expired offsets | Explicit signal when continuation token is outside retention window | ### 8.3 Polaris-Specific Extensions -Polaris extends the Iceberg Events API using the `custom` operation type with `x-polaris-*` prefixed custom types: +Polaris extends the Iceberg Events API using the `custom` operation type with `polaris-*` prefixed custom types: | Custom Type | Polaris Event | Description | |-------------|---------------|-------------| -| `x-polaris-create-catalog` | `AFTER_CREATE_CATALOG` | Catalog created | -| `x-polaris-create-catalog-role` | `AFTER_CREATE_CATALOG_ROLE` | Catalog role created | -| `x-polaris-grant-privilege` | `AFTER_ADD_GRANT_TO_CATALOG_ROLE` | Privilege granted to role | -| `x-polaris-revoke-privilege` | `AFTER_REMOVE_GRANT_FROM_CATALOG_ROLE` | Privilege revoked | -| `x-polaris-create-principal` | `AFTER_CREATE_PRINCIPAL` | Principal created | -| `x-polaris-rotate-credentials` | `AFTER_ROTATE_CREDENTIALS` | Credentials rotated | -| `x-polaris-create-policy` | `AFTER_CREATE_POLICY` | Policy created | -| `x-polaris-attach-policy` | `AFTER_ATTACH_POLICY` | Policy attached to resource | +| `polaris-create-catalog` | `AFTER_CREATE_CATALOG` | Catalog created | +| `polaris-create-catalog-role` | `AFTER_CREATE_CATALOG_ROLE` | Catalog role created | +| `polaris-grant-privilege` | `AFTER_ADD_GRANT_TO_CATALOG_ROLE` | Privilege granted to role | +| `polaris-revoke-privilege` | `AFTER_REMOVE_GRANT_FROM_CATALOG_ROLE` | Privilege revoked | +| `polaris-create-principal` | `AFTER_CREATE_PRINCIPAL` | Principal created | +| `polaris-rotate-credentials` | `AFTER_ROTATE_CREDENTIALS` | Credentials rotated | +| `polaris-create-policy` | `AFTER_CREATE_POLICY` | Policy created | +| `polaris-attach-policy` | `AFTER_ATTACH_POLICY` | Policy attached to resource | ### 8.4 Mapping Polaris Internal Events to Iceberg Operations @@ -1296,7 +1300,7 @@ Polaris extends the Iceberg Events API using the `custom` operation type with `x | `AFTER_CREATE_NAMESPACE` | `create-namespace` | | `AFTER_UPDATE_NAMESPACE_PROPERTIES` | `update-namespace-properties` | | `AFTER_DROP_NAMESPACE` | `drop-namespace` | -| Other Polaris events | `custom` with `x-polaris-*` type | +| Other Polaris events | `custom` with `polaris-*` type | ### 8.5 References From ce646ff47f64622b9cf6a88d86218b1fb7cc5dfd Mon Sep 17 00:00:00 2001 From: Anand Kumar Sankaran Date: Wed, 4 Mar 2026 10:53:30 -0800 Subject: [PATCH 10/15] refactor: Make ListMetricsResponse polymorphic at top level Address dimas-b feedback: the previous oneOf at item level incorrectly allowed mixing scan and commit reports in a single response. Now using discriminator at response level: - ListMetricsResponse is polymorphic with metricType discriminator - ListScanMetricsResponse: metricType=scan, reports array of ScanMetricsReport - ListCommitMetricsResponse: metricType=commit, reports array of CommitMetricsReport This correctly expresses that all items in a response are of the same type. --- .../proposals/observability-rest-api.md | 45 +++++++++++++++---- 1 file changed, 36 insertions(+), 9 deletions(-) diff --git a/site/content/in-dev/unreleased/proposals/observability-rest-api.md b/site/content/in-dev/unreleased/proposals/observability-rest-api.md index f73cb03014..659d59370c 100644 --- a/site/content/in-dev/unreleased/proposals/observability-rest-api.md +++ b/site/content/in-dev/unreleased/proposals/observability-rest-api.md @@ -924,6 +924,19 @@ components: format: int64 ListMetricsResponse: + description: > + Polymorphic response for metrics queries. The concrete type is determined + by the metricType discriminator field. + oneOf: + - $ref: '#/components/schemas/ListScanMetricsResponse' + - $ref: '#/components/schemas/ListCommitMetricsResponse' + discriminator: + propertyName: metricType + mapping: + scan: '#/components/schemas/ListScanMetricsResponse' + commit: '#/components/schemas/ListCommitMetricsResponse' + + ListScanMetricsResponse: type: object required: - metricType @@ -934,18 +947,32 @@ components: description: Cursor for fetching the next page of results metricType: type: string - enum: [scan, commit] - description: The type of metrics in this response + const: scan + description: Discriminator indicating this response contains scan metrics reports: type: array - description: > - Array of metrics reports. The schema of each report depends on metricType: - - For metricType=scan: ScanMetricsReport objects - - For metricType=commit: CommitMetricsReport objects + description: Array of scan metrics reports + items: + $ref: '#/components/schemas/ScanMetricsReport' + + ListCommitMetricsResponse: + type: object + required: + - metricType + - reports + properties: + nextPageToken: + type: string + description: Cursor for fetching the next page of results + metricType: + type: string + const: commit + description: Discriminator indicating this response contains commit metrics + reports: + type: array + description: Array of commit metrics reports items: - oneOf: - - $ref: '#/components/schemas/ScanMetricsReport' - - $ref: '#/components/schemas/CommitMetricsReport' + $ref: '#/components/schemas/CommitMetricsReport' CommitMetricsReport: type: object From ae585d0d0300a050300260e6551b855cbe7d1f3b Mon Sep 17 00:00:00 2001 From: Anand Kumar Sankaran Date: Wed, 4 Mar 2026 11:01:50 -0800 Subject: [PATCH 11/15] docs: Add realm handling to design principles Address dimas-b feedback: mention that the new APIs process Polaris realms consistently with existing APIs. --- .../in-dev/unreleased/proposals/observability-rest-api.md | 1 + 1 file changed, 1 insertion(+) diff --git a/site/content/in-dev/unreleased/proposals/observability-rest-api.md b/site/content/in-dev/unreleased/proposals/observability-rest-api.md index 659d59370c..f9762fc8dd 100644 --- a/site/content/in-dev/unreleased/proposals/observability-rest-api.md +++ b/site/content/in-dev/unreleased/proposals/observability-rest-api.md @@ -104,6 +104,7 @@ Adding read-only REST endpoints enables: | **Consistent pagination** | Follow `continuation-token` pattern (Iceberg) and `pageToken` pattern (Polaris APIs) | | **Flexible filtering** | Time ranges, operation types, catalog objects - common query patterns | | **RBAC integration** | Leverage existing Polaris authorization model | +| **Realm handling** | Process Polaris realms consistently with existing APIs; realm context is derived from the authenticated principal | --- From 429939986864e8d27a6066393b81511bad4c881b Mon Sep 17 00:00:00 2001 From: Anand Kumar Sankaran Date: Wed, 4 Mar 2026 12:00:07 -0800 Subject: [PATCH 12/15] docs: Add prerequisite note about extending event persistence layer Address nandorKollar's observation that PolarisPersistenceEventListener currently only persists AFTER_CREATE_TABLE and AFTER_CREATE_CATALOG events. Added new section 7.1 documenting: - Current state: only 2 event types are persisted - Required changes: list of all AFTER_* mutation events to add - Implementation approach: phased rollout (Iceberg ops, then Polaris-specific) This is a prerequisite for the Events API to be useful. --- .../proposals/observability-rest-api.md | 60 ++++++++++++++++--- 1 file changed, 51 insertions(+), 9 deletions(-) diff --git a/site/content/in-dev/unreleased/proposals/observability-rest-api.md b/site/content/in-dev/unreleased/proposals/observability-rest-api.md index f9762fc8dd..bccdb5267f 100644 --- a/site/content/in-dev/unreleased/proposals/observability-rest-api.md +++ b/site/content/in-dev/unreleased/proposals/observability-rest-api.md @@ -1072,7 +1072,49 @@ components: ## 7. Implementation Notes -### 7.1 Database Queries +### 7.1 Prerequisite: Extend Event Persistence Layer + +> **Important:** The current `PolarisPersistenceEventListener` in Apache Polaris only persists **two event types**: `AFTER_CREATE_TABLE` and `AFTER_CREATE_CATALOG`. All other events are ignored. For the Events API to be useful, the persistence layer must be extended to capture all relevant mutation events. + +#### Current State + +The existing event listener (`PolarisPersistenceEventListener.java`) has a limited switch statement: + +```java +public void onEvent(PolarisEvent event) { + switch (event.type()) { + case AFTER_CREATE_TABLE -> handleAfterCreateTable(event); + case AFTER_CREATE_CATALOG -> handleAfterCreateCatalog(event); + default -> { + // Other events not handled by this listener + } + } +} +``` + +#### Required Changes + +The persistence layer needs to be extended to capture all `AFTER_*` mutation events that should be exposed via the Events API: + +| Category | Events to Add | +|----------|---------------| +| **Table Operations** | `AFTER_UPDATE_TABLE`, `AFTER_DROP_TABLE`, `AFTER_RENAME_TABLE`, `AFTER_REGISTER_TABLE` | +| **View Operations** | `AFTER_CREATE_VIEW`, `AFTER_DROP_VIEW`, `AFTER_REPLACE_VIEW`, `AFTER_RENAME_VIEW` | +| **Namespace Operations** | `AFTER_CREATE_NAMESPACE`, `AFTER_UPDATE_NAMESPACE_PROPERTIES`, `AFTER_DROP_NAMESPACE` | +| **Catalog Operations** | `AFTER_DELETE_CATALOG`, `AFTER_UPDATE_CATALOG` | +| **Principal/Role Management** | `AFTER_CREATE_PRINCIPAL`, `AFTER_DELETE_PRINCIPAL`, `AFTER_ROTATE_CREDENTIALS`, `AFTER_CREATE_PRINCIPAL_ROLE`, `AFTER_DELETE_PRINCIPAL_ROLE`, `AFTER_CREATE_CATALOG_ROLE`, `AFTER_DELETE_CATALOG_ROLE` | +| **Grant Operations** | `AFTER_ADD_GRANT_TO_CATALOG_ROLE`, `AFTER_REVOKE_GRANT_FROM_CATALOG_ROLE`, `AFTER_ASSIGN_PRINCIPAL_ROLE`, `AFTER_REVOKE_PRINCIPAL_ROLE`, `AFTER_ASSIGN_CATALOG_ROLE_TO_PRINCIPAL_ROLE`, `AFTER_REVOKE_CATALOG_ROLE_FROM_PRINCIPAL_ROLE` | +| **Policy Operations** | `AFTER_CREATE_POLICY`, `AFTER_UPDATE_POLICY`, `AFTER_DROP_POLICY`, `AFTER_ATTACH_POLICY`, `AFTER_DETACH_POLICY` | + +**Note:** Read-only operations (`AFTER_LOAD_*`, `AFTER_LIST_*`, `AFTER_GET_*`, `AFTER_CHECK_EXISTS_*`) should **not** be persisted for the Events API as they do not represent catalog mutations. However, deployments requiring read audit may choose to persist these separately. + +#### Implementation Approach + +1. **Phase 1**: Extend `PolarisPersistenceEventListener` to handle all Iceberg-standard operations (tables, views, namespaces) +2. **Phase 2**: Add support for Polaris-specific operations (principals, roles, grants, policies) +3. **Phase 3**: Implement the Events REST API on top of the persisted data + +### 7.2 Database Queries The endpoints will query existing tables with appropriate filtering and pagination: @@ -1099,7 +1141,7 @@ ORDER BY timestamp_ms DESC, report_id DESC LIMIT ?; ``` -### 7.2 Recommended Indexes +### 7.3 Recommended Indexes ```sql -- Events indexes @@ -1115,7 +1157,7 @@ CREATE INDEX IF NOT EXISTS idx_commit_report_lookup ON commit_metrics_report(realm_id, catalog_id, table_id, timestamp_ms DESC); ``` -### 7.3 Files to Modify +### 7.4 Files to Modify | File | Changes | |------|---------| @@ -1128,7 +1170,7 @@ CREATE INDEX IF NOT EXISTS idx_commit_report_lookup | `polaris-core/.../persistence/BasePersistence.java` | Add read methods | | `persistence/relational-jdbc/.../JdbcBasePersistenceImpl.java` | Query implementations | -### 7.4 Pagination Token Format +### 7.5 Pagination Token Format Internal format (base64-encoded JSON, opaque to clients): @@ -1139,11 +1181,11 @@ Internal format (base64-encoded JSON, opaque to clients): } ``` -### 7.5 Mapping PolarisEventType to Iceberg Events API +### 7.6 Mapping PolarisEventType to Iceberg Events API Polaris internally uses a `PolarisEventType` enum that distinguishes between `BEFORE_*` and `AFTER_*` events for each operation (e.g., `BEFORE_CREATE_TABLE` and `AFTER_CREATE_TABLE`). This section explains how these internal events map to the Iceberg Events API. -#### 7.5.1 Design Decision: Only AFTER Events are Exposed +#### 7.6.1 Design Decision: Only AFTER Events are Exposed The Iceberg Events API represents **completed operations** that have been committed to the catalog. Therefore: @@ -1162,7 +1204,7 @@ The Iceberg Events API represents **completed operations** that have been commit 4. **Internal vs external**: `BEFORE_*` events serve internal purposes (request filtering, rate limiting, pre-validation hooks) and are not meaningful to external consumers. -#### 7.5.2 Mapping AFTER Events to Operation Types +#### 7.6.2 Mapping AFTER Events to Operation Types The mapping follows a straightforward pattern - the `AFTER_` prefix is stripped and the remaining name maps to the Iceberg operation type: @@ -1195,7 +1237,7 @@ The mapping follows a straightforward pattern - the `AFTER_` prefix is stripped | `AFTER_ATTACH_POLICY` | `custom` (`polaris-attach-policy`) | Policy management | | `AFTER_CREATE_GENERIC_TABLE` | `custom` (`polaris-create-generic-table`) | Generic table support | -#### 7.5.3 Read-Only Operations: Not Exposed +#### 7.6.3 Read-Only Operations: Not Exposed Events for read-only operations are **not exposed** via the Events API because they do not represent catalog mutations: @@ -1208,7 +1250,7 @@ Events for read-only operations are **not exposed** via the Events API because t | `AFTER_GET_CONFIG`, `AFTER_LOAD_CREDENTIALS` | Configuration/credential reads | | `AFTER_LIST_*` (all list operations) | Read-only enumeration | -#### 7.5.4 Implementation: Event Filtering +#### 7.6.4 Implementation: Event Filtering The event persistence layer should filter events before storing them for the Events API: From edeae43adb608d5ac5424c2a6dd7dc575c46794f Mon Sep 17 00:00:00 2001 From: Anand Kumar Sankaran Date: Fri, 13 Mar 2026 15:54:04 -0700 Subject: [PATCH 13/15] Review comments. --- .../in-dev/unreleased/proposals/_index.md | 3 +- .../proposals/observability-rest-api.md | 1403 ----------------- 2 files changed, 2 insertions(+), 1404 deletions(-) delete mode 100644 site/content/in-dev/unreleased/proposals/observability-rest-api.md diff --git a/site/content/in-dev/unreleased/proposals/_index.md b/site/content/in-dev/unreleased/proposals/_index.md index 302ff5fba2..3344e2360b 100644 --- a/site/content/in-dev/unreleased/proposals/_index.md +++ b/site/content/in-dev/unreleased/proposals/_index.md @@ -30,5 +30,6 @@ This section contains design proposals for new features and enhancements to Apac | Proposal | Status | Description | |----------|--------|-------------| -| [Observability REST API](observability-rest-api.md) | Draft | REST API endpoints for querying table metrics and catalog events | +| [Events REST API](events-rest-api.md) | Draft | REST API endpoint for querying catalog events (Iceberg Events API compatible) | +| [Table Metrics REST API](table-metrics-rest-api.md) | Draft | REST API endpoints for querying table scan and commit metrics | diff --git a/site/content/in-dev/unreleased/proposals/observability-rest-api.md b/site/content/in-dev/unreleased/proposals/observability-rest-api.md deleted file mode 100644 index bccdb5267f..0000000000 --- a/site/content/in-dev/unreleased/proposals/observability-rest-api.md +++ /dev/null @@ -1,1403 +0,0 @@ ---- -title: Observability REST API -linkTitle: Observability REST API -weight: 100 ---- - - -# Proposal: REST API for Querying Table Metrics and Events - -**Author:** Anand Sankaran -**Date:** 2026-03-02 -**Status:** Draft Proposal -**Target:** Apache Polaris - ---- - -## Abstract - -This proposal defines REST API endpoints for querying table metrics and catalog events from Apache Polaris. The endpoints expose data already being persisted via the existing JDBC persistence model (`events`, `scan_metrics_report`, `commit_metrics_report` tables) and follow established Polaris API patterns. - -**Note:** The Events API in this proposal is designed to align with the emerging [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584), which is nearing consensus in the Apache Iceberg community. This ensures forward compatibility and consistency with the broader Iceberg ecosystem. - ---- - -## Table of Contents - -1. [Motivation](#1-motivation) -2. [Use Cases](#2-use-cases) -3. [Design Principles](#3-design-principles) -4. [API Specification](#4-api-specification) -5. [Authorization](#5-authorization) -6. [OpenAPI Schema](#6-openapi-schema) -7. [Implementation Notes](#7-implementation-notes) -8. [Iceberg Events API Alignment](#8-iceberg-events-api-alignment) - ---- - -## 1. Motivation - -Apache Polaris currently persists table metrics (scan reports, commit reports) and catalog events to the database, but provides no REST API to query this data. Users must access the database directly to retrieve metrics or audit information. - -Adding read-only REST endpoints enables: -- Programmatic access to metrics without database credentials -- Integration with monitoring dashboards and alerting systems -- Consistent authorization via Polaris RBAC -- Pagination and filtering without writing SQL - ---- - -## 2. Use Cases - -### 2.1 Table Health Monitoring -- Track write patterns: files added/removed per commit, record counts, duration trends -- Identify tables with high commit frequency or unusually large commits -- Detect issues indicating need for compaction (many small files) or optimization - -### 2.2 Query Performance Analysis -- Understand read patterns: files scanned vs skipped, planning duration -- Identify inefficient queries with low manifest/file pruning ratios -- Correlate performance with filter expressions and projected columns - -### 2.3 Capacity Planning & Chargeback -- Aggregate metrics by table, namespace, or principal over time -- Track storage growth trends (`total_file_size_bytes`) -- Attribute usage to teams/users via `principal_name` - -### 2.4 Debugging & Troubleshooting -- Correlate metrics with distributed traces (`otel_trace_id`, `otel_span_id`) -- Investigate specific commits by `snapshot_id` -- Trace operations via `request_id` - -### 2.5 Audit & Compliance -- Track who created/dropped/modified catalog objects -- Monitor administrative actions (credential rotation, grant changes) -- Generate compliance reports for access patterns - ---- - -## 3. Design Principles - -| Principle | Rationale | -|-----------|-----------| -| **Iceberg Events API alignment** | Events API follows the [Iceberg Events API spec](https://github.com/apache/iceberg/pull/12584) for ecosystem compatibility | -| **Dedicated metrics-reports namespace** | Metrics APIs use `/api/metrics-reports/v1/...` to separate from management and catalog APIs | -| **POST for complex filtering** | Events API uses POST with request body (per Iceberg spec) to support complex filters (arrays, nested objects) | -| **Read-only semantics** | All endpoints are read-only; metrics/events are written via existing flows | -| **Consistent pagination** | Follow `continuation-token` pattern (Iceberg) and `pageToken` pattern (Polaris APIs) | -| **Flexible filtering** | Time ranges, operation types, catalog objects - common query patterns | -| **RBAC integration** | Leverage existing Polaris authorization model | -| **Realm handling** | Process Polaris realms consistently with existing APIs; realm context is derived from the authenticated principal | - ---- - -## 4. API Specification - -### 4.1 Endpoint Summary - -| Method | Path | Description | -|--------|------|-------------| -| POST | `/api/events/v1/{prefix}` | Query events for a catalog | -| GET | `/api/metrics-reports/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}` | List metrics for a table (type specified via query parameter) | - -> **Note:** The Events API uses a dedicated `/api/events/v1/` namespace to avoid URI path clashes with the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584) until that spec is approved. The API design follows Iceberg Events API patterns for future compatibility. The metrics API uses a dedicated `/api/metrics-reports/v1/` namespace since it exposes pre-populated records rather than managing catalog state. - -### 4.2 Path Parameters - -| Parameter | Type | Description | -|-----------|------|-------------| -| `prefix` | string | Catalog prefix (typically the catalog name) | -| `catalogName` | string | Name of the catalog | -| `namespace` | string | Namespace (URL-encoded, multi-level separated by `%1F`) | -| `table` | string | Table name | - -### 4.3 Events API (Iceberg-Compatible) - -The Events API follows the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584) for ecosystem compatibility. Key design decisions from the Iceberg spec: - -- **POST method**: Allows complex filtering with arrays and nested objects in the request body -- **Continuation token**: Opaque cursor for resumable pagination -- **Operation-centric model**: Events are structured around operations (create-table, update-table, etc.) -- **Custom extensions**: Support for `polaris-` prefixed custom operation types for Polaris-specific events - -#### Request Body (`QueryEventsRequest`) - -> **Note:** This request schema matches the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584) for ecosystem compatibility. - -| Property | Type | Required | Description | -|----------|------|----------|-------------| -| `continuation-token` | string | No | Opaque cursor to resume fetching from previous request | -| `page-size` | integer | No | Maximum events per page (server may return fewer) | -| `after-timestamp-ms` | long | No | Filter: events after this timestamp (inclusive) | -| `operation-types` | array[string] | No | Filter by operation types (see below) | -| `catalog-objects-by-name` | array[array[string]] | No | Filter by namespace/table/view names | -| `catalog-objects-by-id` | array[object] | No | Filter by table/view UUIDs | -| `object-types` | array[string] | No | Filter by object type: `namespace`, `table`, `view` | -| `custom-filters` | object | No | Implementation-specific filter extensions | - -#### Standard Operation Types - -| Operation Type | Description | -|----------------|-------------| -| `create-table` | Table created and committed | -| `register-table` | Existing table registered in catalog | -| `drop-table` | Table dropped | -| `update-table` | Table metadata updated | -| `rename-table` | Table renamed | -| `create-view` | View created | -| `drop-view` | View dropped | -| `update-view` | View updated | -| `rename-view` | View renamed | -| `create-namespace` | Namespace created | -| `update-namespace-properties` | Namespace properties updated | -| `drop-namespace` | Namespace dropped | - -#### Polaris Custom Operation Types - -For Polaris-specific events not covered by the Iceberg spec, use the `polaris-` prefix: - -| Custom Operation Type | Description | -|----------------------|-------------| -| `polaris-create-catalog-role` | Catalog role created | -| `polaris-grant-privilege` | Privilege granted | -| `polaris-rotate-credentials` | Principal credentials rotated | -| `polaris-create-policy` | Policy created | -| `polaris-attach-policy` | Policy attached to resource | - -### 4.4 Query Parameters (Metrics API) - -#### List Table Metrics (`/.../tables/{table}`) - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `metricType` | string | **Yes** | - | Type of metrics to retrieve: `scan` or `commit` | -| `pageToken` | string | No | - | Cursor for pagination | -| `pageSize` | integer | No | 100 | Results per page (max: 1000) | -| `snapshotId` | long | No | - | Filter by snapshot ID | -| `principalName` | string | No | - | Filter by principal | -| `timestampFrom` | long | No | - | Start of time range (epoch ms) | -| `timestampTo` | long | No | - | End of time range (epoch ms) | -| `operation` | string | No | - | Filter by commit operation (only applicable when `metricType=commit`): `append`, `overwrite`, `delete`, `replace` | - -> **Note:** The `metricType` parameter is required. This design allows for future extensibility as new metric types are added (e.g., compaction metrics, maintenance metrics) without requiring new endpoints. - -### 4.5 Example Requests and Responses - -#### Query Events (Iceberg-Compatible) - -**Request:** -```http -POST /api/events/v1/my-catalog -Authorization: Bearer -Content-Type: application/json - -{ - "page-size": 2, - "operation-types": ["create-table", "update-table"], - "after-timestamp-ms": 1709251200000, - "catalog-objects-by-name": [ - ["analytics", "events"] - ], - "object-types": ["table"] -} -``` - -**Response:** -```json -{ - "next-page-token": "eyJ0cyI6MTcwOTMzNzYxMjM0NSwiaWQiOiI1NTBlODQwMCJ9", - "highest-processed-timestamp-ms": 1709337612345, - "events": [ - { - "event-id": "550e8400-e29b-41d4-a716-446655440000", - "request-id": "req-12345", - "request-event-count": 1, - "timestamp-ms": 1709337612345, - "actor": { - "principal": "admin@example.com", - "client-ip": "192.168.1.100" - }, - "operation": { - "operation-type": "create-table", - "identifier": { - "namespace": ["analytics", "events"], - "name": "page_views" - }, - "table-uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", - "updates": [ - {"action": "assign-uuid", "uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"}, - {"action": "set-current-schema", "schema-id": 0}, - {"action": "set-default-spec", "spec-id": 0} - ] - } - }, - { - "event-id": "661f9511-f30c-52e5-b827-557766551111", - "request-id": "req-12346", - "request-event-count": 1, - "timestamp-ms": 1709337500000, - "actor": { - "principal": "etl-service@example.com" - }, - "operation": { - "operation-type": "update-table", - "identifier": { - "namespace": ["analytics", "events"], - "name": "user_actions" - }, - "table-uuid": "b2c3d4e5-f6a7-8901-bcde-f23456789012", - "updates": [ - {"action": "add-snapshot", "snapshot-id": 123456789} - ], - "requirements": [ - {"type": "assert-table-uuid", "uuid": "b2c3d4e5-f6a7-8901-bcde-f23456789012"} - ] - } - } - ] -} -``` - -#### Query Events with Custom Polaris Operations - -**Request:** -```http -POST /api/events/v1/my-catalog -Authorization: Bearer -Content-Type: application/json - -{ - "page-size": 10, - "operation-types": ["polaris-grant-privilege", "polaris-rotate-credentials"] -} -``` - -**Response:** -```json -{ - "next-page-token": "eyJ0cyI6MTcwOTMzODAwMDAwMH0=", - "highest-processed-timestamp-ms": 1709338000000, - "events": [ - { - "event-id": "772f0622-g41d-63f6-c938-668877662222", - "request-id": "req-admin-001", - "request-event-count": 1, - "timestamp-ms": 1709338000000, - "actor": { - "principal": "security-admin@example.com" - }, - "operation": { - "operation-type": "custom", - "custom-type": "polaris-grant-privilege", - "identifier": { - "namespace": ["analytics", "events"], - "name": "page_views" - }, - "table-uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", - "privilege": "TABLE_READ_DATA", - "grantee": "data-analyst-role" - } - } - ] -} -``` - -#### List Metrics (Scan) - -**Request:** -```http -GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views?metricType=scan&pageSize=2×tampFrom=1709251200000 -Authorization: Bearer -``` - -**Response:** -```json -{ - "nextPageToken": null, - "metricType": "scan", - "reports": [ - { - "reportId": "scan-001-abc123", - "catalogId": 12345, - "tableId": 67890, - "timestampMs": 1709337612345, - "principalName": "analyst@example.com", - "requestId": "req-scan-001", - "otelTraceId": "abc123def456789012345678901234", - "otelSpanId": "def456789012", - "snapshotId": 1234567890123, - "schemaId": 0, - "filterExpression": "event_date >= '2024-03-01'", - "projectedFieldIds": "1,2,3,5,8", - "projectedFieldNames": "event_id,user_id,event_type,timestamp,page_url", - "resultDataFiles": 150, - "resultDeleteFiles": 5, - "totalFileSizeBytes": 1073741824, - "totalDataManifests": 12, - "totalDeleteManifests": 2, - "scannedDataManifests": 8, - "scannedDeleteManifests": 2, - "skippedDataManifests": 4, - "skippedDeleteManifests": 0, - "skippedDataFiles": 45, - "skippedDeleteFiles": 0, - "totalPlanningDurationMs": 250, - "equalityDeleteFiles": 3, - "positionalDeleteFiles": 2, - "indexedDeleteFiles": 0, - "totalDeleteFileSizeBytes": 52428800 - } - ] -} -``` - -#### List Metrics (Commit) - -**Request:** -```http -GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views?metricType=commit&operation=append&pageSize=2 -Authorization: Bearer -``` - -**Response:** -```json -{ - "nextPageToken": "eyJ0cyI6MTcwOTMzNzcwMDAwMCwiaWQiOiJjb21taXQtMDAyIn0=", - "metricType": "commit", - "reports": [ - { - "reportId": "commit-001-xyz789", - "catalogId": 12345, - "tableId": 67890, - "timestampMs": 1709337800000, - "principalName": "etl-service@example.com", - "requestId": "req-commit-001", - "otelTraceId": "xyz789abc123456789012345678901", - "otelSpanId": "abc123456789", - "snapshotId": 1234567890124, - "sequenceNumber": 42, - "operation": "append", - "addedDataFiles": 10, - "removedDataFiles": 0, - "totalDataFiles": 160, - "addedDeleteFiles": 0, - "removedDeleteFiles": 0, - "totalDeleteFiles": 5, - "addedEqualityDeleteFiles": 0, - "removedEqualityDeleteFiles": 0, - "addedPositionalDeleteFiles": 0, - "removedPositionalDeleteFiles": 0, - "addedRecords": 100000, - "removedRecords": 0, - "totalRecords": 15000000, - "addedFileSizeBytes": 104857600, - "removedFileSizeBytes": 0, - "totalFileSizeBytes": 1178599424, - "totalDurationMs": 5000, - "attempts": 1 - } - ] -} -``` - ---- - -## 5. Authorization - -### 5.1 Required Privileges - -This proposal introduces **new dedicated privileges** for reading observability data, following the principle of **separation of duties**. This ensures that: - -- Read-only audit/monitoring access does not require management permissions -- Monitoring tools can access metrics without requiring data read access -- Fine-grained access control is possible for different operational roles - -| Endpoint | Required Privilege | Scope | New Privilege? | -|----------|-------------------|-------|----------------| -| Query Events | `CATALOG_READ_EVENTS` | Catalog | **Yes** | -| List Scan Metrics | `TABLE_READ_METRICS` | Table | **Yes** | -| List Commit Metrics | `TABLE_READ_METRICS` | Table | **Yes** | - -### 5.2 New Privilege Definitions - -| Privilege | Scope | Description | -|-----------|-------|-------------| -| `CATALOG_READ_EVENTS` | Catalog | Read-only access to catalog events (audit log). Does not grant any management capabilities. | -| `TABLE_READ_METRICS` | Table | Read-only access to table scan and commit metrics. Does not grant access to table data. | - -### 5.3 Rationale: Separation of Duties - -Introducing dedicated read-only privileges enables proper **separation of duties**: - -| Use Case | Required Privilege | Why Not Reuse Existing? | -|----------|-------------------|------------------------| -| Security auditor reviewing catalog changes | `CATALOG_READ_EVENTS` | Should not require `CATALOG_MANAGE_METADATA` (management access) | -| Monitoring tool collecting table metrics | `TABLE_READ_METRICS` | Should not require `TABLE_READ_DATA` (data access) | -| Data analyst with table access | `TABLE_READ_DATA` implies `TABLE_READ_METRICS` | Users who can read data can also see metrics about their queries | -| Catalog admin | `CATALOG_MANAGE_METADATA` implies `CATALOG_READ_EVENTS` | Admins can see all events | - -### 5.4 Privilege Hierarchy - -The new privileges fit into the existing hierarchy as follows: - -``` -CATALOG_MANAGE_METADATA - └── CATALOG_READ_EVENTS (implied) - -TABLE_FULL_METADATA / TABLE_READ_DATA - └── TABLE_READ_METRICS (implied) -``` - -This means: -- Users with `CATALOG_MANAGE_METADATA` automatically have `CATALOG_READ_EVENTS` -- Users with `TABLE_READ_DATA` automatically have `TABLE_READ_METRICS` -- But the reverse is **not** true: `CATALOG_READ_EVENTS` does not grant management access, and `TABLE_READ_METRICS` does not grant data access - -### 5.5 Implementation Notes - -New privileges require: -1. Adding entries to `PolarisPrivilege` enum -2. Updating the privilege hierarchy in the authorizer -3. Adding privilege checks in the new API endpoints - ---- - -## 6. OpenAPI Schema - -> **Note:** The OpenAPI specifications below are embedded in this proposal for review context. Upon approval, these should be extracted into separate files for ease of processing and proper integration: -> - **Events API** → `spec/events-service.yml` (new dedicated service spec with base path `/api/events/v1/`) -> - **Metrics Reports API** → `spec/metrics-reports-service.yml` (new dedicated service spec with base path `/api/metrics-reports/v1/`) - -### 6.1 Events API - -Add the following to a new `spec/events-service.yml`: - -> **Note:** The Events API uses a dedicated `/api/events/v1/` namespace to avoid URI path clashes with the Iceberg REST Catalog events API until that spec is approved. The request/response schemas follow Iceberg Events API patterns for future compatibility. - -```yaml -paths: - /v1/{prefix}: - parameters: - - $ref: '#/components/parameters/prefix' - post: - tags: - - Catalog API - summary: Get events for changes to catalog objects - description: > - Returns a sequence of changes to catalog objects (tables, namespaces, views) - that allows clients to efficiently track metadata modifications without polling - individual resources. Consumers track their progress through a continuation-token, - enabling resumable synchronization after downtime or errors. - - This endpoint primarily supports use cases like catalog federation, workflow - triggering, and basic audit capabilities. - - Consumers should be prepared to handle 410 Gone responses when requested sequences - are outside the server's retention window. Consumers should also de-duplicate - received events based on the event's `event-id`. - operationId: getEvents - requestBody: - required: true - content: - application/json: - schema: - $ref: '#/components/schemas/QueryEventsRequest' - responses: - '200': - description: A sequence of change events to catalog objects - content: - application/json: - schema: - $ref: '#/components/schemas/QueryEventsResponse' - '400': - $ref: '#/components/responses/BadRequestErrorResponse' - '401': - $ref: '#/components/responses/UnauthorizedResponse' - '403': - $ref: '#/components/responses/ForbiddenResponse' - '410': - description: Gone - The requested offset is no longer available - content: - application/json: - schema: - $ref: '#/components/schemas/ErrorModel' - '503': - $ref: '#/components/responses/ServiceUnavailableResponse' - '5XX': - $ref: '#/components/responses/ServerErrorResponse' -``` - -### 6.2 Metrics API (New Metrics Reports Service) - -Add the following to a new `spec/metrics-reports-service.yml` (or extend existing management service): - -> **Note:** The metrics API uses `/api/metrics-reports/v1/` as the base path, separate from the management API. This reflects that metrics reports are read-only access to pre-populated data, not catalog management operations. - -```yaml -paths: - /catalogs/{catalogName}/namespaces/{namespace}/tables/{table}: - parameters: - - $ref: '#/components/parameters/catalogName' - - $ref: '#/components/parameters/namespace' - - $ref: '#/components/parameters/table' - get: - operationId: listTableMetrics - summary: List metrics for a table - description: > - Returns metrics reports for the specified table. The type of metrics - (scan or commit) must be specified via the required metricType parameter. - This unified endpoint supports future extensibility as new metric types - are added. - tags: - - Observability - parameters: - - name: metricType - in: query - required: true - description: Type of metrics to retrieve - schema: - type: string - enum: [scan, commit] - - name: pageToken - in: query - schema: - type: string - - name: pageSize - in: query - schema: - type: integer - minimum: 1 - maximum: 1000 - default: 100 - - name: snapshotId - in: query - schema: - type: integer - format: int64 - - name: principalName - in: query - schema: - type: string - - name: timestampFrom - in: query - schema: - type: integer - format: int64 - - name: timestampTo - in: query - schema: - type: integer - format: int64 - - name: operation - in: query - description: Filter by commit operation (only applicable when metricType=commit) - schema: - type: string - enum: [append, overwrite, delete, replace] - responses: - '200': - description: Paginated list of metrics reports - content: - application/json: - schema: - $ref: '#/components/schemas/ListMetricsResponse' - '400': - description: Bad request (e.g., missing metricType, invalid parameter combination) - '403': - description: Insufficient privileges - '404': - description: Table not found -``` - -### 6.3 Events API Schemas (Iceberg-Compatible) - -Add these schemas to `spec/events-service.yml`: - -```yaml -components: - schemas: - QueryEventsRequest: - type: object - properties: - continuation-token: - type: string - description: > - A continuation token to resume fetching events from a previous request. - If not provided, events are fetched from the beginning of the event log - subject to other filters. - page-size: - type: integer - format: int32 - description: > - The maximum number of events to return in a single response. - Servers may return less results than requested. - after-timestamp-ms: - type: integer - format: int64 - description: > - The timestamp in milliseconds to start consuming events from (inclusive). - operation-types: - type: array - items: - $ref: "#/components/schemas/OperationType" - description: Filter events by operation type. - catalog-objects-by-name: - type: array - items: - $ref: "#/components/schemas/CatalogObjectIdentifier" - description: > - Filter events by catalog objects referenced by name (namespaces, tables, views). - For namespaces, events for all containing objects are returned recursively. - catalog-objects-by-id: - type: array - items: - $ref: "#/components/schemas/CatalogObjectUuid" - description: Filter events by table/view UUIDs. - object-types: - type: array - items: - type: string - enum: [namespace, table, view] - description: Filter events by catalog object type. - custom-filters: - type: object - additionalProperties: true - description: Implementation-specific filter extensions. - - QueryEventsResponse: - type: object - required: - - next-page-token - - highest-processed-timestamp-ms - - events - properties: - next-page-token: - type: string - description: > - An opaque continuation token to fetch the next page of events. - highest-processed-timestamp-ms: - type: integer - format: int64 - description: > - The highest event timestamp processed when generating this response. - events: - type: array - items: - $ref: "#/components/schemas/Event" - - Event: - type: object - required: - - event-id - - request-id - - request-event-count - - timestamp-ms - - operation - properties: - event-id: - type: string - description: Unique ID of this event. Clients should deduplicate based on this ID. - request-id: - type: string - description: > - Opaque ID of the request this event belongs to. Events from the same - request share this ID. - request-event-count: - type: integer - description: > - Total number of events generated by this request. - timestamp-ms: - type: integer - format: int64 - description: Timestamp when this event occurred (epoch milliseconds). - actor: - type: object - additionalProperties: true - description: > - The actor who performed the operation (e.g., user, service account). - Content is implementation-specific. - operation: - type: object - description: The operation that was performed. - discriminator: - propertyName: operation-type - mapping: - create-table: "#/components/schemas/CreateTableOperation" - register-table: "#/components/schemas/RegisterTableOperation" - drop-table: "#/components/schemas/DropTableOperation" - update-table: "#/components/schemas/UpdateTableOperation" - rename-table: "#/components/schemas/RenameTableOperation" - create-view: "#/components/schemas/CreateViewOperation" - drop-view: "#/components/schemas/DropViewOperation" - update-view: "#/components/schemas/UpdateViewOperation" - rename-view: "#/components/schemas/RenameViewOperation" - create-namespace: "#/components/schemas/CreateNamespaceOperation" - update-namespace-properties: "#/components/schemas/UpdateNamespacePropertiesOperation" - drop-namespace: "#/components/schemas/DropNamespaceOperation" - custom: "#/components/schemas/CustomOperation" - - OperationType: - type: string - description: > - Defines the type of operation. Clients should ignore unknown operation types. - anyOf: - - type: string - enum: - - create-table - - register-table - - drop-table - - update-table - - rename-table - - create-view - - drop-view - - update-view - - rename-view - - create-namespace - - update-namespace-properties - - drop-namespace - - $ref: '#/components/schemas/CustomOperationType' - - CustomOperationType: - type: string - description: > - Custom operation type for catalog-specific extensions. - Must start with 'x-' followed by an implementation-specific identifier. - pattern: '^x-[a-zA-Z0-9-_.]+$' - - CustomOperation: - type: object - description: Extension point for catalog-specific operations (e.g., Polaris privileges). - required: - - operation-type - - custom-type - properties: - operation-type: - type: string - const: "custom" - custom-type: - $ref: '#/components/schemas/CustomOperationType' - identifier: - $ref: "#/components/schemas/TableIdentifier" - description: Table or view identifier this operation applies to, if applicable. - namespace: - $ref: "#/components/schemas/Namespace" - description: Namespace this operation applies to, if applicable. - table-uuid: - type: string - format: uuid - view-uuid: - type: string - format: uuid - additionalProperties: true - - CatalogObjectIdentifier: - type: array - items: - type: string - description: Reference to a named object in the catalog (namespace, table, or view). - example: ["accounting", "tax"] - - CatalogObjectUuid: - type: object - required: - - uuid - - type - properties: - uuid: - type: string - description: The UUID of the catalog object. - type: - type: string - enum: [table, view] -``` - -### 6.4 Metrics API Schemas (Polaris Management Service) - -Add these schemas to `spec/polaris-management-service.yml`: - -```yaml -components: - schemas: - ScanMetricsReport: - type: object - required: - - reportId - - catalogId - - tableId - - timestampMs - properties: - reportId: - type: string - catalogId: - type: integer - format: int64 - tableId: - type: integer - format: int64 - timestampMs: - type: integer - format: int64 - principalName: - type: string - requestId: - type: string - otelTraceId: - type: string - description: OpenTelemetry trace ID - otelSpanId: - type: string - description: OpenTelemetry span ID - snapshotId: - type: integer - format: int64 - schemaId: - type: integer - filterExpression: - type: string - projectedFieldIds: - type: string - projectedFieldNames: - type: string - resultDataFiles: - type: integer - format: int64 - resultDeleteFiles: - type: integer - format: int64 - totalFileSizeBytes: - type: integer - format: int64 - totalDataManifests: - type: integer - format: int64 - totalDeleteManifests: - type: integer - format: int64 - scannedDataManifests: - type: integer - format: int64 - scannedDeleteManifests: - type: integer - format: int64 - skippedDataManifests: - type: integer - format: int64 - skippedDeleteManifests: - type: integer - format: int64 - skippedDataFiles: - type: integer - format: int64 - skippedDeleteFiles: - type: integer - format: int64 - totalPlanningDurationMs: - type: integer - format: int64 - equalityDeleteFiles: - type: integer - format: int64 - positionalDeleteFiles: - type: integer - format: int64 - indexedDeleteFiles: - type: integer - format: int64 - totalDeleteFileSizeBytes: - type: integer - format: int64 - - ListMetricsResponse: - description: > - Polymorphic response for metrics queries. The concrete type is determined - by the metricType discriminator field. - oneOf: - - $ref: '#/components/schemas/ListScanMetricsResponse' - - $ref: '#/components/schemas/ListCommitMetricsResponse' - discriminator: - propertyName: metricType - mapping: - scan: '#/components/schemas/ListScanMetricsResponse' - commit: '#/components/schemas/ListCommitMetricsResponse' - - ListScanMetricsResponse: - type: object - required: - - metricType - - reports - properties: - nextPageToken: - type: string - description: Cursor for fetching the next page of results - metricType: - type: string - const: scan - description: Discriminator indicating this response contains scan metrics - reports: - type: array - description: Array of scan metrics reports - items: - $ref: '#/components/schemas/ScanMetricsReport' - - ListCommitMetricsResponse: - type: object - required: - - metricType - - reports - properties: - nextPageToken: - type: string - description: Cursor for fetching the next page of results - metricType: - type: string - const: commit - description: Discriminator indicating this response contains commit metrics - reports: - type: array - description: Array of commit metrics reports - items: - $ref: '#/components/schemas/CommitMetricsReport' - - CommitMetricsReport: - type: object - required: - - reportId - - catalogId - - tableId - - timestampMs - - snapshotId - - operation - properties: - reportId: - type: string - catalogId: - type: integer - format: int64 - tableId: - type: integer - format: int64 - timestampMs: - type: integer - format: int64 - principalName: - type: string - requestId: - type: string - otelTraceId: - type: string - otelSpanId: - type: string - snapshotId: - type: integer - format: int64 - sequenceNumber: - type: integer - format: int64 - operation: - type: string - description: Commit operation (append, overwrite, delete, replace) - addedDataFiles: - type: integer - format: int64 - removedDataFiles: - type: integer - format: int64 - totalDataFiles: - type: integer - format: int64 - addedDeleteFiles: - type: integer - format: int64 - removedDeleteFiles: - type: integer - format: int64 - totalDeleteFiles: - type: integer - format: int64 - addedEqualityDeleteFiles: - type: integer - format: int64 - removedEqualityDeleteFiles: - type: integer - format: int64 - addedPositionalDeleteFiles: - type: integer - format: int64 - removedPositionalDeleteFiles: - type: integer - format: int64 - addedRecords: - type: integer - format: int64 - removedRecords: - type: integer - format: int64 - totalRecords: - type: integer - format: int64 - addedFileSizeBytes: - type: integer - format: int64 - removedFileSizeBytes: - type: integer - format: int64 - totalFileSizeBytes: - type: integer - format: int64 - totalDurationMs: - type: integer - format: int64 - attempts: - type: integer -``` - ---- - -## 7. Implementation Notes - -### 7.1 Prerequisite: Extend Event Persistence Layer - -> **Important:** The current `PolarisPersistenceEventListener` in Apache Polaris only persists **two event types**: `AFTER_CREATE_TABLE` and `AFTER_CREATE_CATALOG`. All other events are ignored. For the Events API to be useful, the persistence layer must be extended to capture all relevant mutation events. - -#### Current State - -The existing event listener (`PolarisPersistenceEventListener.java`) has a limited switch statement: - -```java -public void onEvent(PolarisEvent event) { - switch (event.type()) { - case AFTER_CREATE_TABLE -> handleAfterCreateTable(event); - case AFTER_CREATE_CATALOG -> handleAfterCreateCatalog(event); - default -> { - // Other events not handled by this listener - } - } -} -``` - -#### Required Changes - -The persistence layer needs to be extended to capture all `AFTER_*` mutation events that should be exposed via the Events API: - -| Category | Events to Add | -|----------|---------------| -| **Table Operations** | `AFTER_UPDATE_TABLE`, `AFTER_DROP_TABLE`, `AFTER_RENAME_TABLE`, `AFTER_REGISTER_TABLE` | -| **View Operations** | `AFTER_CREATE_VIEW`, `AFTER_DROP_VIEW`, `AFTER_REPLACE_VIEW`, `AFTER_RENAME_VIEW` | -| **Namespace Operations** | `AFTER_CREATE_NAMESPACE`, `AFTER_UPDATE_NAMESPACE_PROPERTIES`, `AFTER_DROP_NAMESPACE` | -| **Catalog Operations** | `AFTER_DELETE_CATALOG`, `AFTER_UPDATE_CATALOG` | -| **Principal/Role Management** | `AFTER_CREATE_PRINCIPAL`, `AFTER_DELETE_PRINCIPAL`, `AFTER_ROTATE_CREDENTIALS`, `AFTER_CREATE_PRINCIPAL_ROLE`, `AFTER_DELETE_PRINCIPAL_ROLE`, `AFTER_CREATE_CATALOG_ROLE`, `AFTER_DELETE_CATALOG_ROLE` | -| **Grant Operations** | `AFTER_ADD_GRANT_TO_CATALOG_ROLE`, `AFTER_REVOKE_GRANT_FROM_CATALOG_ROLE`, `AFTER_ASSIGN_PRINCIPAL_ROLE`, `AFTER_REVOKE_PRINCIPAL_ROLE`, `AFTER_ASSIGN_CATALOG_ROLE_TO_PRINCIPAL_ROLE`, `AFTER_REVOKE_CATALOG_ROLE_FROM_PRINCIPAL_ROLE` | -| **Policy Operations** | `AFTER_CREATE_POLICY`, `AFTER_UPDATE_POLICY`, `AFTER_DROP_POLICY`, `AFTER_ATTACH_POLICY`, `AFTER_DETACH_POLICY` | - -**Note:** Read-only operations (`AFTER_LOAD_*`, `AFTER_LIST_*`, `AFTER_GET_*`, `AFTER_CHECK_EXISTS_*`) should **not** be persisted for the Events API as they do not represent catalog mutations. However, deployments requiring read audit may choose to persist these separately. - -#### Implementation Approach - -1. **Phase 1**: Extend `PolarisPersistenceEventListener` to handle all Iceberg-standard operations (tables, views, namespaces) -2. **Phase 2**: Add support for Polaris-specific operations (principals, roles, grants, policies) -3. **Phase 3**: Implement the Events REST API on top of the persisted data - -### 7.2 Database Queries - -The endpoints will query existing tables with appropriate filtering and pagination: - -```sql --- List events with cursor-based pagination -SELECT * FROM events -WHERE realm_id = ? - AND catalog_id = ? - AND (timestamp_ms, event_id) < (?, ?) -- cursor - AND event_type = ? -- optional filter - AND timestamp_ms >= ? -- optional filter - AND timestamp_ms < ? -- optional filter -ORDER BY timestamp_ms DESC, event_id DESC -LIMIT ?; - --- List scan metrics -SELECT * FROM scan_metrics_report -WHERE realm_id = ? - AND catalog_id = ? - AND table_id = ? - AND timestamp_ms >= ? - AND timestamp_ms < ? -ORDER BY timestamp_ms DESC, report_id DESC -LIMIT ?; -``` - -### 7.3 Recommended Indexes - -```sql --- Events indexes -CREATE INDEX IF NOT EXISTS idx_events_catalog_ts - ON events(realm_id, catalog_id, timestamp_ms DESC, event_id DESC); -CREATE INDEX IF NOT EXISTS idx_events_type - ON events(realm_id, catalog_id, event_type, timestamp_ms DESC); - --- Metrics indexes (may already exist) -CREATE INDEX IF NOT EXISTS idx_scan_report_lookup - ON scan_metrics_report(realm_id, catalog_id, table_id, timestamp_ms DESC); -CREATE INDEX IF NOT EXISTS idx_commit_report_lookup - ON commit_metrics_report(realm_id, catalog_id, table_id, timestamp_ms DESC); -``` - -### 7.4 Files to Modify - -| File | Changes | -|------|---------| -| `spec/events-service.yml` | **New file** - Events API paths and schemas | -| `spec/metrics-reports-service.yml` | **New file** - Metrics Reports API paths and schemas | -| `api/events-service/` | **New** - Generated Events API interfaces | -| `api/metrics-reports-service/` | **New** - Generated Metrics Reports API interfaces | -| `runtime/service/.../events/` | **New** - Events service implementation | -| `runtime/service/.../metrics/` | **New** - Metrics reports service implementation | -| `polaris-core/.../persistence/BasePersistence.java` | Add read methods | -| `persistence/relational-jdbc/.../JdbcBasePersistenceImpl.java` | Query implementations | - -### 7.5 Pagination Token Format - -Internal format (base64-encoded JSON, opaque to clients): - -```json -{ - "ts": 1709337612345, - "id": "550e8400-e29b-41d4-a716-446655440000" -} -``` - -### 7.6 Mapping PolarisEventType to Iceberg Events API - -Polaris internally uses a `PolarisEventType` enum that distinguishes between `BEFORE_*` and `AFTER_*` events for each operation (e.g., `BEFORE_CREATE_TABLE` and `AFTER_CREATE_TABLE`). This section explains how these internal events map to the Iceberg Events API. - -#### 7.6.1 Design Decision: Only AFTER Events are Exposed - -The Iceberg Events API represents **completed operations** that have been committed to the catalog. Therefore: - -| Internal Event Pattern | Exposed via API? | Rationale | -|------------------------|------------------|-----------| -| `AFTER_*` events | **Yes** | Represent successful, committed operations | -| `BEFORE_*` events | **No** | Represent intent, not outcome; may fail after firing | - -**Why not expose BEFORE events?** - -1. **Semantic mismatch**: The Iceberg Events API is designed for change data capture (CDC) and audit logs of *completed* changes. `BEFORE_*` events fire before validation and persistence, so they may represent operations that ultimately fail. - -2. **Consistency**: Exposing `BEFORE_*` events could lead to consumers seeing "phantom" operations that never actually occurred. - -3. **Use case alignment**: The primary use cases (audit, federation, workflow triggers) all require knowing what *actually happened*, not what was *attempted*. - -4. **Internal vs external**: `BEFORE_*` events serve internal purposes (request filtering, rate limiting, pre-validation hooks) and are not meaningful to external consumers. - -#### 7.6.2 Mapping AFTER Events to Operation Types - -The mapping follows a straightforward pattern - the `AFTER_` prefix is stripped and the remaining name maps to the Iceberg operation type: - -| Polaris `PolarisEventType` | Iceberg `operation-type` | Notes | -|----------------------------|--------------------------|-------| -| **Standard Iceberg Operations** | | | -| `AFTER_CREATE_TABLE` | `create-table` | Direct mapping | -| `AFTER_REGISTER_TABLE` | `register-table` | Direct mapping | -| `AFTER_DROP_TABLE` | `drop-table` | Direct mapping | -| `AFTER_UPDATE_TABLE` | `update-table` | Includes schema evolution, property changes | -| `AFTER_RENAME_TABLE` | `rename-table` | Direct mapping | -| `AFTER_CREATE_VIEW` | `create-view` | Direct mapping | -| `AFTER_DROP_VIEW` | `drop-view` | Direct mapping | -| `AFTER_REPLACE_VIEW` | `update-view` | View replacement maps to update | -| `AFTER_RENAME_VIEW` | `rename-view` | Direct mapping | -| `AFTER_CREATE_NAMESPACE` | `create-namespace` | Direct mapping | -| `AFTER_UPDATE_NAMESPACE_PROPERTIES` | `update-namespace-properties` | Direct mapping | -| `AFTER_DROP_NAMESPACE` | `drop-namespace` | Direct mapping | -| **Polaris Custom Operations** | | Use `custom` type with `polaris-*` | -| `AFTER_CREATE_CATALOG` | `custom` (`polaris-create-catalog`) | Catalog-level, not in Iceberg spec | -| `AFTER_DELETE_CATALOG` | `custom` (`polaris-delete-catalog`) | Catalog-level | -| `AFTER_CREATE_PRINCIPAL` | `custom` (`polaris-create-principal`) | Access management | -| `AFTER_DELETE_PRINCIPAL` | `custom` (`polaris-delete-principal`) | Access management | -| `AFTER_ROTATE_CREDENTIALS` | `custom` (`polaris-rotate-credentials`) | Security operation | -| `AFTER_CREATE_PRINCIPAL_ROLE` | `custom` (`polaris-create-principal-role`) | RBAC | -| `AFTER_CREATE_CATALOG_ROLE` | `custom` (`polaris-create-catalog-role`) | RBAC | -| `AFTER_ADD_GRANT_TO_CATALOG_ROLE` | `custom` (`polaris-grant-privilege`) | RBAC | -| `AFTER_REVOKE_GRANT_FROM_CATALOG_ROLE` | `custom` (`polaris-revoke-privilege`) | RBAC | -| `AFTER_CREATE_POLICY` | `custom` (`polaris-create-policy`) | Policy management | -| `AFTER_ATTACH_POLICY` | `custom` (`polaris-attach-policy`) | Policy management | -| `AFTER_CREATE_GENERIC_TABLE` | `custom` (`polaris-create-generic-table`) | Generic table support | - -#### 7.6.3 Read-Only Operations: Not Exposed - -Events for read-only operations are **not exposed** via the Events API because they do not represent catalog mutations: - -| Excluded Event Types | Reason | -|----------------------|--------| -| `AFTER_GET_CATALOG`, `AFTER_LIST_CATALOGS` | Read-only, no state change | -| `AFTER_LOAD_TABLE`, `AFTER_LIST_TABLES`, `AFTER_CHECK_EXISTS_TABLE` | Read-only | -| `AFTER_LOAD_NAMESPACE_METADATA`, `AFTER_LIST_NAMESPACES` | Read-only | -| `AFTER_LOAD_VIEW`, `AFTER_LIST_VIEWS` | Read-only | -| `AFTER_GET_CONFIG`, `AFTER_LOAD_CREDENTIALS` | Configuration/credential reads | -| `AFTER_LIST_*` (all list operations) | Read-only enumeration | - -#### 7.6.4 Implementation: Event Filtering - -The event persistence layer should filter events before storing them for the Events API: - -```java -// Events eligible for the REST API (completed mutations only) -private static final Set EXPOSED_EVENT_TYPES = Set.of( - // Standard Iceberg operations - AFTER_CREATE_TABLE, AFTER_UPDATE_TABLE, AFTER_DROP_TABLE, - AFTER_RENAME_TABLE, AFTER_REGISTER_TABLE, - AFTER_CREATE_VIEW, AFTER_DROP_VIEW, AFTER_REPLACE_VIEW, AFTER_RENAME_VIEW, - AFTER_CREATE_NAMESPACE, AFTER_UPDATE_NAMESPACE_PROPERTIES, AFTER_DROP_NAMESPACE, - // Polaris custom operations - AFTER_CREATE_CATALOG, AFTER_DELETE_CATALOG, AFTER_UPDATE_CATALOG, - AFTER_CREATE_PRINCIPAL, AFTER_DELETE_PRINCIPAL, AFTER_UPDATE_PRINCIPAL, - AFTER_ROTATE_CREDENTIALS, AFTER_RESET_CREDENTIALS, - AFTER_CREATE_PRINCIPAL_ROLE, AFTER_DELETE_PRINCIPAL_ROLE, - AFTER_CREATE_CATALOG_ROLE, AFTER_DELETE_CATALOG_ROLE, - AFTER_ADD_GRANT_TO_CATALOG_ROLE, AFTER_REVOKE_GRANT_FROM_CATALOG_ROLE, - AFTER_ASSIGN_PRINCIPAL_ROLE, AFTER_REVOKE_PRINCIPAL_ROLE, - AFTER_ASSIGN_CATALOG_ROLE_TO_PRINCIPAL_ROLE, AFTER_REVOKE_CATALOG_ROLE_FROM_PRINCIPAL_ROLE, - AFTER_CREATE_POLICY, AFTER_UPDATE_POLICY, AFTER_DROP_POLICY, - AFTER_ATTACH_POLICY, AFTER_DETACH_POLICY, - AFTER_CREATE_GENERIC_TABLE, AFTER_DROP_GENERIC_TABLE -); - -public boolean shouldPersistForEventsApi(PolarisEventType eventType) { - return EXPOSED_EVENT_TYPES.contains(eventType); -} -``` - ---- - -## Appendix A: Polaris Internal Event Types Reference - -Polaris internal event types are categorized by code ranges. These are mapped to Iceberg-compatible operation types when exposed via the Events API: - -| Range | Category | Internal Event Type | Iceberg Operation Type | -|-------|----------|---------------------|------------------------| -| 100-109 | Catalog | `AFTER_CREATE_CATALOG` | `custom` (`polaris-create-catalog`) | -| | | `AFTER_DELETE_CATALOG` | `custom` (`polaris-delete-catalog`) | -| 200-217 | Catalog Role | `AFTER_CREATE_CATALOG_ROLE` | `custom` (`polaris-create-catalog-role`) | -| | | `AFTER_ADD_GRANT_TO_CATALOG_ROLE` | `custom` (`polaris-grant-privilege`) | -| 300-319 | Principal | `AFTER_CREATE_PRINCIPAL` | `custom` (`polaris-create-principal`) | -| | | `AFTER_ROTATE_CREDENTIALS` | `custom` (`polaris-rotate-credentials`) | -| 400-417 | Principal Role | `AFTER_CREATE_PRINCIPAL_ROLE` | `custom` (`polaris-create-principal-role`) | -| 500-511 | Namespace | `AFTER_CREATE_NAMESPACE` | `create-namespace` | -| | | `AFTER_UPDATE_NAMESPACE_PROPERTIES` | `update-namespace-properties` | -| | | `AFTER_DROP_NAMESPACE` | `drop-namespace` | -| 600-617 | Table | `AFTER_CREATE_TABLE` | `create-table` | -| | | `AFTER_UPDATE_TABLE` | `update-table` | -| | | `AFTER_DROP_TABLE` | `drop-table` | -| | | `AFTER_RENAME_TABLE` | `rename-table` | -| | | `AFTER_REGISTER_TABLE` | `register-table` | -| 700-715 | View | `AFTER_CREATE_VIEW` | `create-view` | -| | | `AFTER_UPDATE_VIEW` | `update-view` | -| | | `AFTER_DROP_VIEW` | `drop-view` | -| | | `AFTER_RENAME_VIEW` | `rename-view` | -| 1200-1215 | Policy | `AFTER_CREATE_POLICY` | `custom` (`polaris-create-policy`) | -| | | `AFTER_ATTACH_POLICY` | `custom` (`polaris-attach-policy`) | -| 1300-1307 | Generic Table | `AFTER_CREATE_GENERIC_TABLE` | `custom` (`polaris-create-generic-table`) | - ---- - -## 8. Iceberg Events API Alignment - -This section documents the alignment with the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584) and explains the rationale for design decisions. - -### 8.1 Why Align with Iceberg Events API? - -The Iceberg Events API is an emerging specification that is nearing consensus in the Apache Iceberg community. Aligning Polaris with this specification provides: - -1. **Ecosystem Compatibility**: Clients built for the Iceberg Events API will work with Polaris without modification -2. **Future-Proofing**: Avoids breaking changes when the Iceberg spec is finalized -3. **Tooling Interoperability**: Monitoring tools, federation services, and workflow triggers can work across Iceberg-compatible catalogs -4. **Reduced Cognitive Load**: Developers familiar with Iceberg don't need to learn a new API - -### 8.2 Key Design Decisions from Iceberg Spec - -| Decision | Iceberg Spec Approach | Rationale | -|----------|----------------------|-----------| -| **HTTP Method** | `POST` (not `GET`) | Allows complex filtering with arrays and nested objects in request body | -| **API Path** | `/v1/{prefix}/events` | Part of Iceberg REST Catalog, not a separate management API | -| **Pagination** | `continuation-token` | Opaque cursor that encodes server state; resumable after downtime | -| **Event Structure** | Operation-centric with discriminator | Each event contains a typed `operation` with operation-specific fields | -| **Operation Types** | Standardized enum + custom prefix extensions | Standard types for Iceberg operations; custom prefix for catalog-specific extensions | -| **Actor Field** | Generic object (implementation-specific) | Flexibility for different auth models (users, service accounts, etc.) | -| **Error Handling** | `410 Gone` for expired offsets | Explicit signal when continuation token is outside retention window | - -### 8.3 Polaris-Specific Extensions - -Polaris extends the Iceberg Events API using the `custom` operation type with `polaris-*` prefixed custom types: - -| Custom Type | Polaris Event | Description | -|-------------|---------------|-------------| -| `polaris-create-catalog` | `AFTER_CREATE_CATALOG` | Catalog created | -| `polaris-create-catalog-role` | `AFTER_CREATE_CATALOG_ROLE` | Catalog role created | -| `polaris-grant-privilege` | `AFTER_ADD_GRANT_TO_CATALOG_ROLE` | Privilege granted to role | -| `polaris-revoke-privilege` | `AFTER_REMOVE_GRANT_FROM_CATALOG_ROLE` | Privilege revoked | -| `polaris-create-principal` | `AFTER_CREATE_PRINCIPAL` | Principal created | -| `polaris-rotate-credentials` | `AFTER_ROTATE_CREDENTIALS` | Credentials rotated | -| `polaris-create-policy` | `AFTER_CREATE_POLICY` | Policy created | -| `polaris-attach-policy` | `AFTER_ATTACH_POLICY` | Policy attached to resource | - -### 8.4 Mapping Polaris Internal Events to Iceberg Operations - -| Polaris Event Type | Iceberg Operation Type | -|-------------------|------------------------| -| `AFTER_CREATE_TABLE` | `create-table` | -| `AFTER_UPDATE_TABLE` | `update-table` | -| `AFTER_DROP_TABLE` | `drop-table` | -| `AFTER_RENAME_TABLE` | `rename-table` | -| `AFTER_REGISTER_TABLE` | `register-table` | -| `AFTER_CREATE_VIEW` | `create-view` | -| `AFTER_UPDATE_VIEW` / `AFTER_REPLACE_VIEW` | `update-view` | -| `AFTER_DROP_VIEW` | `drop-view` | -| `AFTER_RENAME_VIEW` | `rename-view` | -| `AFTER_CREATE_NAMESPACE` | `create-namespace` | -| `AFTER_UPDATE_NAMESPACE_PROPERTIES` | `update-namespace-properties` | -| `AFTER_DROP_NAMESPACE` | `drop-namespace` | -| Other Polaris events | `custom` with `polaris-*` type | - -### 8.5 References - -- **Iceberg Events API Proposal**: [Google Doc](https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit) -- **Iceberg Events API PR**: [apache/iceberg#12584](https://github.com/apache/iceberg/pull/12584) -- **Iceberg REST Catalog Spec**: [rest-catalog-open-api.yaml](https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml) - ---- - -## Open Questions - -1. **Aggregations**: Are aggregated metrics views needed (e.g., daily summaries)? -2. **Event Retention**: What is the default retention period for events? Should it be configurable? -3. **Consistency Guarantees**: What ordering and delivery guarantees should Polaris provide for the Events API? - -## Resolved Questions - -1. ~~**Privileges**: Use existing privileges or introduce new `READ_EVENTS`/`READ_METRICS`?~~ - - **Resolution**: Introduce new dedicated privileges (`CATALOG_READ_EVENTS`, `TABLE_READ_METRICS`) to support separation of duties. See [Section 5](#5-authorization) for details. - ---- - -## References - -- Database schema: `persistence/relational-jdbc/src/main/resources/postgres/schema-v4.sql` -- Event types: `runtime/service/src/main/java/org/apache/polaris/service/events/PolarisEventType.java` -- Metrics persistence: `runtime/service/src/main/java/org/apache/polaris/service/reporting/PersistingMetricsReporter.java` -- Iceberg Events API PR: https://github.com/apache/iceberg/pull/12584 -- Iceberg Events API Design Doc: https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit - From c8da90d61e5333a4c6efc0de003edb28c5c2b47c Mon Sep 17 00:00:00 2001 From: Anand Kumar Sankaran Date: Fri, 13 Mar 2026 15:56:51 -0700 Subject: [PATCH 14/15] Review comments - split into two proposals --- .../unreleased/proposals/events-rest-api.md | 838 ++++++++++++++++++ .../proposals/table-metrics-rest-api.md | 720 +++++++++++++++ 2 files changed, 1558 insertions(+) create mode 100644 site/content/in-dev/unreleased/proposals/events-rest-api.md create mode 100644 site/content/in-dev/unreleased/proposals/table-metrics-rest-api.md diff --git a/site/content/in-dev/unreleased/proposals/events-rest-api.md b/site/content/in-dev/unreleased/proposals/events-rest-api.md new file mode 100644 index 0000000000..2d52329452 --- /dev/null +++ b/site/content/in-dev/unreleased/proposals/events-rest-api.md @@ -0,0 +1,838 @@ +--- +title: Events REST API +linkTitle: Events REST API +weight: 100 +--- + + +# Proposal: REST API for Querying Catalog Events + +**Author:** Anand Sankaran +**Date:** 2026-03-02 +**Status:** Draft Proposal +**Target:** Apache Polaris + +--- + +## Abstract + +This proposal defines a REST API endpoint for querying catalog events from Apache Polaris. The endpoint exposes data already being persisted via the existing JDBC persistence model (`events` table) and follows established Polaris API patterns. + +**Note:** The Events API in this proposal is designed to align with the emerging [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584), which is nearing consensus in the Apache Iceberg community. This ensures forward compatibility and consistency with the broader Iceberg ecosystem. + +--- + +## Table of Contents + +1. [Motivation](#1-motivation) +2. [Use Cases](#2-use-cases) +3. [Design Principles](#3-design-principles) +4. [API Specification](#4-api-specification) +5. [Authorization](#5-authorization) +6. [OpenAPI Schema](#6-openapi-schema) +7. [Implementation Notes](#7-implementation-notes) +8. [Iceberg Events API Alignment](#8-iceberg-events-api-alignment) + +--- + +## 1. Motivation + +Apache Polaris currently persists catalog events to the database, but provides no REST API to query this data. Users must access the database directly to retrieve audit information. + +Adding a read-only REST endpoint enables: +- Programmatic access to events without database credentials +- Integration with monitoring dashboards and alerting systems +- Consistent authorization via Polaris RBAC +- Pagination and filtering without writing SQL + +--- + +## 2. Use Cases + +### 2.1 Audit & Compliance +- Track who created/dropped/modified catalog objects +- Monitor administrative actions (credential rotation, grant changes) +- Generate compliance reports for access patterns + +### 2.2 Catalog Federation +- Synchronize catalog state across distributed systems +- Enable CDC (Change Data Capture) for downstream consumers +- Support workflow triggering based on catalog changes + +### 2.3 Debugging & Troubleshooting +- Investigate catalog state changes over time +- Correlate events with distributed traces (`otel_trace_id`, `otel_span_id`) +- Trace operations via `request_id` + +--- + +## 3. Design Principles + +| Principle | Rationale | +|-----------|-----------| +| **Iceberg Events API alignment** | Events API follows the [Iceberg Events API spec](https://github.com/apache/iceberg/pull/12584) for ecosystem compatibility | +| **POST for complex filtering** | Events API uses POST with request body (per Iceberg spec) to support complex filters (arrays, nested objects) | +| **Read-only semantics** | All endpoints are read-only; events are written via existing flows | +| **Consistent pagination** | Follow `continuation-token` pattern (Iceberg) | +| **Flexible filtering** | Time ranges, operation types, catalog objects - common query patterns | +| **RBAC integration** | Leverage existing Polaris authorization model | +| **Realm handling** | Process Polaris realms consistently with existing APIs; realm context is derived from the authenticated principal | +| **Stable envelope** | Polaris-owned stable envelope fields for resilient client integrations; type-specific payloads are versioned independently | + +### 3.1 Stable Envelope + +To reduce coupling to any single upstream schema (e.g., Iceberg) and keep client integrations resilient, the Events API SHOULD return records using a **stable envelope**: a small, Polaris-owned set of top-level fields that remain consistent across all event types, plus a versioned payload for the type-specific body. + +The envelope enables clients to reliably paginate, deduplicate, and correlate records (request IDs / trace IDs) without needing to understand the full payload schema. Type-specific details (Iceberg-aligned or Polaris-specific) live under `payload`, identified by `payload.type` and `payload.version`. + +#### Envelope Fields (Conceptual) + +| Field | Description | +|-------|-------------| +| `id` | Unique identifier for the event | +| `timestampMs` | Event timestamp (epoch milliseconds) | +| `catalog` | Catalog identifier | +| `realm` | Realm identifier (if applicable) | +| `actor` | Principal/service + optional client metadata | +| `request` | Request context (requestId, trace/span IDs) | +| `object` | Resource identity: namespace/table/view identifiers and optional UUID/snapshotId | +| `payload` | `{ type, version, data, extensions? }` | + +#### Compatibility / Evolution Rules + +1. **Envelope is additive-only**: New envelope fields may be added as optional; existing envelope fields MUST NOT change meaning or type, and MUST NOT be removed. + +2. **Breaking changes require versioning at the API boundary**: Any breaking envelope change requires a new major API version (e.g., `/v2/...`) rather than changing `/v1`. + +3. **Payload is independently versioned**: `payload.type` selects the schema family (e.g., `iceberg.events`, `polaris.events`), and `payload.version` increments on breaking payload changes. + +4. **Unknown payloads are safe to ignore**: Clients MUST treat unknown `payload.type` or higher `payload.version` as opaque and continue to operate using envelope fields (e.g., still paginate / display metadata). + +5. **Payload changes within a version are additive-only**: Within a given `payload.type` + `payload.version`, new fields may be added as optional; removals/renames/type changes require a new `payload.version`. + +6. **Flattening is a presentation choice**: The default representation SHOULD keep domain-specific structures nested under `payload.data`; alternative "flattened" views (if needed) should be offered as an explicit, separately-versioned representation rather than redefining the canonical schema. + +--- + +## 4. API Specification + +### 4.1 Endpoint Summary + +| Method | Path | Description | +|--------|------|-------------| +| POST | `/api/events/v1/{prefix}` | Query events for a catalog | + +> **Note:** The Events API uses a dedicated `/api/events/v1/` namespace to avoid URI path clashes with the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584) until that spec is approved. The API design follows Iceberg Events API patterns for future compatibility. + +### 4.2 Path Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `prefix` | string | Catalog prefix (typically the catalog name) | + +### 4.3 Events API (Iceberg-Compatible) + +The Events API follows the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584) for ecosystem compatibility. + +**Scope:** This endpoint is **catalog-scoped** (via `{prefix}`) and returns events describing changes to **catalog objects** (namespaces, tables, views) within that catalog. Realm-level administrative events (for example, principal lifecycle and credential rotation) are **out of scope** for `/api/events/v1/{prefix}` and would require a separate realm-scoped API surface. + +Key design decisions from the Iceberg spec: + +- **POST method**: Allows complex filtering with arrays and nested objects in the request body +- **Continuation token**: Opaque cursor for resumable pagination +- **Operation-centric model**: Events are structured around operations (create-table, update-table, etc.) +- **Custom extensions**: Support for `polaris-` prefixed custom operation types for Polaris catalog-scoped extensions + +#### Request Body (`QueryEventsRequest`) + +> **Note:** This request schema matches the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584) for ecosystem compatibility. + +| Property | Type | Required | Description | +|----------|------|----------|-------------| +| `continuation-token` | string | No | Opaque cursor to resume fetching from previous request | +| `page-size` | integer | No | Maximum events per page (server may return fewer) | +| `after-timestamp-ms` | long | No | Filter: events after this timestamp (inclusive) | +| `operation-types` | array[string] | No | Filter by operation types (see below) | +| `catalog-objects-by-name` | array[array[string]] | No | Filter by namespace/table/view names | +| `catalog-objects-by-id` | array[object] | No | Filter by table/view UUIDs | +| `object-types` | array[string] | No | Filter by object type: `namespace`, `table`, `view` | +| `custom-filters` | object | No | Implementation-specific filter extensions | + +#### Standard Operation Types + +| Operation Type | Description | +|----------------|-------------| +| `create-table` | Table created and committed | +| `register-table` | Existing table registered in catalog | +| `drop-table` | Table dropped | +| `update-table` | Table metadata updated | +| `rename-table` | Table renamed | +| `create-view` | View created | +| `drop-view` | View dropped | +| `update-view` | View updated | +| `rename-view` | View renamed | +| `create-namespace` | Namespace created | +| `update-namespace-properties` | Namespace properties updated | +| `drop-namespace` | Namespace dropped | + +#### Polaris Custom Operation Types + +For Polaris-specific events not covered by the Iceberg spec, use the `polaris-` prefix. To keep `/api/events/v1/{prefix}` catalog-scoped, these custom operations must be **catalog-scoped** as well. + +| Custom Operation Type | Description | +|----------------------|-------------| +| `polaris-create-catalog-role` | Catalog role created | +| `polaris-grant-privilege` | Privilege granted | +| `polaris-revoke-privilege` | Privilege revoked | + +### 4.4 Example Requests and Responses + +#### Query Events (Iceberg-Compatible) + +**Request:** +```http +POST /api/events/v1/my-catalog +Authorization: Bearer +Content-Type: application/json + +{ + "page-size": 2, + "operation-types": ["create-table", "update-table"], + "after-timestamp-ms": 1709251200000, + "catalog-objects-by-name": [ + ["analytics", "events"] + ], + "object-types": ["table"] +} +``` + +**Response:** +```json +{ + "next-page-token": "eyJ0cyI6MTcwOTMzNzYxMjM0NSwiaWQiOiI1NTBlODQwMCJ9", + "highest-processed-timestamp-ms": 1709337612345, + "events": [ + { + "event-id": "550e8400-e29b-41d4-a716-446655440000", + "request-id": "req-12345", + "request-event-count": 1, + "timestamp-ms": 1709337612345, + "actor": { + "principal": "admin@example.com", + "client-ip": "192.168.1.100" + }, + "operation": { + "operation-type": "create-table", + "identifier": { + "namespace": ["analytics", "events"], + "name": "page_views" + }, + "table-uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "updates": [ + {"action": "assign-uuid", "uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890"}, + {"action": "set-current-schema", "schema-id": 0}, + {"action": "set-default-spec", "spec-id": 0} + ] + } + }, + { + "event-id": "661f9511-f30c-52e5-b827-557766551111", + "request-id": "req-12346", + "request-event-count": 1, + "timestamp-ms": 1709337500000, + "actor": { + "principal": "etl-service@example.com" + }, + "operation": { + "operation-type": "update-table", + "identifier": { + "namespace": ["analytics", "events"], + "name": "user_actions" + }, + "table-uuid": "b2c3d4e5-f6a7-8901-bcde-f23456789012", + "updates": [ + {"action": "add-snapshot", "snapshot-id": 123456789} + ], + "requirements": [ + {"type": "assert-table-uuid", "uuid": "b2c3d4e5-f6a7-8901-bcde-f23456789012"} + ] + } + } + ] +} +``` + +#### Query Events with Custom (Catalog-Scoped) Polaris Operations + +**Request:** +```http +POST /api/events/v1/my-catalog +Authorization: Bearer +Content-Type: application/json + +{ + "page-size": 10, + "operation-types": ["polaris-grant-privilege", "polaris-create-catalog-role"] +} +``` + +**Response:** +```json +{ + "next-page-token": "eyJ0cyI6MTcwOTMzODAwMDAwMH0=", + "highest-processed-timestamp-ms": 1709338000000, + "events": [ + { + "event-id": "772f0622-g41d-63f6-c938-668877662222", + "request-id": "req-admin-001", + "request-event-count": 1, + "timestamp-ms": 1709338000000, + "actor": { + "principal": "security-admin@example.com" + }, + "operation": { + "operation-type": "custom", + "custom-type": "polaris-grant-privilege", + "identifier": { + "namespace": ["analytics", "events"], + "name": "page_views" + }, + "table-uuid": "a1b2c3d4-e5f6-7890-abcd-ef1234567890", + "privilege": "TABLE_READ_DATA", + "grantee": "data-analyst-role" + } + } + ] +} +``` + +--- + +## 5. Authorization + +### 5.1 Required Privileges + +This proposal introduces a **new dedicated privilege** for reading events, following the principle of **separation of duties**. This ensures that: + +- Read-only audit/monitoring access does not require management permissions +- Fine-grained access control is possible for different operational roles + +| Endpoint | Required Privilege | Scope | New Privilege? | +|----------|-------------------|-------|----------------| +| Query Events | `CATALOG_READ_EVENTS` | Catalog | **Yes** | + +### 5.2 New Privilege Definition + +| Privilege | Scope | Description | +|-----------|-------|-------------| +| `CATALOG_READ_EVENTS` | Catalog | Read-only access to catalog events (audit log). Does not grant any management capabilities. | + +### 5.3 Rationale: Separation of Duties + +Introducing a dedicated read-only privilege enables proper **separation of duties**: + +| Use Case | Required Privilege | Why Not Reuse Existing? | +|----------|-------------------|------------------------| +| Security auditor reviewing catalog changes | `CATALOG_READ_EVENTS` | Should not require `CATALOG_MANAGE_METADATA` (management access) | +| Catalog admin | `CATALOG_MANAGE_METADATA` implies `CATALOG_READ_EVENTS` | Admins can see all events | + +### 5.4 Privilege Hierarchy + +The new privilege fits into the existing hierarchy as follows: + +``` +CATALOG_MANAGE_METADATA + └── CATALOG_READ_EVENTS (implied) +``` + +This means: +- Users with `CATALOG_MANAGE_METADATA` automatically have `CATALOG_READ_EVENTS` +- But the reverse is **not** true: `CATALOG_READ_EVENTS` does not grant management access + +### 5.5 Implementation Notes + +New privileges require: +1. Adding entries to `PolarisPrivilege` enum +2. Updating the privilege hierarchy in the authorizer +3. Adding privilege checks in the new API endpoint + +--- + +## 6. OpenAPI Schema + +> **Note:** The OpenAPI specifications below are embedded in this proposal for review context. Upon approval, these should be extracted into separate files for ease of processing and proper integration: +> - **Events API** → `spec/events-service.yml` (new dedicated service spec with base path `/api/events/v1/`) + +### 6.1 Events API + +Add the following to a new `spec/events-service.yml`: + +> **Note:** The Events API uses a dedicated `/api/events/v1/` namespace to avoid URI path clashes with the Iceberg REST Catalog events API until that spec is approved. The request/response schemas follow Iceberg Events API patterns for future compatibility. + +```yaml +paths: + /v1/{prefix}: + parameters: + - $ref: '#/components/parameters/prefix' + post: + tags: + - Catalog API + summary: Get events for changes to catalog objects + description: > + Returns a sequence of changes to catalog objects (tables, namespaces, views) + that allows clients to efficiently track metadata modifications without polling + individual resources. Consumers track their progress through a continuation-token, + enabling resumable synchronization after downtime or errors. + + This endpoint primarily supports use cases like catalog federation, workflow + triggering, and basic audit capabilities. + + Consumers should be prepared to handle 410 Gone responses when requested sequences + are outside the server's retention window. Consumers should also de-duplicate + received events based on the event's `event-id`. + operationId: getEvents + requestBody: + required: true + content: + application/json: + schema: + $ref: '#/components/schemas/QueryEventsRequest' + responses: + '200': + description: A sequence of change events to catalog objects + content: + application/json: + schema: + $ref: '#/components/schemas/QueryEventsResponse' + '400': + $ref: '#/components/responses/BadRequestErrorResponse' + '401': + $ref: '#/components/responses/UnauthorizedResponse' + '403': + $ref: '#/components/responses/ForbiddenResponse' + '410': + description: Gone - The requested offset is no longer available + content: + application/json: + schema: + $ref: '#/components/schemas/ErrorModel' + '503': + $ref: '#/components/responses/ServiceUnavailableResponse' + '5XX': + $ref: '#/components/responses/ServerErrorResponse' +``` + +### 6.2 Events API Schemas (Iceberg-Compatible) + +Add these schemas to `spec/events-service.yml`: + +```yaml +components: + schemas: + QueryEventsRequest: + type: object + properties: + continuation-token: + type: string + description: > + A continuation token to resume fetching events from a previous request. + If not provided, events are fetched from the beginning of the event log + subject to other filters. + page-size: + type: integer + format: int32 + description: > + The maximum number of events to return in a single response. + Servers may return less results than requested. + after-timestamp-ms: + type: integer + format: int64 + description: > + The timestamp in milliseconds to start consuming events from (inclusive). + operation-types: + type: array + items: + $ref: "#/components/schemas/OperationType" + description: Filter events by operation type. + catalog-objects-by-name: + type: array + items: + $ref: "#/components/schemas/CatalogObjectIdentifier" + description: > + Filter events by catalog objects referenced by name (namespaces, tables, views). + For namespaces, events for all containing objects are returned recursively. + catalog-objects-by-id: + type: array + items: + $ref: "#/components/schemas/CatalogObjectUuid" + description: Filter events by table/view UUIDs. + object-types: + type: array + items: + type: string + enum: [namespace, table, view] + description: Filter events by catalog object type. + custom-filters: + type: object + additionalProperties: true + description: Implementation-specific filter extensions. + + QueryEventsResponse: + type: object + required: + - next-page-token + - highest-processed-timestamp-ms + - events + properties: + next-page-token: + type: string + description: > + An opaque continuation token to fetch the next page of events. + highest-processed-timestamp-ms: + type: integer + format: int64 + description: > + The highest event timestamp processed when generating this response. + events: + type: array + items: + $ref: "#/components/schemas/Event" + + Event: + type: object + required: + - event-id + - request-id + - request-event-count + - timestamp-ms + - operation + properties: + event-id: + type: string + description: Unique ID of this event. Clients should deduplicate based on this ID. + request-id: + type: string + description: > + Opaque ID of the request this event belongs to. Events from the same + request share this ID. + request-event-count: + type: integer + description: > + Total number of events generated by this request. + timestamp-ms: + type: integer + format: int64 + description: Timestamp when this event occurred (epoch milliseconds). + actor: + type: object + additionalProperties: true + description: > + The actor who performed the operation (e.g., user, service account). + Content is implementation-specific. + operation: + type: object + description: The operation that was performed. + discriminator: + propertyName: operation-type + mapping: + create-table: "#/components/schemas/CreateTableOperation" + register-table: "#/components/schemas/RegisterTableOperation" + drop-table: "#/components/schemas/DropTableOperation" + update-table: "#/components/schemas/UpdateTableOperation" + rename-table: "#/components/schemas/RenameTableOperation" + create-view: "#/components/schemas/CreateViewOperation" + drop-view: "#/components/schemas/DropViewOperation" + update-view: "#/components/schemas/UpdateViewOperation" + rename-view: "#/components/schemas/RenameViewOperation" + create-namespace: "#/components/schemas/CreateNamespaceOperation" + update-namespace-properties: "#/components/schemas/UpdateNamespacePropertiesOperation" + drop-namespace: "#/components/schemas/DropNamespaceOperation" + custom: "#/components/schemas/CustomOperation" + + OperationType: + type: string + description: > + Defines the type of operation. Clients should ignore unknown operation types. + anyOf: + - type: string + enum: + - create-table + - register-table + - drop-table + - update-table + - rename-table + - create-view + - drop-view + - update-view + - rename-view + - create-namespace + - update-namespace-properties + - drop-namespace + - $ref: '#/components/schemas/CustomOperationType' + + CustomOperationType: + type: string + description: > + Custom operation type for Polaris catalog-specific extensions. + Must start with 'polaris-' followed by an implementation-specific identifier. + pattern: '^polaris-[a-zA-Z0-9-_.]+$' + + CustomOperation: + type: object + description: Extension point for catalog-specific operations (e.g., Polaris privileges). + required: + - operation-type + - custom-type + properties: + operation-type: + type: string + const: "custom" + custom-type: + $ref: '#/components/schemas/CustomOperationType' + identifier: + $ref: "#/components/schemas/TableIdentifier" + description: Table or view identifier this operation applies to, if applicable. + namespace: + $ref: "#/components/schemas/Namespace" + description: Namespace this operation applies to, if applicable. + table-uuid: + type: string + format: uuid + view-uuid: + type: string + format: uuid + additionalProperties: true + + CatalogObjectIdentifier: + type: array + items: + type: string + description: Reference to a named object in the catalog (namespace, table, or view). + example: ["accounting", "tax"] + + CatalogObjectUuid: + type: object + required: + - uuid + - type + properties: + uuid: + type: string + description: The UUID of the catalog object. + type: + type: string + enum: [table, view] +``` + +--- + +## 7. Implementation Notes + +### 7.1 Prerequisite: Extend Event Persistence Layer + +> **Important:** The current `PolarisPersistenceEventListener` in Apache Polaris only persists **two event types**: `AFTER_CREATE_TABLE` and `AFTER_CREATE_CATALOG`. All other events are ignored. For the Events API to be useful, the persistence layer must be extended to capture all relevant mutation events. + +#### Current State + +The existing event listener (`PolarisPersistenceEventListener.java`) has a limited switch statement: + +```java +public void onEvent(PolarisEvent event) { + switch (event.type()) { + case AFTER_CREATE_TABLE -> handleAfterCreateTable(event); + case AFTER_CREATE_CATALOG -> handleAfterCreateCatalog(event); + default -> { + // Other events not handled by this listener + } + } +} +``` + +#### Required Changes + +The persistence layer needs to be extended to capture all `AFTER_*` mutation events that should be exposed via the Events API. For v1, this is limited to **catalog-object mutations** (tables, views, namespaces) within a catalog. + +| Category | Events to Add | +|----------|---------------| +| **Table Operations** | `AFTER_UPDATE_TABLE`, `AFTER_DROP_TABLE`, `AFTER_RENAME_TABLE`, `AFTER_REGISTER_TABLE` | +| **View Operations** | `AFTER_CREATE_VIEW`, `AFTER_DROP_VIEW`, `AFTER_REPLACE_VIEW`, `AFTER_RENAME_VIEW` | +| **Namespace Operations** | `AFTER_CREATE_NAMESPACE`, `AFTER_UPDATE_NAMESPACE_PROPERTIES`, `AFTER_DROP_NAMESPACE` | + +**Note:** Realm-level administrative events (for example, principal lifecycle and credential rotation) are intentionally out of scope for `/api/events/v1/{prefix}`. + +**Note:** Read-only operations (`AFTER_LOAD_*`, `AFTER_LIST_*`, `AFTER_GET_*`, `AFTER_CHECK_EXISTS_*`) should **not** be persisted for the Events API as they do not represent catalog mutations. + +#### Implementation Approach + +1. **Phase 1**: Extend `PolarisPersistenceEventListener` to handle all Iceberg-standard operations (tables, views, namespaces) +2. **Phase 2**: Optionally add support for **catalog-scoped** Polaris extensions (for example, catalog-role and grant changes) using `polaris-*` custom operation types +3. **Phase 3**: Implement the Events REST API on top of the persisted data + +### 7.2 Database Queries + +The endpoints will query existing tables with appropriate filtering and pagination: + +```sql +-- List events with cursor-based pagination +SELECT * FROM events +WHERE realm_id = ? + AND catalog_id = ? + AND (timestamp_ms, event_id) < (?, ?) -- cursor + AND event_type = ? -- optional filter + AND timestamp_ms >= ? -- optional filter + AND timestamp_ms < ? -- optional filter +ORDER BY timestamp_ms DESC, event_id DESC +LIMIT ?; +``` + +### 7.3 Recommended Indexes + +```sql +-- Events indexes +CREATE INDEX IF NOT EXISTS idx_events_catalog_ts + ON events(realm_id, catalog_id, timestamp_ms DESC, event_id DESC); +CREATE INDEX IF NOT EXISTS idx_events_type + ON events(realm_id, catalog_id, event_type, timestamp_ms DESC); +``` + +### 7.4 Files to Modify + +| File | Changes | +|------|---------| +| `spec/events-service.yml` | **New file** - Events API paths and schemas | +| `api/events-service/` | **New** - Generated Events API interfaces | +| `runtime/service/.../events/` | **New** - Events service implementation | +| `polaris-core/.../persistence/BasePersistence.java` | Add read methods | +| `persistence/relational-jdbc/.../JdbcBasePersistenceImpl.java` | Query implementations | + +### 7.5 Pagination Token Format + +Internal format (base64-encoded JSON, opaque to clients): + +```json +{ + "ts": 1709337612345, + "id": "550e8400-e29b-41d4-a716-446655440000" +} +``` + +### 7.6 Mapping PolarisEventType to Iceberg Events API + +Polaris internally uses a `PolarisEventType` enum that distinguishes between `BEFORE_*` and `AFTER_*` events for each operation. This section explains how these internal events map to the Iceberg Events API. + +#### 7.6.1 Design Decision: Only AFTER Events are Exposed + +The Iceberg Events API represents **completed operations** that have been committed to the catalog. Therefore: + +| Internal Event Pattern | Exposed via API? | Rationale | +|------------------------|------------------|-----------| +| `AFTER_*` events | **Yes** | Represent successful, committed operations | +| `BEFORE_*` events | **No** | Represent intent, not outcome; may fail after firing | + +**Why not expose BEFORE events?** + +1. **Semantic mismatch**: The Iceberg Events API is designed for change data capture (CDC) and audit logs of *completed* changes. +2. **Consistency**: Exposing `BEFORE_*` events could lead to consumers seeing "phantom" operations that never actually occurred. +3. **Use case alignment**: The primary use cases (audit, federation, workflow triggers) all require knowing what *actually happened*. +4. **Internal vs external**: `BEFORE_*` events serve internal purposes (request filtering, rate limiting, pre-validation hooks). + +#### 7.6.2 Mapping AFTER Events to Operation Types + +The mapping follows a straightforward pattern - the `AFTER_` prefix is stripped and the remaining name maps to the Iceberg operation type: + +| Polaris `PolarisEventType` | Iceberg `operation-type` | Notes | +|----------------------------|--------------------------|-------| +| **Standard Iceberg Operations** | | | +| `AFTER_CREATE_TABLE` | `create-table` | Direct mapping | +| `AFTER_REGISTER_TABLE` | `register-table` | Direct mapping | +| `AFTER_DROP_TABLE` | `drop-table` | Direct mapping | +| `AFTER_UPDATE_TABLE` | `update-table` | Includes schema evolution, property changes | +| `AFTER_RENAME_TABLE` | `rename-table` | Direct mapping | +| `AFTER_CREATE_VIEW` | `create-view` | Direct mapping | +| `AFTER_DROP_VIEW` | `drop-view` | Direct mapping | +| `AFTER_REPLACE_VIEW` | `update-view` | View replacement maps to update | +| `AFTER_RENAME_VIEW` | `rename-view` | Direct mapping | +| `AFTER_CREATE_NAMESPACE` | `create-namespace` | Direct mapping | +| `AFTER_UPDATE_NAMESPACE_PROPERTIES` | `update-namespace-properties` | Direct mapping | +| `AFTER_DROP_NAMESPACE` | `drop-namespace` | Direct mapping | +| **Polaris Custom Operations (catalog-scoped)** | | Use `custom` type with `polaris-*` | +| `AFTER_CREATE_CATALOG_ROLE` | `custom` (`polaris-create-catalog-role`) | RBAC | +| `AFTER_ADD_GRANT_TO_CATALOG_ROLE` | `custom` (`polaris-grant-privilege`) | RBAC | +| `AFTER_REVOKE_GRANT_FROM_CATALOG_ROLE` | `custom` (`polaris-revoke-privilege`) | RBAC | + +--- + +## 8. Iceberg Events API Alignment + +This section documents the alignment with the [Iceberg Events API specification](https://github.com/apache/iceberg/pull/12584) and explains the rationale for design decisions. + +### 8.1 Why Align with Iceberg Events API? + +The Iceberg Events API is an emerging specification that is nearing consensus in the Apache Iceberg community. Aligning Polaris with this specification provides: + +1. **Ecosystem Compatibility**: Clients built for the Iceberg Events API will work with Polaris without modification +2. **Future-Proofing**: Avoids breaking changes when the Iceberg spec is finalized +3. **Tooling Interoperability**: Monitoring tools, federation services, and workflow triggers can work across Iceberg-compatible catalogs +4. **Reduced Cognitive Load**: Developers familiar with Iceberg don't need to learn a new API + +### 8.2 Key Design Decisions from Iceberg Spec + +| Decision | Iceberg Spec Approach | Rationale | +|----------|----------------------|-----------| +| **HTTP Method** | `POST` (not `GET`) | Allows complex filtering with arrays and nested objects in request body | +| **API Path** | `/v1/{prefix}/events` | Part of Iceberg REST Catalog, not a separate management API | +| **Pagination** | `continuation-token` | Opaque cursor that encodes server state; resumable after downtime | +| **Event Structure** | Operation-centric with discriminator | Each event contains a typed `operation` with operation-specific fields | +| **Operation Types** | Standardized enum + custom prefix extensions | Standard types for Iceberg operations; custom prefix for catalog-specific extensions | +| **Actor Field** | Generic object (implementation-specific) | Flexibility for different auth models (users, service accounts, etc.) | +| **Error Handling** | `410 Gone` for expired offsets | Explicit signal when continuation token is outside retention window | + +### 8.3 Polaris-Specific Extensions + +Polaris extends the Iceberg Events API using the `custom` operation type with `polaris-*` prefixed custom types: + +| Custom Type | Polaris Event | Description | +|-------------|---------------|-------------| +| `polaris-create-catalog-role` | `AFTER_CREATE_CATALOG_ROLE` | Catalog role created | +| `polaris-grant-privilege` | `AFTER_ADD_GRANT_TO_CATALOG_ROLE` | Privilege granted to role | +| `polaris-revoke-privilege` | `AFTER_REVOKE_GRANT_FROM_CATALOG_ROLE` | Privilege revoked | + +### 8.4 References + +- **Iceberg Events API Proposal**: [Google Doc](https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit) +- **Iceberg Events API PR**: [apache/iceberg#12584](https://github.com/apache/iceberg/pull/12584) +- **Iceberg REST Catalog Spec**: [rest-catalog-open-api.yaml](https://github.com/apache/iceberg/blob/main/open-api/rest-catalog-open-api.yaml) + +--- + +## Open Questions + +1. **Event Retention**: What is the default retention period for events? Should it be configurable? +2. **Consistency Guarantees**: What ordering and delivery guarantees should Polaris provide for the Events API? + +--- + +## References + +- Database schema: `persistence/relational-jdbc/src/main/resources/postgres/schema-v4.sql` +- Event types: `runtime/service/src/main/java/org/apache/polaris/service/events/PolarisEventType.java` +- Iceberg Events API PR: https://github.com/apache/iceberg/pull/12584 +- Iceberg Events API Design Doc: https://docs.google.com/document/d/1WtIsNGVX75-_MsQIOJhXLAWg6IbplV4-DkLllQEiFT8/edit + diff --git a/site/content/in-dev/unreleased/proposals/table-metrics-rest-api.md b/site/content/in-dev/unreleased/proposals/table-metrics-rest-api.md new file mode 100644 index 0000000000..dacf44121a --- /dev/null +++ b/site/content/in-dev/unreleased/proposals/table-metrics-rest-api.md @@ -0,0 +1,720 @@ +--- +title: Table Metrics REST API +linkTitle: Table Metrics REST API +weight: 110 +--- + + +# Proposal: REST API for Querying Table Metrics + +**Author:** Anand Sankaran +**Date:** 2026-03-02 +**Status:** Draft Proposal +**Target:** Apache Polaris + +--- + +## Abstract + +This proposal defines REST API endpoints for querying table metrics (scan reports, commit reports) from Apache Polaris. The endpoints expose data already being persisted via the existing JDBC persistence model (`scan_metrics_report`, `commit_metrics_report` tables) and follow established Polaris API patterns. + +--- + +## Table of Contents + +1. [Motivation](#1-motivation) +2. [Use Cases](#2-use-cases) +3. [Design Principles](#3-design-principles) +4. [API Specification](#4-api-specification) +5. [Authorization](#5-authorization) +6. [OpenAPI Schema](#6-openapi-schema) +7. [Implementation Notes](#7-implementation-notes) + +--- + +## 1. Motivation + +Apache Polaris currently persists table metrics (scan reports, commit reports) to the database, but provides no REST API to query this data. Users must access the database directly to retrieve metrics information. + +Adding read-only REST endpoints enables: +- Programmatic access to metrics without database credentials +- Integration with monitoring dashboards and alerting systems +- Consistent authorization via Polaris RBAC +- Pagination and filtering without writing SQL + +--- + +## 2. Use Cases + +### 2.1 Table Health Monitoring +- Track write patterns: files added/removed per commit, record counts, duration trends +- Identify tables with high commit frequency or unusually large commits +- Detect issues indicating need for compaction (many small files) or optimization + +### 2.2 Query Performance Analysis +- Understand read patterns: files scanned vs skipped, planning duration +- Identify inefficient queries with low manifest/file pruning ratios +- Correlate performance with filter expressions and projected columns + +### 2.3 Capacity Planning & Chargeback +- Aggregate metrics by table, namespace, or principal over time +- Track storage growth trends (`total_file_size_bytes`) +- Attribute usage to teams/users via `principal_name` + +### 2.4 Debugging & Troubleshooting +- Correlate metrics with distributed traces (`otel_trace_id`, `otel_span_id`) +- Investigate specific commits by `snapshot_id` +- Trace operations via `request_id` + +--- + +## 3. Design Principles + +| Principle | Rationale | +|-----------|-----------| +| **Dedicated metrics-reports namespace** | Metrics APIs use `/api/metrics-reports/v1/...` to separate from management and catalog APIs | +| **Read-only semantics** | All endpoints are read-only; metrics are written via existing flows | +| **Consistent pagination** | Follow `pageToken` pattern (Polaris APIs) | +| **Flexible filtering** | Time ranges, snapshot IDs, principals - common query patterns | +| **RBAC integration** | Leverage existing Polaris authorization model | +| **Realm handling** | Process Polaris realms consistently with existing APIs; realm context is derived from the authenticated principal | +| **Stable envelope** | Polaris-owned stable envelope fields for resilient client integrations; type-specific payloads are versioned independently | + +### 3.1 Stable Envelope + +To reduce coupling to any single upstream schema and keep client integrations resilient, the Metrics API SHOULD return records using a **stable envelope**: a small, Polaris-owned set of top-level fields that remain consistent across all metric report types, plus a versioned payload for the type-specific body. + +The envelope enables clients to reliably paginate, deduplicate, and correlate records (request IDs / trace IDs) without needing to understand the full payload schema. Type-specific details live under `payload`, identified by `payload.type` and `payload.version`. + +#### Envelope Fields (Conceptual) + +| Field | Description | +|-------|-------------| +| `id` | Unique identifier for the report | +| `timestampMs` | Report timestamp (epoch milliseconds) | +| `catalog` | Catalog identifier | +| `realm` | Realm identifier (if applicable) | +| `actor` | Principal/service + optional client metadata | +| `request` | Request context (requestId, trace/span IDs) | +| `object` | Resource identity: namespace/table identifiers and optional UUID/snapshotId | +| `payload` | `{ type, version, data, extensions? }` | + +#### Compatibility / Evolution Rules + +1. **Envelope is additive-only**: New envelope fields may be added as optional; existing envelope fields MUST NOT change meaning or type, and MUST NOT be removed. + +2. **Breaking changes require versioning at the API boundary**: Any breaking envelope change requires a new major API version (e.g., `/v2/...`) rather than changing `/v1`. + +3. **Payload is independently versioned**: `payload.type` selects the schema family (e.g., `iceberg.metrics.scan`, `iceberg.metrics.commit`), and `payload.version` increments on breaking payload changes. + +4. **Unknown payloads are safe to ignore**: Clients MUST treat unknown `payload.type` or higher `payload.version` as opaque and continue to operate using envelope fields (e.g., still paginate / display metadata). + +5. **Payload changes within a version are additive-only**: Within a given `payload.type` + `payload.version`, new fields may be added as optional; removals/renames/type changes require a new `payload.version`. + +6. **Flattening is a presentation choice**: The default representation SHOULD keep domain-specific structures nested under `payload.data`; alternative "flattened" views (if needed) should be offered as an explicit, separately-versioned representation rather than redefining the canonical schema. + +--- + +## 4. API Specification + +### 4.1 Endpoint Summary + +| Method | Path | Description | +|--------|------|-------------| +| GET | `/api/metrics-reports/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}` | List metrics for a table (type specified via query parameter) | + +> **Note:** The metrics API uses a dedicated `/api/metrics-reports/v1/` namespace since it exposes pre-populated records rather than managing catalog state. + +### 4.2 Path Parameters + +| Parameter | Type | Description | +|-----------|------|-------------| +| `catalogName` | string | Name of the catalog | +| `namespace` | string | Namespace (URL-encoded, multi-level separated by `%1F`) | +| `table` | string | Table name | + +### 4.3 Query Parameters + +#### List Table Metrics (`/.../tables/{table}`) + +| Parameter | Type | Required | Default | Description | +|-----------|------|----------|---------|-------------| +| `metricType` | string | **Yes** | - | Type of metrics to retrieve: `scan` or `commit` | +| `pageToken` | string | No | - | Cursor for pagination | +| `pageSize` | integer | No | 100 | Results per page (max: 1000) | +| `snapshotId` | long | No | - | Filter by snapshot ID | +| `principalName` | string | No | - | Filter by principal | +| `timestampFrom` | long | No | - | Start of time range (epoch ms) | +| `timestampTo` | long | No | - | End of time range (epoch ms) | + +> **Note:** The `metricType` parameter is required. This design allows for future extensibility as new metric types are added (e.g., compaction metrics, maintenance metrics) without requiring new endpoints. + +**Non-goals (v1):** This endpoint is intentionally limited to **paged retrieval** of persisted metrics reports with basic filtering (primarily by time range and identifiers). It does not aim to be a general-purpose metrics query system (no aggregation/group-by, no derived computations, no complex query language); richer analytics are expected to be handled by exporting/sinking these reports to an external observability system. + +### 4.4 Example Requests and Responses + +#### List Metrics (Scan) + +**Request:** +```http +GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views?metricType=scan&pageSize=2×tampFrom=1709251200000 +Authorization: Bearer +``` + +**Response:** +```json +{ + "nextPageToken": null, + "metricType": "scan", + "reports": [ + { + "reportId": "scan-001-abc123", + "catalogId": 12345, + "tableId": 67890, + "timestampMs": 1709337612345, + "principalName": "analyst@example.com", + "requestId": "req-scan-001", + "otelTraceId": "abc123def456789012345678901234", + "otelSpanId": "def456789012", + "snapshotId": 1234567890123, + "schemaId": 0, + "filterExpression": "event_date >= '2024-03-01'", + "projectedFieldIds": "1,2,3,5,8", + "projectedFieldNames": "event_id,user_id,event_type,timestamp,page_url", + "resultDataFiles": 150, + "resultDeleteFiles": 5, + "totalFileSizeBytes": 1073741824, + "totalDataManifests": 12, + "totalDeleteManifests": 2, + "scannedDataManifests": 8, + "scannedDeleteManifests": 2, + "skippedDataManifests": 4, + "skippedDeleteManifests": 0, + "skippedDataFiles": 45, + "skippedDeleteFiles": 0, + "totalPlanningDurationMs": 250, + "equalityDeleteFiles": 3, + "positionalDeleteFiles": 2, + "indexedDeleteFiles": 0, + "totalDeleteFileSizeBytes": 52428800 + } + ] +} +``` + +#### List Metrics (Commit) + +**Request:** +```http +GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views?metricType=commit&pageSize=2 +Authorization: Bearer +``` + +**Response:** +```json +{ + "nextPageToken": "eyJ0cyI6MTcwOTMzNzcwMDAwMCwiaWQiOiJjb21taXQtMDAyIn0=", + "metricType": "commit", + "reports": [ + { + "reportId": "commit-001-xyz789", + "catalogId": 12345, + "tableId": 67890, + "timestampMs": 1709337800000, + "principalName": "etl-service@example.com", + "requestId": "req-commit-001", + "otelTraceId": "xyz789abc123456789012345678901", + "otelSpanId": "abc123456789", + "snapshotId": 1234567890124, + "sequenceNumber": 42, + "operation": "append", + "addedDataFiles": 10, + "removedDataFiles": 0, + "totalDataFiles": 160, + "addedDeleteFiles": 0, + "removedDeleteFiles": 0, + "totalDeleteFiles": 5, + "addedEqualityDeleteFiles": 0, + "removedEqualityDeleteFiles": 0, + "addedPositionalDeleteFiles": 0, + "removedPositionalDeleteFiles": 0, + "addedRecords": 100000, + "removedRecords": 0, + "totalRecords": 15000000, + "addedFileSizeBytes": 104857600, + "removedFileSizeBytes": 0, + "totalFileSizeBytes": 1178599424, + "totalDurationMs": 5000, + "attempts": 1 + } + ] +} +``` + +--- + +## 5. Authorization + +### 5.1 Required Privileges + +This proposal introduces a **new dedicated privilege** for reading table metrics, following the principle of **separation of duties**. This ensures that: + +- Monitoring tools can access metrics without requiring data read access +- Fine-grained access control is possible for different operational roles + +| Endpoint | Required Privilege | Scope | New Privilege? | +|----------|-------------------|-------|----------------| +| List Scan Metrics | `TABLE_READ_METRICS` | Table | **Yes** | +| List Commit Metrics | `TABLE_READ_METRICS` | Table | **Yes** | + +### 5.2 New Privilege Definition + +| Privilege | Scope | Description | +|-----------|-------|-------------| +| `TABLE_READ_METRICS` | Table | Read-only access to table scan and commit metrics. Does not grant access to table data. | + +### 5.3 Rationale: Separation of Duties + +Introducing a dedicated read-only privilege enables proper **separation of duties**: + +| Use Case | Required Privilege | Why Not Reuse Existing? | +|----------|-------------------|------------------------| +| Monitoring tool collecting table metrics | `TABLE_READ_METRICS` | Should not require `TABLE_READ_DATA` (data access) | +| Data analyst with table access | `TABLE_READ_DATA` implies `TABLE_READ_METRICS` | Users who can read data can also see metrics about their queries | + +### 5.4 Privilege Hierarchy + +The new privilege fits into the existing hierarchy as follows: + +``` +TABLE_FULL_METADATA / TABLE_READ_DATA + └── TABLE_READ_METRICS (implied) +``` + +This means: +- Users with `TABLE_READ_DATA` automatically have `TABLE_READ_METRICS` +- But the reverse is **not** true: `TABLE_READ_METRICS` does not grant data access + +### 5.5 Implementation Notes + +New privileges require: +1. Adding entries to `PolarisPrivilege` enum +2. Updating the privilege hierarchy in the authorizer +3. Adding privilege checks in the new API endpoint + +--- + +## 6. OpenAPI Schema + +> **Note:** The OpenAPI specifications below are embedded in this proposal for review context. Upon approval, these should be extracted into separate files for ease of processing and proper integration: +> - **Metrics Reports API** → `spec/metrics-reports-service.yml` (new dedicated service spec with base path `/api/metrics-reports/v1/`) + +### 6.1 Metrics API (New Metrics Reports Service) + +Add the following to a new `spec/metrics-reports-service.yml`: + +> **Note:** The metrics API uses `/api/metrics-reports/v1/` as the base path, separate from the management API. This reflects that metrics reports are read-only access to pre-populated data, not catalog management operations. + +```yaml +paths: + /catalogs/{catalogName}/namespaces/{namespace}/tables/{table}: + parameters: + - $ref: '#/components/parameters/catalogName' + - $ref: '#/components/parameters/namespace' + - $ref: '#/components/parameters/table' + get: + operationId: listTableMetrics + summary: List metrics for a table + description: > + Returns metrics reports for the specified table. The type of metrics + (scan or commit) must be specified via the required metricType parameter. + This unified endpoint supports future extensibility as new metric types + are added. + tags: + - Observability + parameters: + - name: metricType + in: query + required: true + description: Type of metrics to retrieve + schema: + type: string + enum: [scan, commit] + - name: pageToken + in: query + schema: + type: string + - name: pageSize + in: query + schema: + type: integer + minimum: 1 + maximum: 1000 + default: 100 + - name: snapshotId + in: query + schema: + type: integer + format: int64 + - name: principalName + in: query + schema: + type: string + - name: timestampFrom + in: query + schema: + type: integer + format: int64 + - name: timestampTo + in: query + schema: + type: integer + format: int64 + responses: + '200': + description: Paginated list of metrics reports + content: + application/json: + schema: + $ref: '#/components/schemas/ListMetricsResponse' + '400': + description: Bad request (e.g., missing metricType, invalid parameter combination) + '403': + description: Insufficient privileges + '404': + description: Table not found +``` + +### 6.2 Metrics API Schemas + +Add these schemas to `spec/metrics-reports-service.yml`: + +```yaml +components: + schemas: + ScanMetricsReport: + type: object + required: + - reportId + - catalogId + - tableId + - timestampMs + properties: + reportId: + type: string + catalogId: + type: integer + format: int64 + tableId: + type: integer + format: int64 + timestampMs: + type: integer + format: int64 + principalName: + type: string + requestId: + type: string + otelTraceId: + type: string + description: OpenTelemetry trace ID + otelSpanId: + type: string + description: OpenTelemetry span ID + snapshotId: + type: integer + format: int64 + schemaId: + type: integer + filterExpression: + type: string + projectedFieldIds: + type: string + projectedFieldNames: + type: string + resultDataFiles: + type: integer + format: int64 + resultDeleteFiles: + type: integer + format: int64 + totalFileSizeBytes: + type: integer + format: int64 + totalDataManifests: + type: integer + format: int64 + totalDeleteManifests: + type: integer + format: int64 + scannedDataManifests: + type: integer + format: int64 + scannedDeleteManifests: + type: integer + format: int64 + skippedDataManifests: + type: integer + format: int64 + skippedDeleteManifests: + type: integer + format: int64 + skippedDataFiles: + type: integer + format: int64 + skippedDeleteFiles: + type: integer + format: int64 + totalPlanningDurationMs: + type: integer + format: int64 + equalityDeleteFiles: + type: integer + format: int64 + positionalDeleteFiles: + type: integer + format: int64 + indexedDeleteFiles: + type: integer + format: int64 + totalDeleteFileSizeBytes: + type: integer + format: int64 + + ListMetricsResponse: + description: > + Polymorphic response for metrics queries. The concrete type is determined + by the metricType discriminator field. + oneOf: + - $ref: '#/components/schemas/ListScanMetricsResponse' + - $ref: '#/components/schemas/ListCommitMetricsResponse' + discriminator: + propertyName: metricType + mapping: + scan: '#/components/schemas/ListScanMetricsResponse' + commit: '#/components/schemas/ListCommitMetricsResponse' + + ListScanMetricsResponse: + type: object + required: + - metricType + - reports + properties: + nextPageToken: + type: string + description: Cursor for fetching the next page of results + metricType: + type: string + const: scan + description: Discriminator indicating this response contains scan metrics + reports: + type: array + description: Array of scan metrics reports + items: + $ref: '#/components/schemas/ScanMetricsReport' + + ListCommitMetricsResponse: + type: object + required: + - metricType + - reports + properties: + nextPageToken: + type: string + description: Cursor for fetching the next page of results + metricType: + type: string + const: commit + description: Discriminator indicating this response contains commit metrics + reports: + type: array + description: Array of commit metrics reports + items: + $ref: '#/components/schemas/CommitMetricsReport' + + CommitMetricsReport: + type: object + required: + - reportId + - catalogId + - tableId + - timestampMs + - snapshotId + - operation + properties: + reportId: + type: string + catalogId: + type: integer + format: int64 + tableId: + type: integer + format: int64 + timestampMs: + type: integer + format: int64 + principalName: + type: string + requestId: + type: string + otelTraceId: + type: string + otelSpanId: + type: string + snapshotId: + type: integer + format: int64 + sequenceNumber: + type: integer + format: int64 + operation: + type: string + description: Commit operation (append, overwrite, delete, replace) + addedDataFiles: + type: integer + format: int64 + removedDataFiles: + type: integer + format: int64 + totalDataFiles: + type: integer + format: int64 + addedDeleteFiles: + type: integer + format: int64 + removedDeleteFiles: + type: integer + format: int64 + totalDeleteFiles: + type: integer + format: int64 + addedEqualityDeleteFiles: + type: integer + format: int64 + removedEqualityDeleteFiles: + type: integer + format: int64 + addedPositionalDeleteFiles: + type: integer + format: int64 + removedPositionalDeleteFiles: + type: integer + format: int64 + addedRecords: + type: integer + format: int64 + removedRecords: + type: integer + format: int64 + totalRecords: + type: integer + format: int64 + addedFileSizeBytes: + type: integer + format: int64 + removedFileSizeBytes: + type: integer + format: int64 + totalFileSizeBytes: + type: integer + format: int64 + totalDurationMs: + type: integer + format: int64 + attempts: + type: integer +``` + +--- + +## 7. Implementation Notes + +### 7.1 Database Queries + +The endpoints will query existing tables with appropriate filtering and pagination: + +```sql +-- List scan metrics +SELECT * FROM scan_metrics_report +WHERE realm_id = ? + AND catalog_id = ? + AND table_id = ? + AND timestamp_ms >= ? + AND timestamp_ms < ? +ORDER BY timestamp_ms DESC, report_id DESC +LIMIT ?; + +-- List commit metrics +SELECT * FROM commit_metrics_report +WHERE realm_id = ? + AND catalog_id = ? + AND table_id = ? + AND timestamp_ms >= ? + AND timestamp_ms < ? +ORDER BY timestamp_ms DESC, report_id DESC +LIMIT ?; +``` + +### 7.2 Recommended Indexes + +```sql +-- Metrics indexes (may already exist) +CREATE INDEX IF NOT EXISTS idx_scan_report_lookup + ON scan_metrics_report(realm_id, catalog_id, table_id, timestamp_ms DESC); +CREATE INDEX IF NOT EXISTS idx_commit_report_lookup + ON commit_metrics_report(realm_id, catalog_id, table_id, timestamp_ms DESC); +``` + +### 7.3 Files to Modify + +| File | Changes | +|------|---------| +| `spec/metrics-reports-service.yml` | **New file** - Metrics Reports API paths and schemas | +| `api/metrics-reports-service/` | **New** - Generated Metrics Reports API interfaces | +| `runtime/service/.../metrics/` | **New** - Metrics reports service implementation | +| `polaris-core/.../persistence/BasePersistence.java` | Add read methods | +| `persistence/relational-jdbc/.../JdbcBasePersistenceImpl.java` | Query implementations | + +### 7.4 Pagination Token Format + +Internal format (base64-encoded JSON, opaque to clients): + +```json +{ + "ts": 1709337612345, + "id": "report-abc123" +} +``` + +--- + +## Open Questions + +1. **Aggregations**: Are aggregated metrics views needed (e.g., daily summaries)? +2. **Metric Retention**: What is the default retention period for metrics? Should it be configurable? + +--- + +## References + +- Database schema: `persistence/relational-jdbc/src/main/resources/postgres/schema-v4.sql` +- Metrics persistence: `runtime/service/src/main/java/org/apache/polaris/service/reporting/PersistingMetricsReporter.java` +- Metrics record converter: `polaris-core/src/main/java/org/apache/polaris/core/metrics/iceberg/MetricsRecordConverter.java` + From 1dab56a74c2b7eee36b9686982e9b76bdf065610 Mon Sep 17 00:00:00 2001 From: Anand Kumar Sankaran Date: Tue, 17 Mar 2026 08:40:30 -0700 Subject: [PATCH 15/15] refactor: Split proposal - keep Events API only in this PR The Table Metrics REST API proposal has been moved to a separate PR #4010 to allow independent progress on both proposals. This PR now contains only the Events REST API proposal. Requested by: @dimas-b in review comment --- .../in-dev/unreleased/proposals/_index.md | 1 - .../proposals/table-metrics-rest-api.md | 720 ------------------ 2 files changed, 721 deletions(-) delete mode 100644 site/content/in-dev/unreleased/proposals/table-metrics-rest-api.md diff --git a/site/content/in-dev/unreleased/proposals/_index.md b/site/content/in-dev/unreleased/proposals/_index.md index 3344e2360b..1cc57db5a4 100644 --- a/site/content/in-dev/unreleased/proposals/_index.md +++ b/site/content/in-dev/unreleased/proposals/_index.md @@ -31,5 +31,4 @@ This section contains design proposals for new features and enhancements to Apac | Proposal | Status | Description | |----------|--------|-------------| | [Events REST API](events-rest-api.md) | Draft | REST API endpoint for querying catalog events (Iceberg Events API compatible) | -| [Table Metrics REST API](table-metrics-rest-api.md) | Draft | REST API endpoints for querying table scan and commit metrics | diff --git a/site/content/in-dev/unreleased/proposals/table-metrics-rest-api.md b/site/content/in-dev/unreleased/proposals/table-metrics-rest-api.md deleted file mode 100644 index dacf44121a..0000000000 --- a/site/content/in-dev/unreleased/proposals/table-metrics-rest-api.md +++ /dev/null @@ -1,720 +0,0 @@ ---- -title: Table Metrics REST API -linkTitle: Table Metrics REST API -weight: 110 ---- - - -# Proposal: REST API for Querying Table Metrics - -**Author:** Anand Sankaran -**Date:** 2026-03-02 -**Status:** Draft Proposal -**Target:** Apache Polaris - ---- - -## Abstract - -This proposal defines REST API endpoints for querying table metrics (scan reports, commit reports) from Apache Polaris. The endpoints expose data already being persisted via the existing JDBC persistence model (`scan_metrics_report`, `commit_metrics_report` tables) and follow established Polaris API patterns. - ---- - -## Table of Contents - -1. [Motivation](#1-motivation) -2. [Use Cases](#2-use-cases) -3. [Design Principles](#3-design-principles) -4. [API Specification](#4-api-specification) -5. [Authorization](#5-authorization) -6. [OpenAPI Schema](#6-openapi-schema) -7. [Implementation Notes](#7-implementation-notes) - ---- - -## 1. Motivation - -Apache Polaris currently persists table metrics (scan reports, commit reports) to the database, but provides no REST API to query this data. Users must access the database directly to retrieve metrics information. - -Adding read-only REST endpoints enables: -- Programmatic access to metrics without database credentials -- Integration with monitoring dashboards and alerting systems -- Consistent authorization via Polaris RBAC -- Pagination and filtering without writing SQL - ---- - -## 2. Use Cases - -### 2.1 Table Health Monitoring -- Track write patterns: files added/removed per commit, record counts, duration trends -- Identify tables with high commit frequency or unusually large commits -- Detect issues indicating need for compaction (many small files) or optimization - -### 2.2 Query Performance Analysis -- Understand read patterns: files scanned vs skipped, planning duration -- Identify inefficient queries with low manifest/file pruning ratios -- Correlate performance with filter expressions and projected columns - -### 2.3 Capacity Planning & Chargeback -- Aggregate metrics by table, namespace, or principal over time -- Track storage growth trends (`total_file_size_bytes`) -- Attribute usage to teams/users via `principal_name` - -### 2.4 Debugging & Troubleshooting -- Correlate metrics with distributed traces (`otel_trace_id`, `otel_span_id`) -- Investigate specific commits by `snapshot_id` -- Trace operations via `request_id` - ---- - -## 3. Design Principles - -| Principle | Rationale | -|-----------|-----------| -| **Dedicated metrics-reports namespace** | Metrics APIs use `/api/metrics-reports/v1/...` to separate from management and catalog APIs | -| **Read-only semantics** | All endpoints are read-only; metrics are written via existing flows | -| **Consistent pagination** | Follow `pageToken` pattern (Polaris APIs) | -| **Flexible filtering** | Time ranges, snapshot IDs, principals - common query patterns | -| **RBAC integration** | Leverage existing Polaris authorization model | -| **Realm handling** | Process Polaris realms consistently with existing APIs; realm context is derived from the authenticated principal | -| **Stable envelope** | Polaris-owned stable envelope fields for resilient client integrations; type-specific payloads are versioned independently | - -### 3.1 Stable Envelope - -To reduce coupling to any single upstream schema and keep client integrations resilient, the Metrics API SHOULD return records using a **stable envelope**: a small, Polaris-owned set of top-level fields that remain consistent across all metric report types, plus a versioned payload for the type-specific body. - -The envelope enables clients to reliably paginate, deduplicate, and correlate records (request IDs / trace IDs) without needing to understand the full payload schema. Type-specific details live under `payload`, identified by `payload.type` and `payload.version`. - -#### Envelope Fields (Conceptual) - -| Field | Description | -|-------|-------------| -| `id` | Unique identifier for the report | -| `timestampMs` | Report timestamp (epoch milliseconds) | -| `catalog` | Catalog identifier | -| `realm` | Realm identifier (if applicable) | -| `actor` | Principal/service + optional client metadata | -| `request` | Request context (requestId, trace/span IDs) | -| `object` | Resource identity: namespace/table identifiers and optional UUID/snapshotId | -| `payload` | `{ type, version, data, extensions? }` | - -#### Compatibility / Evolution Rules - -1. **Envelope is additive-only**: New envelope fields may be added as optional; existing envelope fields MUST NOT change meaning or type, and MUST NOT be removed. - -2. **Breaking changes require versioning at the API boundary**: Any breaking envelope change requires a new major API version (e.g., `/v2/...`) rather than changing `/v1`. - -3. **Payload is independently versioned**: `payload.type` selects the schema family (e.g., `iceberg.metrics.scan`, `iceberg.metrics.commit`), and `payload.version` increments on breaking payload changes. - -4. **Unknown payloads are safe to ignore**: Clients MUST treat unknown `payload.type` or higher `payload.version` as opaque and continue to operate using envelope fields (e.g., still paginate / display metadata). - -5. **Payload changes within a version are additive-only**: Within a given `payload.type` + `payload.version`, new fields may be added as optional; removals/renames/type changes require a new `payload.version`. - -6. **Flattening is a presentation choice**: The default representation SHOULD keep domain-specific structures nested under `payload.data`; alternative "flattened" views (if needed) should be offered as an explicit, separately-versioned representation rather than redefining the canonical schema. - ---- - -## 4. API Specification - -### 4.1 Endpoint Summary - -| Method | Path | Description | -|--------|------|-------------| -| GET | `/api/metrics-reports/v1/catalogs/{catalogName}/namespaces/{namespace}/tables/{table}` | List metrics for a table (type specified via query parameter) | - -> **Note:** The metrics API uses a dedicated `/api/metrics-reports/v1/` namespace since it exposes pre-populated records rather than managing catalog state. - -### 4.2 Path Parameters - -| Parameter | Type | Description | -|-----------|------|-------------| -| `catalogName` | string | Name of the catalog | -| `namespace` | string | Namespace (URL-encoded, multi-level separated by `%1F`) | -| `table` | string | Table name | - -### 4.3 Query Parameters - -#### List Table Metrics (`/.../tables/{table}`) - -| Parameter | Type | Required | Default | Description | -|-----------|------|----------|---------|-------------| -| `metricType` | string | **Yes** | - | Type of metrics to retrieve: `scan` or `commit` | -| `pageToken` | string | No | - | Cursor for pagination | -| `pageSize` | integer | No | 100 | Results per page (max: 1000) | -| `snapshotId` | long | No | - | Filter by snapshot ID | -| `principalName` | string | No | - | Filter by principal | -| `timestampFrom` | long | No | - | Start of time range (epoch ms) | -| `timestampTo` | long | No | - | End of time range (epoch ms) | - -> **Note:** The `metricType` parameter is required. This design allows for future extensibility as new metric types are added (e.g., compaction metrics, maintenance metrics) without requiring new endpoints. - -**Non-goals (v1):** This endpoint is intentionally limited to **paged retrieval** of persisted metrics reports with basic filtering (primarily by time range and identifiers). It does not aim to be a general-purpose metrics query system (no aggregation/group-by, no derived computations, no complex query language); richer analytics are expected to be handled by exporting/sinking these reports to an external observability system. - -### 4.4 Example Requests and Responses - -#### List Metrics (Scan) - -**Request:** -```http -GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views?metricType=scan&pageSize=2×tampFrom=1709251200000 -Authorization: Bearer -``` - -**Response:** -```json -{ - "nextPageToken": null, - "metricType": "scan", - "reports": [ - { - "reportId": "scan-001-abc123", - "catalogId": 12345, - "tableId": 67890, - "timestampMs": 1709337612345, - "principalName": "analyst@example.com", - "requestId": "req-scan-001", - "otelTraceId": "abc123def456789012345678901234", - "otelSpanId": "def456789012", - "snapshotId": 1234567890123, - "schemaId": 0, - "filterExpression": "event_date >= '2024-03-01'", - "projectedFieldIds": "1,2,3,5,8", - "projectedFieldNames": "event_id,user_id,event_type,timestamp,page_url", - "resultDataFiles": 150, - "resultDeleteFiles": 5, - "totalFileSizeBytes": 1073741824, - "totalDataManifests": 12, - "totalDeleteManifests": 2, - "scannedDataManifests": 8, - "scannedDeleteManifests": 2, - "skippedDataManifests": 4, - "skippedDeleteManifests": 0, - "skippedDataFiles": 45, - "skippedDeleteFiles": 0, - "totalPlanningDurationMs": 250, - "equalityDeleteFiles": 3, - "positionalDeleteFiles": 2, - "indexedDeleteFiles": 0, - "totalDeleteFileSizeBytes": 52428800 - } - ] -} -``` - -#### List Metrics (Commit) - -**Request:** -```http -GET /api/metrics-reports/v1/catalogs/my-catalog/namespaces/analytics%1Fevents/tables/page_views?metricType=commit&pageSize=2 -Authorization: Bearer -``` - -**Response:** -```json -{ - "nextPageToken": "eyJ0cyI6MTcwOTMzNzcwMDAwMCwiaWQiOiJjb21taXQtMDAyIn0=", - "metricType": "commit", - "reports": [ - { - "reportId": "commit-001-xyz789", - "catalogId": 12345, - "tableId": 67890, - "timestampMs": 1709337800000, - "principalName": "etl-service@example.com", - "requestId": "req-commit-001", - "otelTraceId": "xyz789abc123456789012345678901", - "otelSpanId": "abc123456789", - "snapshotId": 1234567890124, - "sequenceNumber": 42, - "operation": "append", - "addedDataFiles": 10, - "removedDataFiles": 0, - "totalDataFiles": 160, - "addedDeleteFiles": 0, - "removedDeleteFiles": 0, - "totalDeleteFiles": 5, - "addedEqualityDeleteFiles": 0, - "removedEqualityDeleteFiles": 0, - "addedPositionalDeleteFiles": 0, - "removedPositionalDeleteFiles": 0, - "addedRecords": 100000, - "removedRecords": 0, - "totalRecords": 15000000, - "addedFileSizeBytes": 104857600, - "removedFileSizeBytes": 0, - "totalFileSizeBytes": 1178599424, - "totalDurationMs": 5000, - "attempts": 1 - } - ] -} -``` - ---- - -## 5. Authorization - -### 5.1 Required Privileges - -This proposal introduces a **new dedicated privilege** for reading table metrics, following the principle of **separation of duties**. This ensures that: - -- Monitoring tools can access metrics without requiring data read access -- Fine-grained access control is possible for different operational roles - -| Endpoint | Required Privilege | Scope | New Privilege? | -|----------|-------------------|-------|----------------| -| List Scan Metrics | `TABLE_READ_METRICS` | Table | **Yes** | -| List Commit Metrics | `TABLE_READ_METRICS` | Table | **Yes** | - -### 5.2 New Privilege Definition - -| Privilege | Scope | Description | -|-----------|-------|-------------| -| `TABLE_READ_METRICS` | Table | Read-only access to table scan and commit metrics. Does not grant access to table data. | - -### 5.3 Rationale: Separation of Duties - -Introducing a dedicated read-only privilege enables proper **separation of duties**: - -| Use Case | Required Privilege | Why Not Reuse Existing? | -|----------|-------------------|------------------------| -| Monitoring tool collecting table metrics | `TABLE_READ_METRICS` | Should not require `TABLE_READ_DATA` (data access) | -| Data analyst with table access | `TABLE_READ_DATA` implies `TABLE_READ_METRICS` | Users who can read data can also see metrics about their queries | - -### 5.4 Privilege Hierarchy - -The new privilege fits into the existing hierarchy as follows: - -``` -TABLE_FULL_METADATA / TABLE_READ_DATA - └── TABLE_READ_METRICS (implied) -``` - -This means: -- Users with `TABLE_READ_DATA` automatically have `TABLE_READ_METRICS` -- But the reverse is **not** true: `TABLE_READ_METRICS` does not grant data access - -### 5.5 Implementation Notes - -New privileges require: -1. Adding entries to `PolarisPrivilege` enum -2. Updating the privilege hierarchy in the authorizer -3. Adding privilege checks in the new API endpoint - ---- - -## 6. OpenAPI Schema - -> **Note:** The OpenAPI specifications below are embedded in this proposal for review context. Upon approval, these should be extracted into separate files for ease of processing and proper integration: -> - **Metrics Reports API** → `spec/metrics-reports-service.yml` (new dedicated service spec with base path `/api/metrics-reports/v1/`) - -### 6.1 Metrics API (New Metrics Reports Service) - -Add the following to a new `spec/metrics-reports-service.yml`: - -> **Note:** The metrics API uses `/api/metrics-reports/v1/` as the base path, separate from the management API. This reflects that metrics reports are read-only access to pre-populated data, not catalog management operations. - -```yaml -paths: - /catalogs/{catalogName}/namespaces/{namespace}/tables/{table}: - parameters: - - $ref: '#/components/parameters/catalogName' - - $ref: '#/components/parameters/namespace' - - $ref: '#/components/parameters/table' - get: - operationId: listTableMetrics - summary: List metrics for a table - description: > - Returns metrics reports for the specified table. The type of metrics - (scan or commit) must be specified via the required metricType parameter. - This unified endpoint supports future extensibility as new metric types - are added. - tags: - - Observability - parameters: - - name: metricType - in: query - required: true - description: Type of metrics to retrieve - schema: - type: string - enum: [scan, commit] - - name: pageToken - in: query - schema: - type: string - - name: pageSize - in: query - schema: - type: integer - minimum: 1 - maximum: 1000 - default: 100 - - name: snapshotId - in: query - schema: - type: integer - format: int64 - - name: principalName - in: query - schema: - type: string - - name: timestampFrom - in: query - schema: - type: integer - format: int64 - - name: timestampTo - in: query - schema: - type: integer - format: int64 - responses: - '200': - description: Paginated list of metrics reports - content: - application/json: - schema: - $ref: '#/components/schemas/ListMetricsResponse' - '400': - description: Bad request (e.g., missing metricType, invalid parameter combination) - '403': - description: Insufficient privileges - '404': - description: Table not found -``` - -### 6.2 Metrics API Schemas - -Add these schemas to `spec/metrics-reports-service.yml`: - -```yaml -components: - schemas: - ScanMetricsReport: - type: object - required: - - reportId - - catalogId - - tableId - - timestampMs - properties: - reportId: - type: string - catalogId: - type: integer - format: int64 - tableId: - type: integer - format: int64 - timestampMs: - type: integer - format: int64 - principalName: - type: string - requestId: - type: string - otelTraceId: - type: string - description: OpenTelemetry trace ID - otelSpanId: - type: string - description: OpenTelemetry span ID - snapshotId: - type: integer - format: int64 - schemaId: - type: integer - filterExpression: - type: string - projectedFieldIds: - type: string - projectedFieldNames: - type: string - resultDataFiles: - type: integer - format: int64 - resultDeleteFiles: - type: integer - format: int64 - totalFileSizeBytes: - type: integer - format: int64 - totalDataManifests: - type: integer - format: int64 - totalDeleteManifests: - type: integer - format: int64 - scannedDataManifests: - type: integer - format: int64 - scannedDeleteManifests: - type: integer - format: int64 - skippedDataManifests: - type: integer - format: int64 - skippedDeleteManifests: - type: integer - format: int64 - skippedDataFiles: - type: integer - format: int64 - skippedDeleteFiles: - type: integer - format: int64 - totalPlanningDurationMs: - type: integer - format: int64 - equalityDeleteFiles: - type: integer - format: int64 - positionalDeleteFiles: - type: integer - format: int64 - indexedDeleteFiles: - type: integer - format: int64 - totalDeleteFileSizeBytes: - type: integer - format: int64 - - ListMetricsResponse: - description: > - Polymorphic response for metrics queries. The concrete type is determined - by the metricType discriminator field. - oneOf: - - $ref: '#/components/schemas/ListScanMetricsResponse' - - $ref: '#/components/schemas/ListCommitMetricsResponse' - discriminator: - propertyName: metricType - mapping: - scan: '#/components/schemas/ListScanMetricsResponse' - commit: '#/components/schemas/ListCommitMetricsResponse' - - ListScanMetricsResponse: - type: object - required: - - metricType - - reports - properties: - nextPageToken: - type: string - description: Cursor for fetching the next page of results - metricType: - type: string - const: scan - description: Discriminator indicating this response contains scan metrics - reports: - type: array - description: Array of scan metrics reports - items: - $ref: '#/components/schemas/ScanMetricsReport' - - ListCommitMetricsResponse: - type: object - required: - - metricType - - reports - properties: - nextPageToken: - type: string - description: Cursor for fetching the next page of results - metricType: - type: string - const: commit - description: Discriminator indicating this response contains commit metrics - reports: - type: array - description: Array of commit metrics reports - items: - $ref: '#/components/schemas/CommitMetricsReport' - - CommitMetricsReport: - type: object - required: - - reportId - - catalogId - - tableId - - timestampMs - - snapshotId - - operation - properties: - reportId: - type: string - catalogId: - type: integer - format: int64 - tableId: - type: integer - format: int64 - timestampMs: - type: integer - format: int64 - principalName: - type: string - requestId: - type: string - otelTraceId: - type: string - otelSpanId: - type: string - snapshotId: - type: integer - format: int64 - sequenceNumber: - type: integer - format: int64 - operation: - type: string - description: Commit operation (append, overwrite, delete, replace) - addedDataFiles: - type: integer - format: int64 - removedDataFiles: - type: integer - format: int64 - totalDataFiles: - type: integer - format: int64 - addedDeleteFiles: - type: integer - format: int64 - removedDeleteFiles: - type: integer - format: int64 - totalDeleteFiles: - type: integer - format: int64 - addedEqualityDeleteFiles: - type: integer - format: int64 - removedEqualityDeleteFiles: - type: integer - format: int64 - addedPositionalDeleteFiles: - type: integer - format: int64 - removedPositionalDeleteFiles: - type: integer - format: int64 - addedRecords: - type: integer - format: int64 - removedRecords: - type: integer - format: int64 - totalRecords: - type: integer - format: int64 - addedFileSizeBytes: - type: integer - format: int64 - removedFileSizeBytes: - type: integer - format: int64 - totalFileSizeBytes: - type: integer - format: int64 - totalDurationMs: - type: integer - format: int64 - attempts: - type: integer -``` - ---- - -## 7. Implementation Notes - -### 7.1 Database Queries - -The endpoints will query existing tables with appropriate filtering and pagination: - -```sql --- List scan metrics -SELECT * FROM scan_metrics_report -WHERE realm_id = ? - AND catalog_id = ? - AND table_id = ? - AND timestamp_ms >= ? - AND timestamp_ms < ? -ORDER BY timestamp_ms DESC, report_id DESC -LIMIT ?; - --- List commit metrics -SELECT * FROM commit_metrics_report -WHERE realm_id = ? - AND catalog_id = ? - AND table_id = ? - AND timestamp_ms >= ? - AND timestamp_ms < ? -ORDER BY timestamp_ms DESC, report_id DESC -LIMIT ?; -``` - -### 7.2 Recommended Indexes - -```sql --- Metrics indexes (may already exist) -CREATE INDEX IF NOT EXISTS idx_scan_report_lookup - ON scan_metrics_report(realm_id, catalog_id, table_id, timestamp_ms DESC); -CREATE INDEX IF NOT EXISTS idx_commit_report_lookup - ON commit_metrics_report(realm_id, catalog_id, table_id, timestamp_ms DESC); -``` - -### 7.3 Files to Modify - -| File | Changes | -|------|---------| -| `spec/metrics-reports-service.yml` | **New file** - Metrics Reports API paths and schemas | -| `api/metrics-reports-service/` | **New** - Generated Metrics Reports API interfaces | -| `runtime/service/.../metrics/` | **New** - Metrics reports service implementation | -| `polaris-core/.../persistence/BasePersistence.java` | Add read methods | -| `persistence/relational-jdbc/.../JdbcBasePersistenceImpl.java` | Query implementations | - -### 7.4 Pagination Token Format - -Internal format (base64-encoded JSON, opaque to clients): - -```json -{ - "ts": 1709337612345, - "id": "report-abc123" -} -``` - ---- - -## Open Questions - -1. **Aggregations**: Are aggregated metrics views needed (e.g., daily summaries)? -2. **Metric Retention**: What is the default retention period for metrics? Should it be configurable? - ---- - -## References - -- Database schema: `persistence/relational-jdbc/src/main/resources/postgres/schema-v4.sql` -- Metrics persistence: `runtime/service/src/main/java/org/apache/polaris/service/reporting/PersistingMetricsReporter.java` -- Metrics record converter: `polaris-core/src/main/java/org/apache/polaris/core/metrics/iceberg/MetricsRecordConverter.java` -