datalake_fdw: Apache Iceberg lake tables as an extension (skeleton)

Add contrib/datalake_fdw, a skeleton for Iceberg lake-table support that needs
no kernel changes: a lake table is an ordinary CREATE TABLE ... USING iceberg,
and its binding travels in reloptions naming a catalog server and a volume
server, both created through foreign-data wrappers this extension registers.

Making the binding a pair of foreign servers is what keeps the kernel out of it.
Server options, ownership, privileges and dump/restore already exist for foreign
servers, reloptions already reach every segment with pg_class, and recording the
two servers in pg_depend makes DROP SERVER refuse to strand a table -- none of
which needs new catalogs or grammar.

The DDL path is complete against a stub metadata engine, so CREATE TABLE and
DROP TABLE work end to end with no catalog service, object store, Arrow or JVM
in the picture.  Everything that would touch data reports a clean
"iceberg: <operation> is not supported yet".  The interfaces the later work
plugs into ship whole so they can be reviewed before there is an implementation
behind them: the IcebergMetaEngine vtable with a capability bitmap the registry
validates and dispatches through, the FormatReader/FormatWriter instance
interfaces, and the storage facade over open/read/write/list.

Details worth a reviewer's attention:

* Table metadata always goes through one engine, the Java agent, and nothing
  selects between implementations -- no option, no setting.  The vtable stays
  because the implementation is expected to change; that is a property of the
  build, never of a table or a session, so an existing table can never be
  reinterpreted by a configuration change.

* The table access method fills every callback GetTableAmRoutine() asserts.
  ANALYZE succeeds as a zero-sample no-op through relation_acquire_sample_rows,
  which keeps it off the scan path that reports not-supported, and VACUUM is a
  no-op, so database-wide maintenance never dies on a lake table.

* The object-access hook records the server dependencies on the coordinator and
  on every segment, while only GP_ROLE_DISPATCH calls the metadata engine, so
  each node can protect its own catalog and the remote side sees one call.
  Utility-mode DDL is refused rather than creating local state without dispatch.

* VACUUM FULL is refused in the utility hook, not in the access method: relation
  rewriting creates a transient relation first, which reaches OAT_POST_CREATE
  and has the engine create a table remotely before the rewrite reports its
  error, leaving an orphan behind.

* Credentials are refused in server options and belong in user mappings, which
  stay optional so ambient object-store credentials remain usable.  Binding
  resolution never reads them, so DDL and DROP work with none configured.

* Volume URIs are parsed once, in the options layer, into a versioned
  DatalakeLocation; backends receive only that canonical form.

* C++ translation units reach the server headers through common/dl_pg_api.h,
  which applies extern "C" -- without it the module builds and then fails to
  dlopen on a mangled errmsg.  The C/C++ boundary macros follow the PAX pattern,
  including deferring ereport() until after the catch handler is left, since
  longjmp() out of a handler is undefined.  Exported symbols are limited to the
  PG entry points listed in exports.txt, ELF and Mach-O each getting the right
  linker mechanism, so a future static Arrow cannot leak into other extensions.

The regression suite covers the DDL path including per-segment catalog state,
the rejection matrices, and the privilege model; installcheck is green on a
three-segment cluster.  Testing against real catalogs and object storage needs
a service harness rather than expected-output files, and arrives with the change
that first talks to them -- see test/automation/README.md.

Author: MisterRaindrop

contrib/Makefile

@@ -25,6 +25,7 @@ SUBDIRS = \
 		btree_gin	\
 		btree_gist	\
 		citext		\
+		datalake_fdw	\
 		dblink		\
 		dict_int	\
 		dict_xsyn	\

contrib/datalake_fdw/.gitignore

@@ -0,0 +1,8 @@
+# Generated subdirectories
+/log/
+/results/
+/tmp_check/
+
+# Generated from exports.txt at build time
+/exports.map
+/exports_darwin.list

contrib/datalake_fdw/Makefile

@@ -0,0 +1,95 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# contrib/datalake_fdw/Makefile
+
+MODULE_big = datalake_fdw
+EXTENSION = datalake_fdw
+DATA = datalake_fdw--1.0.sql
+
+OBJS = \
+	src/am_iceberg/pg_iceberg_am_handler.o \
+	src/am_iceberg/pg_iceberg_extensible.o \
+	src/am_iceberg/pg_iceberg_ddl.o \
+	src/am_iceberg/pg_iceberg_options.o \
+	src/am_iceberg/pg_iceberg_guc.o \
+	src/am_iceberg/pg_iceberg_reject.o \
+	src/iceberg_catalog_fdw/iceberg_catalog_fdw.o \
+	src/iceberg_volume_fdw/iceberg_volume_fdw.o \
+	src/meta/meta_engine_registry.o \
+	src/meta/meta_engine_init.o \
+	src/meta/engine_stub/stub_engine.o \
+	src/format/format_registry.o \
+	src/common/file_system_wrapper.o \
+	src/common/s3_file_system.o \
+	src/common/backend_registry.o
+
+# Use the documented PGXS knobs: pgxs.mk appends these AFTER the flags configure
+# chose, so optimization/warning settings survive.  A pre-include
+# "override CFLAGS +=" would give CFLAGS override origin and silently discard
+# Makefile.global's own "CFLAGS = @CFLAGS@" assignment.
+PG_CFLAGS = -fvisibility=hidden
+PG_CXXFLAGS = -fvisibility=hidden -fvisibility-inlines-hidden -std=c++17
+PG_CPPFLAGS = -I$(srcdir)/src
+
+REGRESS = iceberg_am_ddl iceberg_am_reject iceberg_am_acl
+
+EXTRA_CLEAN = exports_darwin.list exports.map
+
+# Keep the aggregate target as make's default goal.
+all:
+
+ifdef USE_PGXS
+PG_CONFIG = pg_config
+PGXS := $(shell $(PG_CONFIG) --pgxs)
+include $(PGXS)
+else
+subdir = contrib/datalake_fdw
+top_builddir = ../..
+include $(top_builddir)/src/Makefile.global
+include $(top_srcdir)/contrib/contrib-global.mk
+endif
+
+# Everything below needs variables that Makefile.global defines (PORTNAME), and
+# SHLIB_LINK additions still apply because the link recipe expands it when it
+# runs.
+
+# Shared libraries are linked with $(CC) (see src/Makefile.shlib COMPILER), so a
+# module containing C++ translation units must pull in the C++ runtime itself.
+SHLIB_LINK += -lstdc++
+
+# Arrow and other C++ dependencies land in this module later; the export list is
+# the single place that decides what stays visible, so the mechanism ships now.
+ifeq ($(PORTNAME), darwin)
+EXPORT_LIST = exports_darwin.list
+SHLIB_LINK += -Wl,-exported_symbols_list,exports_darwin.list
+
+exports_darwin.list: exports.txt
+	sed -e '/^[[:space:]]*#/d' -e '/^[[:space:]]*$$/d' -e 's/^/_/' $< > $@
+else
+EXPORT_LIST = exports.map
+SHLIB_LINK += -Wl,--version-script=exports.map -Wl,--exclude-libs,ALL
+
+exports.map: exports.txt
+	{ echo '{ global:'; sed -e '/^[[:space:]]*#/d' -e '/^[[:space:]]*$$/d' -e 's/$$/;/' $<; echo 'local: *; };'; } > $@
+endif
+
+all: $(EXPORT_LIST)
+$(shlib): $(EXPORT_LIST)
+
+%.o: %.cc
+	$(CXX) $(CPPFLAGS) $(CXXFLAGS) -c $< -o $@

contrib/datalake_fdw/README.md

@@ -0,0 +1,107 @@
+# datalake_fdw
+
+`datalake_fdw` is a skeleton of Apache Iceberg lake-table support for Apache
+Cloudberry, built as an ordinary extension: no kernel changes, no new grammar.
+A lake table is a regular `CREATE TABLE ... USING iceberg`, and its binding to a
+catalog and a storage volume travels in reloptions that point at foreign servers.
+
+This first drop wires up the whole DDL path against a stub metadata engine, so
+`CREATE TABLE` and `DROP TABLE` work end to end with no external dependency --
+no catalog service, no object store, no Arrow, no JVM.  Everything that would
+touch data reports a clean not-supported error.  The interfaces the later work
+plugs into (metadata engine vtable, format reader/writer, storage facade) ship
+here in full so they can be reviewed before there is an implementation behind
+them.
+
+## Build and configuration
+
+```sh
+make USE_PGXS=1 install
+```
+
+The extension installs hooks from `_PG_init`, so it must be preloaded:
+
+```conf
+shared_preload_libraries = 'datalake_fdw'
+```
+
+```sh
+gpconfig -c shared_preload_libraries -v datalake_fdw && gpstop -ar
+```
+
+## Example
+
+```sql
+CREATE EXTENSION datalake_fdw;
+
+-- Where the table metadata lives.
+CREATE SERVER hive_cat FOREIGN DATA WRAPPER iceberg_catalog_fdw
+    OPTIONS (type 'hive', uri 'thrift://metastore:9083');
+
+-- Where the data files live.
+CREATE SERVER s3_vol FOREIGN DATA WRAPPER iceberg_volume_fdw
+    OPTIONS (base_path 's3://bucket/prefix', endpoint 'http://minio:9000',
+             region 'us-east-1');
+
+-- Credentials are optional: without a user mapping the storage backend falls
+-- back to ambient credentials (instance role, environment, and so on).
+CREATE USER MAPPING FOR CURRENT_USER SERVER s3_vol
+    OPTIONS (access_key '...', secret_key '...');
+
+CREATE TABLE t (a int, b text)
+    USING iceberg
+    WITH (catalog = 'hive_cat', volume = 's3_vol');
+
+DROP TABLE t;
+```
+
+Both servers are recorded as dependencies of the table, so `DROP SERVER` is
+refused while lake tables reference it, and `DROP SERVER ... CASCADE` removes
+them together.  Creating a table requires `USAGE` on both servers.
+
+`iceberg.default_catalog` and `iceberg.default_volume` supply the binding when
+`WITH` omits it.
+
+Table metadata always goes through one metadata engine, the Java agent, and
+nothing selects between implementations -- no option, no setting.  The engine
+sits behind a vtable so that the implementation can change (an in-process C++
+one is the expected next step), but that is a property of the build, never of a
+table or a session, so an existing table can never be reinterpreted by a
+configuration change.
+
+## What this skeleton does
+
+| Works | Behaviour |
+| --- | --- |
+| `CREATE TABLE ... USING iceberg` | Binding validated, distribution forced to random, stub engine reports the create |
+| `DROP TABLE` | Stub engine reports the drop; dependencies removed on every segment |
+| `ANALYZE`, `VACUUM` | Succeed as no-ops, so database-wide maintenance is never blocked by a lake table |
+| Catalog and volume validators | Per-context option allowlists; credentials are refused on servers and belong in user mappings |
+
+| Rejected for now | |
+| --- | --- |
+| `SELECT`, `INSERT`, `UPDATE`, `DELETE`, `COPY` | Data paths land in a later change |
+| `CREATE INDEX`, `TABLESAMPLE`, `TRUNCATE`, `CLUSTER`, `VACUUM FULL` | Not meaningful, or unsafe, without the data path |
+| Every `ALTER` form on a lake table, and on servers a lake table references | Schema evolution and identity tracking land in a later change |
+| `CREATE TABLE AS`, materialized views, partitioning, inheritance, typed tables, `TEMP`, `UNLOGGED`, `ON COMMIT`, explicit `TABLESPACE` | Out of scope for the skeleton |
+
+## Running the tests
+
+Against a running cluster with the library preloaded:
+
+```sh
+make USE_PGXS=1 installcheck
+```
+
+The suite covers the DDL path (including per-segment catalog state), the
+rejection matrices, and the privilege model.
+
+## Layout
+
+| Directory | Contents |
+| --- | --- |
+| `src/am_iceberg/` | Table access method, utility and object-access hooks, binding resolution, GUCs |
+| `src/iceberg_catalog_fdw/`, `src/iceberg_volume_fdw/` | Foreign-data-wrapper validators for the two server kinds |
+| `src/meta/` | `IcebergMetaEngine` vtable, registry with capability checking, stub engine |
+| `src/format/` | `FormatReader` / `FormatWriter` interfaces and their registry |
+| `src/common/` | Storage facade, canonical location type, C/C++ exception boundary macros |

contrib/datalake_fdw/datalake_fdw--1.0.sql

@@ -0,0 +1,40 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ *   http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied.  See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ *
+ * contrib/datalake_fdw/datalake_fdw--1.0.sql
+ */
+
+-- complain if script is sourced in psql, rather than via CREATE EXTENSION
+\echo Use "CREATE EXTENSION datalake_fdw" to load this file. \quit
+
+CREATE FUNCTION iceberg_am_handler(internal)
+RETURNS table_am_handler AS 'MODULE_PATHNAME' LANGUAGE C;
+
+CREATE ACCESS METHOD iceberg TYPE TABLE HANDLER iceberg_am_handler;
+
+CREATE FUNCTION iceberg_catalog_fdw_validator(text[], oid)
+RETURNS void AS 'MODULE_PATHNAME' LANGUAGE C STRICT;
+
+CREATE FOREIGN DATA WRAPPER iceberg_catalog_fdw
+  VALIDATOR iceberg_catalog_fdw_validator;
+
+CREATE FUNCTION iceberg_volume_fdw_validator(text[], oid)
+RETURNS void AS 'MODULE_PATHNAME' LANGUAGE C STRICT;
+
+CREATE FOREIGN DATA WRAPPER iceberg_volume_fdw
+  VALIDATOR iceberg_volume_fdw_validator;

contrib/datalake_fdw/datalake_fdw.control

@@ -0,0 +1,23 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing,
+# software distributed under the License is distributed on an
+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+# KIND, either express or implied.  See the License for the
+# specific language governing permissions and limitations
+# under the License.
+#
+# contrib/datalake_fdw/datalake_fdw.control
+
+comment = 'Apache Iceberg lake tables for Cloudberry (demo skeleton)'
+default_version = '1.0'
+module_pathname = '$libdir/datalake_fdw'
+relocatable = false

contrib/datalake_fdw/expected/iceberg_am_acl.out

@@ -0,0 +1,63 @@
+-- Foreign-server USAGE is required; user mappings remain optional.
+SET client_min_messages = warning;
+RESET ROLE;
+DROP SCHEMA IF EXISTS dlskel_s CASCADE;
+DROP SERVER IF EXISTS dlskel_acl_cat CASCADE;
+DROP SERVER IF EXISTS dlskel_acl_vol CASCADE;
+DROP ROLE IF EXISTS dlskel_user;
+RESET client_min_messages;
+CREATE EXTENSION IF NOT EXISTS datalake_fdw;
+NOTICE:  extension "datalake_fdw" already exists, skipping
+CREATE SERVER dlskel_acl_cat
+  FOREIGN DATA WRAPPER iceberg_catalog_fdw
+  OPTIONS (type 'hive', uri 'thrift://fake:9083');
+CREATE SERVER dlskel_acl_vol
+  FOREIGN DATA WRAPPER iceberg_volume_fdw
+  OPTIONS (base_path 's3://dlskel-bucket/acl',
+           endpoint 'http://fake:9000');
+CREATE ROLE dlskel_user LOGIN;
+NOTICE:  resource queue required -- using default resource queue "pg_default"
+CREATE SCHEMA dlskel_s;
+GRANT CREATE, USAGE ON SCHEMA dlskel_s TO dlskel_user;
+SET ROLE dlskel_user;
+CREATE TABLE dlskel_s.t (a int)
+  USING iceberg
+  WITH (catalog = 'dlskel_acl_cat', volume = 'dlskel_acl_vol');
+ERROR:  permission denied for foreign server dlskel_acl_cat
+RESET ROLE;
+GRANT USAGE ON FOREIGN SERVER dlskel_acl_cat TO dlskel_user;
+SET ROLE dlskel_user;
+CREATE TABLE dlskel_s.t (a int)
+  USING iceberg
+  WITH (catalog = 'dlskel_acl_cat', volume = 'dlskel_acl_vol');
+ERROR:  permission denied for foreign server dlskel_acl_vol
+RESET ROLE;
+GRANT USAGE ON FOREIGN SERVER dlskel_acl_vol TO dlskel_user;
+SET ROLE dlskel_user;
+CREATE TABLE dlskel_s.t (a int)
+  USING iceberg
+  WITH (catalog = 'dlskel_acl_cat', volume = 'dlskel_acl_vol');
+NOTICE:  stub engine: created iceberg table "dlskel_s.t" in catalog "dlskel_acl_cat"
+CREATE USER MAPPING FOR dlskel_user
+  SERVER dlskel_acl_cat
+  OPTIONS (user 'u', password 'p');
+ALTER USER MAPPING FOR dlskel_user
+  SERVER dlskel_acl_cat
+  OPTIONS (ADD warehouse 'x');
+ERROR:  invalid iceberg catalog user mapping option "warehouse"
+HINT:  Allowed options are "user", "password", and "token".
+RESET ROLE;
+DROP SCHEMA dlskel_s CASCADE;
+NOTICE:  drop cascades to table dlskel_s.t
+NOTICE:  stub engine: dropped iceberg table "dlskel_s.t" from catalog "dlskel_acl_cat"
+DROP USER MAPPING FOR dlskel_user SERVER dlskel_acl_cat;
+DROP SERVER dlskel_acl_cat;
+DROP SERVER dlskel_acl_vol;
+DROP ROLE dlskel_user;
+SET client_min_messages = warning;
+RESET ROLE;
+DROP SCHEMA IF EXISTS dlskel_s CASCADE;
+DROP SERVER IF EXISTS dlskel_acl_cat CASCADE;
+DROP SERVER IF EXISTS dlskel_acl_vol CASCADE;
+DROP ROLE IF EXISTS dlskel_user;
+RESET client_min_messages;