Skip to content

docs: Update confs to bypass Iceberg Spark issues #2166

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
parthchandra merged 4 commits into from
Aug 19, 2025
Merged

docs: Update confs to bypass Iceberg Spark issues #2166

Changes from 3 commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
ea198d9
Update confs to bypass Iceberg Spark issues
hsiang-c Aug 16, 2025
4cf0b14
Update iceberg.md
hsiang-c Aug 18, 2025
47f85a3
Users can diable Spark's AQE as well
hsiang-c Aug 18, 2025
63d9b71
Let users turn off AQE or Comet's broadcastExchange
hsiang-c Aug 18, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 13 additions & 3 deletions docs/source/user-guide/iceberg.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -80,11 +80,14 @@ $SPARK_HOME/bin/spark-shell \
--conf spark.sql.catalog.spark_catalog.type=hadoop \
--conf spark.sql.catalog.spark_catalog.warehouse=/tmp/warehouse \
--conf spark.plugins=org.apache.spark.CometPlugin \
--conf spark.shuffle.manager=org.apache.spark.sql.comet.execution.shuffle.CometShuffleManager \
--conf spark.comet.exec.shuffle.enabled=false \
--conf spark.sql.iceberg.parquet.reader-type=COMET \
--conf spark.comet.explainFallback.enabled=true \
--conf spark.memory.offHeap.enabled=true \
--conf spark.memory.offHeap.size=2g
--conf spark.memory.offHeap.size=2g \
--conf spark.comet.use.lazyMaterialization=false \
--conf spark.comet.schemaEvolution.enabled=true \
--conf spark.sql.adaptive.enabled=false # or spark.comet.exec.broadcastExchange.enabled=false, see `Known issues`
Copy link
Copy Markdown
Contributor

@parthchandra parthchandra Aug 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would rather not have this here. Users that do not need this workaround may copy and paste from here and will lose functionality unnecessarily

```

Create an Iceberg table. Note that Comet will not accelerate this part.
Expand Down Expand Up @@ -138,4 +141,11 @@ scala> spark.sql(s"SELECT * from t1").explain()
== Physical Plan ==
*(1) CometColumnarToRow
+- CometBatchScan spark_catalog.default.t1[c0#26, c1#27] spark_catalog.default.t1 (branch=null) [filters=, groupedBy=] RuntimeFilters: []
```
```

## Known issues
- We temporarily disable Comet when there are delete files in Iceberg scan, see Iceberg [1.8.1 diff](../../../dev/diffs/iceberg/1.8.1.diff) and this [PR](https://github.com/apache/iceberg/pull/13793)
- Iceberg scan w/ delete files lead to [runtime exceptions](https://github.com/apache/datafusion-comet/issues/2117) and [incorrect results](https://github.com/apache/datafusion-comet/issues/2118)
- Enabling `CometShuffleManager` leads to [runtime exceptions](https://github.com/apache/datafusion-comet/issues/2086)
- Spark Runtime Filtering isn't [working](https://github.com/apache/datafusion-comet/issues/2116)
- You can bypass the issue by either setting `spark.sql.adaptive.enabled=false` or `spark.comet.exec.broadcastExchange.enabled=false`