diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index bf2263c228b..94bc08a8f14 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -127,15 +127,15 @@ mvn -pl dist -PnoSnapshots package -DskipTests Verify that shim-specific classes are hidden from a conventional classloader. ```bash -$ javap -cp dist/target/rapids-4-spark_2.12-26.10.0-SNAPSHOT-cuda12.jar com.nvidia.spark.rapids.shims.SparkShimImpl +$ javap -cp dist/target/rapids-4-spark_2.12-26.12.0-SNAPSHOT-cuda12.jar com.nvidia.spark.rapids.shims.SparkShimImpl Error: class not found: com.nvidia.spark.rapids.shims.SparkShimImpl ``` However, its bytecode can be loaded if prefixed with `spark3XY` not contained in the package name ```bash -$ javap -cp dist/target/rapids-4-spark_2.12-26.10.0-SNAPSHOT-cuda12.jar spark330.com.nvidia.spark.rapids.shims.SparkShimImpl | head -2 -Warning: File dist/target/rapids-4-spark_2.12-26.10.0-SNAPSHOT-cuda12.jar(/spark330/com/nvidia/spark/rapids/shims/SparkShimImpl.class) does not contain class spark330.com.nvidia.spark.rapids.shims.SparkShimImpl +$ javap -cp dist/target/rapids-4-spark_2.12-26.12.0-SNAPSHOT-cuda12.jar spark330.com.nvidia.spark.rapids.shims.SparkShimImpl | head -2 +Warning: File dist/target/rapids-4-spark_2.12-26.12.0-SNAPSHOT-cuda12.jar(/spark330/com/nvidia/spark/rapids/shims/SparkShimImpl.class) does not contain class spark330.com.nvidia.spark.rapids.shims.SparkShimImpl Compiled from "SparkShims.scala" public final class com.nvidia.spark.rapids.shims.SparkShimImpl { ``` @@ -177,7 +177,7 @@ mvn package -pl dist -am -Dbuildver=340 -DallowConventionalDistJar=true Verify `com.nvidia.spark.rapids.shims.SparkShimImpl` is conventionally loadable: ```bash -$ javap -cp dist/target/rapids-4-spark_2.12-26.10.0-SNAPSHOT-cuda12.jar com.nvidia.spark.rapids.shims.SparkShimImpl | head -2 +$ javap -cp dist/target/rapids-4-spark_2.12-26.12.0-SNAPSHOT-cuda12.jar com.nvidia.spark.rapids.shims.SparkShimImpl | head -2 Compiled from "SparkShims.scala" public final class com.nvidia.spark.rapids.shims.SparkShimImpl { ``` diff --git a/README.md b/README.md index 7c820aff1e3..ea10d8e9a10 100644 --- a/README.md +++ b/README.md @@ -78,7 +78,7 @@ GPU-accelerated UDFs) we recommend you declare the distribution artifact as a `p com.nvidia rapids-4-spark_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT provided ``` diff --git a/aggregator/pom.xml b/aggregator/pom.xml index a63b020ef14..99982fe7135 100644 --- a/aggregator/pom.xml +++ b/aggregator/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../pom.xml rapids-4-spark-aggregator_2.12 RAPIDS Accelerator for Apache Spark Aggregator Creates an aggregated shaded package of the RAPIDS plugin for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT aggregator diff --git a/api_validation/pom.xml b/api_validation/pom.xml index 514476558a8..ac9eb4463a7 100644 --- a/api_validation/pom.xml +++ b/api_validation/pom.xml @@ -22,11 +22,11 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-api-validation_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT api_validation diff --git a/datagen/README.md b/datagen/README.md index ad4baf791fe..5aeaf9edebf 100644 --- a/datagen/README.md +++ b/datagen/README.md @@ -23,12 +23,12 @@ Where `$SPARK_VERSION` is a compressed version number, like 330 for Spark 3.3.0. After this the jar should be at `target/datagen_2.12-$PLUGIN_VERSION-spark$SPARK_VERSION.jar` -for example a Spark 3.3.0 jar for the 26.10.0 release would be -`target/datagen_2.12-26.10.0-spark330.jar` +for example a Spark 3.3.0 jar for the 26.12.0 release would be +`target/datagen_2.12-26.12.0-spark330.jar` To get a spark shell with this you can run ```shell -spark-shell --jars target/datagen_2.12-26.10.0-spark330.jar +spark-shell --jars target/datagen_2.12-26.12.0-spark330.jar ``` After that you should be good to go. diff --git a/datagen/ScaleTest.md b/datagen/ScaleTest.md index 8e2731e16f3..1f55ac8070c 100644 --- a/datagen/ScaleTest.md +++ b/datagen/ScaleTest.md @@ -44,7 +44,7 @@ $SPARK_HOME/bin/spark-submit \ --conf spark.sql.parquet.datetimeRebaseModeInWrite=CORRECTED \ --class com.nvidia.rapids.tests.scaletest.ScaleTestDataGen \ # the main class --jars $SPARK_HOME/examples/jars/scopt_2.12-3.7.1.jar \ # one dependency jar just shipped with Spark under $SPARK_HOME -./target/datagen_2.12-26.10.0-SNAPSHOT-spark332.jar \ +./target/datagen_2.12-26.12.0-SNAPSHOT-spark332.jar \ 1 \ 10 \ parquet \ diff --git a/datagen/pom.xml b/datagen/pom.xml index 9d956981e56..6779df5d4c6 100644 --- a/datagen/pom.xml +++ b/datagen/pom.xml @@ -21,13 +21,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml datagen_2.12 Data Generator Tools for generating large amounts of data - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT datagen diff --git a/delta-lake/delta-20x/pom.xml b/delta-lake/delta-20x/pom.xml index 83370987b00..4452bc3cf3f 100644 --- a/delta-lake/delta-20x/pom.xml +++ b/delta-lake/delta-20x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-20x_2.12 RAPIDS Accelerator for Apache Spark Delta Lake 2.0.x Support Delta Lake 2.0.x support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-20x diff --git a/delta-lake/delta-21x/pom.xml b/delta-lake/delta-21x/pom.xml index 1e38c410a03..3280ab8d610 100644 --- a/delta-lake/delta-21x/pom.xml +++ b/delta-lake/delta-21x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-21x_2.12 RAPIDS Accelerator for Apache Spark Delta Lake 2.1.x Support Delta Lake 2.1.x support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-21x diff --git a/delta-lake/delta-22x/pom.xml b/delta-lake/delta-22x/pom.xml index 3ff57338620..ed9bdf9f2ff 100644 --- a/delta-lake/delta-22x/pom.xml +++ b/delta-lake/delta-22x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-22x_2.12 RAPIDS Accelerator for Apache Spark Delta Lake 2.2.x Support Delta Lake 2.2.x support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-22x diff --git a/delta-lake/delta-23x/pom.xml b/delta-lake/delta-23x/pom.xml index 5d2cf1dbf7e..8fd3f0507f6 100644 --- a/delta-lake/delta-23x/pom.xml +++ b/delta-lake/delta-23x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-23x_2.12 RAPIDS Accelerator for Apache Spark Delta Lake 2.3.x Support Delta Lake 2.3.x support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-23x diff --git a/delta-lake/delta-24x/pom.xml b/delta-lake/delta-24x/pom.xml index 8874388e556..260254d21be 100644 --- a/delta-lake/delta-24x/pom.xml +++ b/delta-lake/delta-24x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-24x_2.12 RAPIDS Accelerator for Apache Spark Delta Lake 2.4.x Support Delta Lake 2.4.x support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-24x diff --git a/delta-lake/delta-33x/pom.xml b/delta-lake/delta-33x/pom.xml index 66828eb202d..32597ce572b 100644 --- a/delta-lake/delta-33x/pom.xml +++ b/delta-lake/delta-33x/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-33x_2.12 RAPIDS Accelerator for Apache Spark Delta Lake 3.3.x Support - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-33x diff --git a/delta-lake/delta-40x/pom.xml b/delta-lake/delta-40x/pom.xml index 5c5fa1e8187..fb16a6d0114 100644 --- a/delta-lake/delta-40x/pom.xml +++ b/delta-lake/delta-40x/pom.xml @@ -23,13 +23,13 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-40x_2.12 RAPIDS Accelerator for Apache Spark Delta Lake 4.0.x Support - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-40x diff --git a/delta-lake/delta-spark330db/pom.xml b/delta-lake/delta-spark330db/pom.xml index 1f54eae408e..ae5941a475a 100644 --- a/delta-lake/delta-spark330db/pom.xml +++ b/delta-lake/delta-spark330db/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../shim-deps/pom.xml rapids-4-spark-delta-spark330db_2.12 RAPIDS Accelerator for Apache Spark Databricks 11.3 Delta Lake Support Databricks 11.3 Delta Lake support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-spark330db diff --git a/delta-lake/delta-spark332db/pom.xml b/delta-lake/delta-spark332db/pom.xml index 065937f9e67..4bcf3f5d29b 100644 --- a/delta-lake/delta-spark332db/pom.xml +++ b/delta-lake/delta-spark332db/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../shim-deps/pom.xml rapids-4-spark-delta-spark332db_2.12 RAPIDS Accelerator for Apache Spark Databricks 12.2 Delta Lake Support Databricks 12.2 Delta Lake support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-spark332db diff --git a/delta-lake/delta-spark350db143/pom.xml b/delta-lake/delta-spark350db143/pom.xml index 2353c16759d..4d95b4dadb2 100644 --- a/delta-lake/delta-spark350db143/pom.xml +++ b/delta-lake/delta-spark350db143/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../shim-deps/pom.xml rapids-4-spark-delta-spark350db143_2.12 RAPIDS Accelerator for Apache Spark Databricks 14.3 Delta Lake Support Databricks 14.3 Delta Lake support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT false diff --git a/delta-lake/delta-spark400db173/pom.xml b/delta-lake/delta-spark400db173/pom.xml index bf374b363fc..29cf8395d24 100644 --- a/delta-lake/delta-spark400db173/pom.xml +++ b/delta-lake/delta-spark400db173/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../shim-deps/pom.xml rapids-4-spark-delta-spark400db173_2.12 RAPIDS Accelerator for Apache Spark Databricks 17.3 Delta Lake Support Databricks 17.3 Delta Lake support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-spark400db173 diff --git a/delta-lake/delta-stub/pom.xml b/delta-lake/delta-stub/pom.xml index 9dc9e710e69..a7e39ed9837 100644 --- a/delta-lake/delta-stub/pom.xml +++ b/delta-lake/delta-stub/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-stub_2.12 RAPIDS Accelerator for Apache Spark Delta Lake Stub Delta Lake stub for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-stub diff --git a/dist/pom.xml b/dist/pom.xml index e232b075524..3c043b58cd5 100644 --- a/dist/pom.xml +++ b/dist/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../pom.xml rapids-4-spark_2.12 RAPIDS Accelerator for Apache Spark Distribution Creates the distribution package of the RAPIDS plugin for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT com.nvidia diff --git a/docs/configs.md b/docs/configs.md index ac8a272ac23..229d3b15463 100644 --- a/docs/configs.md +++ b/docs/configs.md @@ -16,7 +16,7 @@ The following is the list of options that `rapids-plugin-4-spark` supports. On startup use: `--conf [conf key]=[conf value]`. For example: ``` -${SPARK_HOME}/bin/spark-shell --jars rapids-4-spark_2.12-26.10.0-SNAPSHOT-cuda12.jar \ +${SPARK_HOME}/bin/spark-shell --jars rapids-4-spark_2.12-26.12.0-SNAPSHOT-cuda12.jar \ --conf spark.plugins=com.nvidia.spark.SQLPlugin \ --conf spark.rapids.sql.concurrentGpuTasks=2 ``` diff --git a/docs/dev/shims.md b/docs/dev/shims.md index de2ecb3f660..429d164aff6 100644 --- a/docs/dev/shims.md +++ b/docs/dev/shims.md @@ -181,17 +181,17 @@ Using JarURLConnection URLs we create a Parallel World of the current version wi Spark 3.3.0's URLs: ```text -jar:file:/home/spark/rapids-4-spark_2.12-26.10.0.jar!/ -jar:file:/home/spark/rapids-4-spark_2.12-26.10.0.jar!/spark-shared/ -jar:file:/home/spark/rapids-4-spark_2.12-26.10.0.jar!/spark330/ +jar:file:/home/spark/rapids-4-spark_2.12-26.12.0.jar!/ +jar:file:/home/spark/rapids-4-spark_2.12-26.12.0.jar!/spark-shared/ +jar:file:/home/spark/rapids-4-spark_2.12-26.12.0.jar!/spark330/ ``` Spark 3.5.1's URLs: ```text -jar:file:/home/spark/rapids-4-spark_2.12-26.10.0.jar!/ -jar:file:/home/spark/rapids-4-spark_2.12-26.10.0.jar!/spark-shared/ -jar:file:/home/spark/rapids-4-spark_2.12-26.10.0.jar!/spark351/ +jar:file:/home/spark/rapids-4-spark_2.12-26.12.0.jar!/ +jar:file:/home/spark/rapids-4-spark_2.12-26.12.0.jar!/spark-shared/ +jar:file:/home/spark/rapids-4-spark_2.12-26.12.0.jar!/spark351/ ``` ### Classloader Coupling for Package-Private Integrations diff --git a/iceberg-common/pom.xml b/iceberg-common/pom.xml index 9b6f688f836..928f71c7479 100644 --- a/iceberg-common/pom.xml +++ b/iceberg-common/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-iceberg-common_2.12 Root-safe Apache Iceberg helpers for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT iceberg-common diff --git a/iceberg/iceberg-1-10-x/pom.xml b/iceberg/iceberg-1-10-x/pom.xml index 32ed835abf2..e265224bd3d 100644 --- a/iceberg/iceberg-1-10-x/pom.xml +++ b/iceberg/iceberg-1-10-x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-iceberg-1-10-x_2.12 RAPIDS Accelerator for Apache Iceberg Apache Iceberg support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../iceberg/iceberg-1-10-x diff --git a/iceberg/iceberg-1-11-x/pom.xml b/iceberg/iceberg-1-11-x/pom.xml index 012af9cc3b8..55af36ae506 100644 --- a/iceberg/iceberg-1-11-x/pom.xml +++ b/iceberg/iceberg-1-11-x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-iceberg-1-11-x_2.12 RAPIDS Accelerator for Apache Iceberg Apache Iceberg support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../iceberg/iceberg-1-11-x diff --git a/iceberg/iceberg-1-6-x/pom.xml b/iceberg/iceberg-1-6-x/pom.xml index 6adfe4d3621..e91199cd2ed 100644 --- a/iceberg/iceberg-1-6-x/pom.xml +++ b/iceberg/iceberg-1-6-x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-iceberg-1-6-x_2.12 RAPIDS Accelerator for Apache Iceberg Apache Iceberg support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../iceberg/iceberg-1-6-x diff --git a/iceberg/iceberg-1-9-x/pom.xml b/iceberg/iceberg-1-9-x/pom.xml index 6361f5d72b4..aef98605750 100644 --- a/iceberg/iceberg-1-9-x/pom.xml +++ b/iceberg/iceberg-1-9-x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-iceberg-1-9-x_2.12 RAPIDS Accelerator for Apache Iceberg Apache Iceberg support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../iceberg/iceberg-1-9-x diff --git a/iceberg/iceberg-stub/pom.xml b/iceberg/iceberg-stub/pom.xml index 9a3940afe8b..98f2e360e99 100644 --- a/iceberg/iceberg-stub/pom.xml +++ b/iceberg/iceberg-stub/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-iceberg-stub_2.12 RAPIDS Accelerator for Apache Spark Iceberg Stub Iceberg stub for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../iceberg/iceberg-stub diff --git a/integration_tests/README.md b/integration_tests/README.md index f1ee94a8134..140f4f93af3 100644 --- a/integration_tests/README.md +++ b/integration_tests/README.md @@ -265,7 +265,7 @@ individually, so you don't risk running unit tests along with the integration te http://www.scalatest.org/user_guide/using_the_scalatest_shell ```shell -spark-shell --jars rapids-4-spark-tests_2.12-26.10.0-SNAPSHOT-tests.jar,rapids-4-spark-integration-tests_2.12-26.10.0-SNAPSHOT-tests.jar,scalatest_2.12-3.0.5.jar,scalactic_2.12-3.0.5.jar +spark-shell --jars rapids-4-spark-tests_2.12-26.12.0-SNAPSHOT-tests.jar,rapids-4-spark-integration-tests_2.12-26.12.0-SNAPSHOT-tests.jar,scalatest_2.12-3.0.5.jar,scalactic_2.12-3.0.5.jar ``` First you import the `scalatest_shell` and tell the tests where they can find the test files you @@ -288,7 +288,7 @@ If you just want to verify the SQL replacement is working you will need to add t assumes CUDA 12 is being used and the Spark distribution is built with Scala 2.12. ``` -$SPARK_HOME/bin/spark-submit --jars "rapids-4-spark_2.12-26.10.0-SNAPSHOT-cuda12.jar" ./runtests.py +$SPARK_HOME/bin/spark-submit --jars "rapids-4-spark_2.12-26.12.0-SNAPSHOT-cuda12.jar" ./runtests.py ``` You don't have to enable the plugin for this to work, the test framework will do that for you. @@ -521,7 +521,7 @@ To run cudf_udf tests, need following configuration changes: As an example, here is the `spark-submit` command with the cudf_udf parameter on CUDA 12: ``` -$SPARK_HOME/bin/spark-submit --jars "rapids-4-spark_2.12-26.10.0-SNAPSHOT-cuda12.jar,rapids-4-spark-tests_2.12-26.10.0-SNAPSHOT.jar" --conf spark.rapids.memory.gpu.allocFraction=0.3 --conf spark.rapids.python.memory.gpu.allocFraction=0.3 --conf spark.rapids.python.concurrentPythonWorkers=2 --py-files "rapids-4-spark_2.12-26.10.0-SNAPSHOT-cuda12.jar" --conf spark.executorEnv.PYTHONPATH="rapids-4-spark_2.12-26.10.0-SNAPSHOT-cuda12.jar" ./runtests.py --cudf_udf +$SPARK_HOME/bin/spark-submit --jars "rapids-4-spark_2.12-26.12.0-SNAPSHOT-cuda12.jar,rapids-4-spark-tests_2.12-26.12.0-SNAPSHOT.jar" --conf spark.rapids.memory.gpu.allocFraction=0.3 --conf spark.rapids.python.memory.gpu.allocFraction=0.3 --conf spark.rapids.python.concurrentPythonWorkers=2 --py-files "rapids-4-spark_2.12-26.12.0-SNAPSHOT-cuda12.jar" --conf spark.executorEnv.PYTHONPATH="rapids-4-spark_2.12-26.12.0-SNAPSHOT-cuda12.jar" ./runtests.py --cudf_udf ``` ### Enabling fuzz tests diff --git a/integration_tests/ScaleTest.md b/integration_tests/ScaleTest.md index 957ca371b76..4dee970a08a 100644 --- a/integration_tests/ScaleTest.md +++ b/integration_tests/ScaleTest.md @@ -98,7 +98,7 @@ $SPARK_HOME/bin/spark-submit \ --conf spark.sql.parquet.datetimeRebaseModeInWrite=CORRECTED \ --jars $SPARK_HOME/examples/jars/scopt_2.12-3.7.1.jar \ --class com.nvidia.spark.rapids.tests.scaletest.ScaleTest \ -./target/rapids-4-spark-integration-tests_2.12-26.10.0-SNAPSHOT-spark332.jar \ +./target/rapids-4-spark-integration-tests_2.12-26.12.0-SNAPSHOT-spark332.jar \ 10 \ 100 \ parquet \ diff --git a/integration_tests/pom.xml b/integration_tests/pom.xml index dbf9184905d..da6707c0d83 100644 --- a/integration_tests/pom.xml +++ b/integration_tests/pom.xml @@ -22,11 +22,11 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-integration-tests_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT integration_tests diff --git a/jenkins/databricks/create.py b/jenkins/databricks/create.py index a1cf209e843..684d3efc2c3 100644 --- a/jenkins/databricks/create.py +++ b/jenkins/databricks/create.py @@ -27,7 +27,7 @@ def main(): workspace = 'https://dbc-9ff9942e-a9c4.cloud.databricks.com' token = '' sshkey = '' - cluster_name = 'CI-GPU-databricks-26.10.0-SNAPSHOT' + cluster_name = 'CI-GPU-databricks-26.12.0-SNAPSHOT' idletime = 240 runtime = '14.3.x-gpu-ml-scala2.12' num_workers = 1 diff --git a/jenkins/databricks/init_cudf_udf.sh b/jenkins/databricks/init_cudf_udf.sh index 14da5bdb2d3..5cba66a575c 100755 --- a/jenkins/databricks/init_cudf_udf.sh +++ b/jenkins/databricks/init_cudf_udf.sh @@ -20,7 +20,7 @@ set -ex -CUDF_VER=${CUDF_VER:-26.10.0-SNAPSHOT} +CUDF_VER=${CUDF_VER:-26.12.0-SNAPSHOT} CUDA_VER=${CUDA_VER:-12.9} # Need to explicitly add conda into PATH environment, to activate conda environment. diff --git a/jenkins/version-def.sh b/jenkins/version-def.sh index 7d208f8ef61..25ac56435fb 100755 --- a/jenkins/version-def.sh +++ b/jenkins/version-def.sh @@ -34,8 +34,8 @@ fi CUDA_CLASSIFIER=${CUDA_CLASSIFIER:-"cuda12"} CLASSIFIER=${CLASSIFIER:-"$CUDA_CLASSIFIER"} # default as CUDA_CLASSIFIER for compatibility -PROJECT_VER=${PROJECT_VER:-"26.10.0-SNAPSHOT"} -PROJECT_TEST_VER=${PROJECT_TEST_VER:-"26.10.0-SNAPSHOT"} +PROJECT_VER=${PROJECT_VER:-"26.12.0-SNAPSHOT"} +PROJECT_TEST_VER=${PROJECT_TEST_VER:-"26.12.0-SNAPSHOT"} SPARK_VER=${SPARK_VER:-"3.3.0"} SPARK_VER_213=${SPARK_VER_213:-"3.5.0"} SCALA_BINARY_VER=${SCALA_BINARY_VER:-"2.12"} diff --git a/pom.xml b/pom.xml index 9818c6e96f6..f76e05fab29 100644 --- a/pom.xml +++ b/pom.xml @@ -23,7 +23,7 @@ rapids-4-spark-parent_2.12 NVIDIA cuDF plugin for Apache Spark Root Project The root project of the NVIDIA cuDF plugin for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT pom https://nvidia.github.io/cudf-spark/ @@ -1082,9 +1082,9 @@ spark${buildver} cuda12 ${cuda.version} - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT cudf-spark-jni - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT cudf-spark-private_${scala.binary.version} 2.12 incremental diff --git a/scala2.13/aggregator/pom.xml b/scala2.13/aggregator/pom.xml index 746e83ad015..b451a4d6d90 100644 --- a/scala2.13/aggregator/pom.xml +++ b/scala2.13/aggregator/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../pom.xml rapids-4-spark-aggregator_2.13 RAPIDS Accelerator for Apache Spark Aggregator Creates an aggregated shaded package of the RAPIDS plugin for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT aggregator diff --git a/scala2.13/api_validation/pom.xml b/scala2.13/api_validation/pom.xml index 72c5e50eb6a..136364bdfbf 100644 --- a/scala2.13/api_validation/pom.xml +++ b/scala2.13/api_validation/pom.xml @@ -22,11 +22,11 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-api-validation_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT api_validation diff --git a/scala2.13/datagen/pom.xml b/scala2.13/datagen/pom.xml index 65806a44bbe..a837d000aff 100644 --- a/scala2.13/datagen/pom.xml +++ b/scala2.13/datagen/pom.xml @@ -21,13 +21,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml datagen_2.13 Data Generator Tools for generating large amounts of data - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT datagen diff --git a/scala2.13/delta-lake/delta-20x/pom.xml b/scala2.13/delta-lake/delta-20x/pom.xml index 9fb37f9cc54..41936128ad9 100644 --- a/scala2.13/delta-lake/delta-20x/pom.xml +++ b/scala2.13/delta-lake/delta-20x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-20x_2.13 RAPIDS Accelerator for Apache Spark Delta Lake 2.0.x Support Delta Lake 2.0.x support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-20x diff --git a/scala2.13/delta-lake/delta-21x/pom.xml b/scala2.13/delta-lake/delta-21x/pom.xml index a615134a4d3..8d0e812391c 100644 --- a/scala2.13/delta-lake/delta-21x/pom.xml +++ b/scala2.13/delta-lake/delta-21x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-21x_2.13 RAPIDS Accelerator for Apache Spark Delta Lake 2.1.x Support Delta Lake 2.1.x support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-21x diff --git a/scala2.13/delta-lake/delta-22x/pom.xml b/scala2.13/delta-lake/delta-22x/pom.xml index 58c691f7643..5110f42849b 100644 --- a/scala2.13/delta-lake/delta-22x/pom.xml +++ b/scala2.13/delta-lake/delta-22x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-22x_2.13 RAPIDS Accelerator for Apache Spark Delta Lake 2.2.x Support Delta Lake 2.2.x support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-22x diff --git a/scala2.13/delta-lake/delta-23x/pom.xml b/scala2.13/delta-lake/delta-23x/pom.xml index c7a22cc7150..f44889c68a3 100644 --- a/scala2.13/delta-lake/delta-23x/pom.xml +++ b/scala2.13/delta-lake/delta-23x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-23x_2.13 RAPIDS Accelerator for Apache Spark Delta Lake 2.3.x Support Delta Lake 2.3.x support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-23x diff --git a/scala2.13/delta-lake/delta-24x/pom.xml b/scala2.13/delta-lake/delta-24x/pom.xml index 30980a6049d..35eaa4fcc11 100644 --- a/scala2.13/delta-lake/delta-24x/pom.xml +++ b/scala2.13/delta-lake/delta-24x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-24x_2.13 RAPIDS Accelerator for Apache Spark Delta Lake 2.4.x Support Delta Lake 2.4.x support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-24x diff --git a/scala2.13/delta-lake/delta-33x/pom.xml b/scala2.13/delta-lake/delta-33x/pom.xml index 3ec73030477..11d9772c094 100644 --- a/scala2.13/delta-lake/delta-33x/pom.xml +++ b/scala2.13/delta-lake/delta-33x/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-33x_2.13 RAPIDS Accelerator for Apache Spark Delta Lake 3.3.x Support - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-33x diff --git a/scala2.13/delta-lake/delta-40x/pom.xml b/scala2.13/delta-lake/delta-40x/pom.xml index 49a58338cb0..328c62363ef 100644 --- a/scala2.13/delta-lake/delta-40x/pom.xml +++ b/scala2.13/delta-lake/delta-40x/pom.xml @@ -23,13 +23,13 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-40x_2.13 RAPIDS Accelerator for Apache Spark Delta Lake 4.0.x Support - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-40x diff --git a/scala2.13/delta-lake/delta-41x/pom.xml b/scala2.13/delta-lake/delta-41x/pom.xml index 9f0c8735fa3..8e109015f9d 100644 --- a/scala2.13/delta-lake/delta-41x/pom.xml +++ b/scala2.13/delta-lake/delta-41x/pom.xml @@ -23,13 +23,13 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-41x_2.13 RAPIDS Accelerator for Apache Spark Delta Lake 4.1.x Support - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-41x diff --git a/scala2.13/delta-lake/delta-42x/pom.xml b/scala2.13/delta-lake/delta-42x/pom.xml index 8e9cf72b9d2..fa7a9d9ad1e 100644 --- a/scala2.13/delta-lake/delta-42x/pom.xml +++ b/scala2.13/delta-lake/delta-42x/pom.xml @@ -23,13 +23,13 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-42x_2.13 RAPIDS Accelerator for Apache Spark Delta Lake 4.2.x Support - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-42x diff --git a/scala2.13/delta-lake/delta-43x/pom.xml b/scala2.13/delta-lake/delta-43x/pom.xml index 99ca1fc2c49..14ed473ea62 100644 --- a/scala2.13/delta-lake/delta-43x/pom.xml +++ b/scala2.13/delta-lake/delta-43x/pom.xml @@ -23,13 +23,13 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-43x_2.13 RAPIDS Accelerator for Apache Spark Delta Lake 4.3.x Support - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-43x diff --git a/scala2.13/delta-lake/delta-spark330db/pom.xml b/scala2.13/delta-lake/delta-spark330db/pom.xml index 1d64988d08b..e8e628d7bd9 100644 --- a/scala2.13/delta-lake/delta-spark330db/pom.xml +++ b/scala2.13/delta-lake/delta-spark330db/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../shim-deps/pom.xml rapids-4-spark-delta-spark330db_2.13 RAPIDS Accelerator for Apache Spark Databricks 11.3 Delta Lake Support Databricks 11.3 Delta Lake support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-spark330db diff --git a/scala2.13/delta-lake/delta-spark332db/pom.xml b/scala2.13/delta-lake/delta-spark332db/pom.xml index 01b23164064..9304b51c19b 100644 --- a/scala2.13/delta-lake/delta-spark332db/pom.xml +++ b/scala2.13/delta-lake/delta-spark332db/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../shim-deps/pom.xml rapids-4-spark-delta-spark332db_2.13 RAPIDS Accelerator for Apache Spark Databricks 12.2 Delta Lake Support Databricks 12.2 Delta Lake support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-spark332db diff --git a/scala2.13/delta-lake/delta-spark350db143/pom.xml b/scala2.13/delta-lake/delta-spark350db143/pom.xml index ae3e4a3e3d9..944876c86f4 100644 --- a/scala2.13/delta-lake/delta-spark350db143/pom.xml +++ b/scala2.13/delta-lake/delta-spark350db143/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../shim-deps/pom.xml rapids-4-spark-delta-spark350db143_2.13 RAPIDS Accelerator for Apache Spark Databricks 14.3 Delta Lake Support Databricks 14.3 Delta Lake support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT false diff --git a/scala2.13/delta-lake/delta-spark400db173/pom.xml b/scala2.13/delta-lake/delta-spark400db173/pom.xml index 9b2cf397dc5..421649e3f5f 100644 --- a/scala2.13/delta-lake/delta-spark400db173/pom.xml +++ b/scala2.13/delta-lake/delta-spark400db173/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../shim-deps/pom.xml rapids-4-spark-delta-spark400db173_2.13 RAPIDS Accelerator for Apache Spark Databricks 17.3 Delta Lake Support Databricks 17.3 Delta Lake support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-spark400db173 diff --git a/scala2.13/delta-lake/delta-stub/pom.xml b/scala2.13/delta-lake/delta-stub/pom.xml index e3857f6e1cf..7ad00cfa972 100644 --- a/scala2.13/delta-lake/delta-stub/pom.xml +++ b/scala2.13/delta-lake/delta-stub/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-delta-stub_2.13 RAPIDS Accelerator for Apache Spark Delta Lake Stub Delta Lake stub for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../delta-lake/delta-stub diff --git a/scala2.13/dist/pom.xml b/scala2.13/dist/pom.xml index 3d5c45a80f1..59c2f1cf2e8 100644 --- a/scala2.13/dist/pom.xml +++ b/scala2.13/dist/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../pom.xml rapids-4-spark_2.13 RAPIDS Accelerator for Apache Spark Distribution Creates the distribution package of the RAPIDS plugin for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT com.nvidia diff --git a/scala2.13/iceberg-common/pom.xml b/scala2.13/iceberg-common/pom.xml index 3087eac4baa..c138a3ba433 100644 --- a/scala2.13/iceberg-common/pom.xml +++ b/scala2.13/iceberg-common/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-iceberg-common_2.13 Root-safe Apache Iceberg helpers for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT iceberg-common diff --git a/scala2.13/iceberg/iceberg-1-10-x/pom.xml b/scala2.13/iceberg/iceberg-1-10-x/pom.xml index 42a2e6cd6a3..b319c4c7050 100644 --- a/scala2.13/iceberg/iceberg-1-10-x/pom.xml +++ b/scala2.13/iceberg/iceberg-1-10-x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-iceberg-1-10-x_2.13 RAPIDS Accelerator for Apache Iceberg Apache Iceberg support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../iceberg/iceberg-1-10-x diff --git a/scala2.13/iceberg/iceberg-1-11-x/pom.xml b/scala2.13/iceberg/iceberg-1-11-x/pom.xml index c17c4a510c5..f935dfdecf8 100644 --- a/scala2.13/iceberg/iceberg-1-11-x/pom.xml +++ b/scala2.13/iceberg/iceberg-1-11-x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-iceberg-1-11-x_2.13 RAPIDS Accelerator for Apache Iceberg Apache Iceberg support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../iceberg/iceberg-1-11-x diff --git a/scala2.13/iceberg/iceberg-1-6-x/pom.xml b/scala2.13/iceberg/iceberg-1-6-x/pom.xml index f8fd3bc0b9f..3894c375734 100644 --- a/scala2.13/iceberg/iceberg-1-6-x/pom.xml +++ b/scala2.13/iceberg/iceberg-1-6-x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-iceberg-1-6-x_2.13 RAPIDS Accelerator for Apache Iceberg Apache Iceberg support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../iceberg/iceberg-1-6-x diff --git a/scala2.13/iceberg/iceberg-1-9-x/pom.xml b/scala2.13/iceberg/iceberg-1-9-x/pom.xml index ba311489ca7..5f56b012946 100644 --- a/scala2.13/iceberg/iceberg-1-9-x/pom.xml +++ b/scala2.13/iceberg/iceberg-1-9-x/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-iceberg-1-9-x_2.13 RAPIDS Accelerator for Apache Iceberg Apache Iceberg support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../iceberg/iceberg-1-9-x diff --git a/scala2.13/iceberg/iceberg-stub/pom.xml b/scala2.13/iceberg/iceberg-stub/pom.xml index accdcc8eb53..bdc68cb6622 100644 --- a/scala2.13/iceberg/iceberg-stub/pom.xml +++ b/scala2.13/iceberg/iceberg-stub/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-iceberg-stub_2.13 RAPIDS Accelerator for Apache Spark Iceberg Stub Iceberg stub for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../iceberg/iceberg-stub diff --git a/scala2.13/iceberg/pom.xml b/scala2.13/iceberg/pom.xml index f6685cf43e5..df07a0a6bd2 100644 --- a/scala2.13/iceberg/pom.xml +++ b/scala2.13/iceberg/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../pom.xml rapids-4-spark-iceberg_2.13 RAPIDS Accelerator for Apache Iceberg Apache Iceberg support for the RAPIDS Accelerator for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT iceberg diff --git a/scala2.13/integration_tests/pom.xml b/scala2.13/integration_tests/pom.xml index 83541b90297..1266e19e79e 100644 --- a/scala2.13/integration_tests/pom.xml +++ b/scala2.13/integration_tests/pom.xml @@ -22,11 +22,11 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-integration-tests_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT integration_tests diff --git a/scala2.13/pom.xml b/scala2.13/pom.xml index fd4942da5ec..524845d2e9c 100644 --- a/scala2.13/pom.xml +++ b/scala2.13/pom.xml @@ -23,7 +23,7 @@ rapids-4-spark-parent_2.13 NVIDIA cuDF plugin for Apache Spark Root Project The root project of the NVIDIA cuDF plugin for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT pom https://nvidia.github.io/cudf-spark/ @@ -1082,9 +1082,9 @@ spark${buildver} cuda12 ${cuda.version} - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT cudf-spark-jni - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT cudf-spark-private_${scala.binary.version} 2.13 incremental diff --git a/scala2.13/shim-deps/cloudera/pom.xml b/scala2.13/shim-deps/cloudera/pom.xml index 6e4afddaa10..2f1514d36ad 100644 --- a/scala2.13/shim-deps/cloudera/pom.xml +++ b/scala2.13/shim-deps/cloudera/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-cdh-bom pom CDH Shim Dependencies - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/cloudera diff --git a/scala2.13/shim-deps/databricks/pom.xml b/scala2.13/shim-deps/databricks/pom.xml index 778aa2de5a6..00cf94be0c0 100644 --- a/scala2.13/shim-deps/databricks/pom.xml +++ b/scala2.13/shim-deps/databricks/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-db-bom pom Databricks Shim Dependencies - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/databricks diff --git a/scala2.13/shim-deps/pom.xml b/scala2.13/shim-deps/pom.xml index f3be33766a4..cd29c2a8ee4 100644 --- a/scala2.13/shim-deps/pom.xml +++ b/scala2.13/shim-deps/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../pom.xml rapids-4-spark-shim-deps-parent_2.13 pom Shim Dependencies Profiles - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT release330cdh diff --git a/scala2.13/shuffle-plugin/pom.xml b/scala2.13/shuffle-plugin/pom.xml index ae56c34cbea..8b0461a805d 100644 --- a/scala2.13/shuffle-plugin/pom.xml +++ b/scala2.13/shuffle-plugin/pom.xml @@ -21,13 +21,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-shuffle_2.13 RAPIDS Accelerator for Apache Spark Shuffle Plugin Accelerated shuffle plugin for the RAPIDS plugin for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT shuffle-plugin diff --git a/scala2.13/sql-plugin-api/pom.xml b/scala2.13/sql-plugin-api/pom.xml index 7d81264ef92..414f38005f2 100644 --- a/scala2.13/sql-plugin-api/pom.xml +++ b/scala2.13/sql-plugin-api/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-sql-plugin-api_2.13 Module for Non-Shimmable API - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT sql-plugin-api false diff --git a/scala2.13/sql-plugin-columnar/pom.xml b/scala2.13/sql-plugin-columnar/pom.xml index b4fa004f8e3..9a1b0c3495b 100644 --- a/scala2.13/sql-plugin-columnar/pom.xml +++ b/scala2.13/sql-plugin-columnar/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-sql-plugin-columnar_2.13 Root-safe columnar runtime plumbing for the cuDF Plugin - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT sql-plugin-columnar diff --git a/scala2.13/sql-plugin-fileio/pom.xml b/scala2.13/sql-plugin-fileio/pom.xml index de096ad80a4..a276a1439e6 100644 --- a/scala2.13/sql-plugin-fileio/pom.xml +++ b/scala2.13/sql-plugin-fileio/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-sql-plugin-fileio_2.13 Java-only file I/O runtime plumbing for the RAPIDS SQL plugin - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT sql-plugin-fileio diff --git a/scala2.13/sql-plugin-format/pom.xml b/scala2.13/sql-plugin-format/pom.xml index 0592d1125ca..d53eda035ff 100644 --- a/scala2.13/sql-plugin-format/pom.xml +++ b/scala2.13/sql-plugin-format/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-sql-plugin-format_2.13 Java-only FlatBuffers format classes for the RAPIDS SQL plugin - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT sql-plugin-format diff --git a/scala2.13/sql-plugin-shims/pom.xml b/scala2.13/sql-plugin-shims/pom.xml index 59e605787b7..cb2504220c1 100644 --- a/scala2.13/sql-plugin-shims/pom.xml +++ b/scala2.13/sql-plugin-shims/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-sql-shims_2.13 RAPIDS Accelerator for Apache Spark SQL Plugin Shims Compile-time isolated SQL plugin shims - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT sql-plugin-shims diff --git a/scala2.13/sql-plugin/pom.xml b/scala2.13/sql-plugin/pom.xml index 1fc2d30d773..2cde7e1441f 100644 --- a/scala2.13/sql-plugin/pom.xml +++ b/scala2.13/sql-plugin/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-sql_2.13 RAPIDS Accelerator for Apache Spark SQL Plugin The RAPIDS SQL plugin for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT sql-plugin diff --git a/scala2.13/tests/pom.xml b/scala2.13/tests/pom.xml index d27bac1e5b7..82b433a2170 100644 --- a/scala2.13/tests/pom.xml +++ b/scala2.13/tests/pom.xml @@ -21,13 +21,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-tests_2.13 RAPIDS Accelerator for Apache Spark Tests RAPIDS plugin for Apache Spark integration tests - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT tests diff --git a/scala2.13/tools/pom.xml b/scala2.13/tools/pom.xml index a0c47295b72..eda4d2fd338 100644 --- a/scala2.13/tools/pom.xml +++ b/scala2.13/tools/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../pom.xml rapids-4-spark-tools-support pom RAPIDS Accelerator for Apache Spark Tools Support Supporting code for RAPIDS Accelerator tools - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT com.nvidia diff --git a/scala2.13/udf-compiler/pom.xml b/scala2.13/udf-compiler/pom.xml index 1b96ece0598..5b3d442260c 100644 --- a/scala2.13/udf-compiler/pom.xml +++ b/scala2.13/udf-compiler/pom.xml @@ -21,13 +21,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.13 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-udf_2.13 RAPIDS Accelerator for Apache Spark Scala UDF Plugin The RAPIDS Scala UDF plugin for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT udf-compiler diff --git a/shim-deps/cloudera/pom.xml b/shim-deps/cloudera/pom.xml index 9feaadff4ba..f5923de7f9c 100644 --- a/shim-deps/cloudera/pom.xml +++ b/shim-deps/cloudera/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-cdh-bom pom CDH Shim Dependencies - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/cloudera diff --git a/shim-deps/databricks/pom.xml b/shim-deps/databricks/pom.xml index c0d0dbd4298..2eee4115933 100644 --- a/shim-deps/databricks/pom.xml +++ b/shim-deps/databricks/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../../pom.xml rapids-4-spark-db-bom pom Databricks Shim Dependencies - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/databricks diff --git a/shim-deps/pom.xml b/shim-deps/pom.xml index e1e293eb8bf..4b64069da0e 100644 --- a/shim-deps/pom.xml +++ b/shim-deps/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../pom.xml rapids-4-spark-shim-deps-parent_2.12 pom Shim Dependencies Profiles - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT release330cdh diff --git a/shuffle-plugin/pom.xml b/shuffle-plugin/pom.xml index 7328adfefa4..a14b02543e7 100644 --- a/shuffle-plugin/pom.xml +++ b/shuffle-plugin/pom.xml @@ -21,13 +21,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-shuffle_2.12 RAPIDS Accelerator for Apache Spark Shuffle Plugin Accelerated shuffle plugin for the RAPIDS plugin for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT shuffle-plugin diff --git a/sql-plugin-api/pom.xml b/sql-plugin-api/pom.xml index 7bb14018dd2..8b36eae3da9 100644 --- a/sql-plugin-api/pom.xml +++ b/sql-plugin-api/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-sql-plugin-api_2.12 Module for Non-Shimmable API - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT sql-plugin-api false diff --git a/sql-plugin-api/src/main/scala/com/nvidia/spark/rapids/ShimLoader.scala b/sql-plugin-api/src/main/scala/com/nvidia/spark/rapids/ShimLoader.scala index f62a320831a..056647b3036 100644 --- a/sql-plugin-api/src/main/scala/com/nvidia/spark/rapids/ShimLoader.scala +++ b/sql-plugin-api/src/main/scala/com/nvidia/spark/rapids/ShimLoader.scala @@ -48,11 +48,11 @@ import org.apache.spark.util.MutableURLClassLoader Each shim can see a consistent parallel world without conflicts by referencing only one conflicting directory. E.g., Spark 3.2.0 Shim will use - jar:file:/home/spark/rapids-4-spark_2.12-26.10.0.jar!/spark-shared/ - jar:file:/home/spark/rapids-4-spark_2.12-26.10.0.jar!/spark320/ + jar:file:/home/spark/rapids-4-spark_2.12-26.12.0.jar!/spark-shared/ + jar:file:/home/spark/rapids-4-spark_2.12-26.12.0.jar!/spark320/ Spark 3.3.1 will use - jar:file:/home/spark/rapids-4-spark_2.12-26.10.0.jar!/spark-shared/ - jar:file:/home/spark/rapids-4-spark_2.12-26.10.0.jar!/spark331/ + jar:file:/home/spark/rapids-4-spark_2.12-26.12.0.jar!/spark-shared/ + jar:file:/home/spark/rapids-4-spark_2.12-26.12.0.jar!/spark331/ Using these Jar URL's allows referencing different bytecode produced from identical sources by incompatible Scala / Spark dependencies. */ diff --git a/sql-plugin-columnar/pom.xml b/sql-plugin-columnar/pom.xml index 6482f4601ef..cf3bb6f0ae2 100644 --- a/sql-plugin-columnar/pom.xml +++ b/sql-plugin-columnar/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-sql-plugin-columnar_2.12 Root-safe columnar runtime plumbing for the cuDF Plugin - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT sql-plugin-columnar diff --git a/sql-plugin-fileio/pom.xml b/sql-plugin-fileio/pom.xml index 40028e94837..5f51dbf9d58 100644 --- a/sql-plugin-fileio/pom.xml +++ b/sql-plugin-fileio/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-sql-plugin-fileio_2.12 Java-only file I/O runtime plumbing for the RAPIDS SQL plugin - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT sql-plugin-fileio diff --git a/sql-plugin-format/pom.xml b/sql-plugin-format/pom.xml index a10313e7a19..8d25437feaf 100644 --- a/sql-plugin-format/pom.xml +++ b/sql-plugin-format/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-sql-plugin-format_2.12 Java-only FlatBuffers format classes for the RAPIDS SQL plugin - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT sql-plugin-format diff --git a/sql-plugin-shims/pom.xml b/sql-plugin-shims/pom.xml index fb1617a6a15..770bb29afad 100644 --- a/sql-plugin-shims/pom.xml +++ b/sql-plugin-shims/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-sql-shims_2.12 RAPIDS Accelerator for Apache Spark SQL Plugin Shims Compile-time isolated SQL plugin shims - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT sql-plugin-shims diff --git a/sql-plugin/pom.xml b/sql-plugin/pom.xml index 7fc717b9bb5..3fce9b0c4eb 100644 --- a/sql-plugin/pom.xml +++ b/sql-plugin/pom.xml @@ -22,13 +22,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-sql_2.12 RAPIDS Accelerator for Apache Spark SQL Plugin The RAPIDS SQL plugin for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT sql-plugin diff --git a/sql-plugin/src/main/scala/com/nvidia/spark/rapids/Plugin.scala b/sql-plugin/src/main/scala/com/nvidia/spark/rapids/Plugin.scala index 658edd804cd..4fb9c56561d 100644 --- a/sql-plugin/src/main/scala/com/nvidia/spark/rapids/Plugin.scala +++ b/sql-plugin/src/main/scala/com/nvidia/spark/rapids/Plugin.scala @@ -155,11 +155,11 @@ object RapidsPluginUtils extends Logging { val possibleRapidsJarURLs = classloader.getResources(propName).asScala.toSet.toSeq.filter { url => { val urlPath = url.toString - // Filter out submodule jars, e.g. rapids-4-spark-aggregator_2.12-26.10.0-spark341.jar, + // Filter out submodule jars, e.g. rapids-4-spark-aggregator_2.12-26.12.0-spark341.jar, // and files stored under subdirs of '!/', e.g. - // rapids-4-spark_2.12-26.10.0-cuda12.jar!/spark330/rapids4spark-version-info.properties + // rapids-4-spark_2.12-26.12.0-cuda12.jar!/spark330/rapids4spark-version-info.properties // We only want to find the main jar, e.g. - // rapids-4-spark_2.12-26.10.0-cuda12.jar!/rapids4spark-version-info.properties + // rapids-4-spark_2.12-26.12.0-cuda12.jar!/rapids4spark-version-info.properties !urlPath.contains("rapids-4-spark-") && urlPath.endsWith("!/" + propName) } } diff --git a/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala b/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala index 5408089fd7c..ca72c11d28d 100644 --- a/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala +++ b/sql-plugin/src/main/scala/com/nvidia/spark/rapids/RapidsConf.scala @@ -479,7 +479,7 @@ object RapidsConf extends Logging with RapidsConfEntries { |On startup use: `--conf [conf key]=[conf value]`. For example: | |``` - |${SPARK_HOME}/bin/spark-shell --jars rapids-4-spark_2.12-26.10.0-SNAPSHOT-cuda12.jar \ + |${SPARK_HOME}/bin/spark-shell --jars rapids-4-spark_2.12-26.12.0-SNAPSHOT-cuda12.jar \ |--conf spark.plugins=com.nvidia.spark.SQLPlugin \ |--conf spark.rapids.sql.concurrentGpuTasks=2 |``` diff --git a/tests/pom.xml b/tests/pom.xml index 681518ffbb0..06b3f7f24c8 100644 --- a/tests/pom.xml +++ b/tests/pom.xml @@ -21,13 +21,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-tests_2.12 RAPIDS Accelerator for Apache Spark Tests RAPIDS plugin for Apache Spark integration tests - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT tests diff --git a/tools/pom.xml b/tools/pom.xml index d704df58f50..ca261d12af0 100644 --- a/tools/pom.xml +++ b/tools/pom.xml @@ -22,14 +22,14 @@ com.nvidia rapids-4-spark-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../pom.xml rapids-4-spark-tools-support pom RAPIDS Accelerator for Apache Spark Tools Support Supporting code for RAPIDS Accelerator tools - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT com.nvidia diff --git a/udf-compiler/pom.xml b/udf-compiler/pom.xml index fa891354220..f1c946cf6de 100644 --- a/udf-compiler/pom.xml +++ b/udf-compiler/pom.xml @@ -21,13 +21,13 @@ com.nvidia rapids-4-spark-shim-deps-parent_2.12 - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT ../shim-deps/pom.xml rapids-4-spark-udf_2.12 RAPIDS Accelerator for Apache Spark Scala UDF Plugin The RAPIDS Scala UDF plugin for Apache Spark - 26.10.0-SNAPSHOT + 26.12.0-SNAPSHOT udf-compiler