[BUG] Cluster startup fails with SparkMisconfiguration error on fully_private workspace

[yotaenom]

Category

core-platform

Project Name

aws-byovpc-backend-private-link

What's the bug?

In a newly deployed workspace, starting a cluster fails with the following error message: “Spark Driver was down due to misconfiguration, please check your config. [details] SparkMisconfiguration: java.lang.UnsupportedOperationException: scanEntries is not supported in LocalStubWrapper.”

This workspace is deployed in a fully_private VPC (no NAT/IGW), and DBFS is enabled.
From my investigation, it appears that in this fully_private configuration, enabling DBFS causes the above SparkMisconfiguration and prevents the cluster from starting.

Since this is a fully_private workspace, we need to consider DBFS to be disabled.

How to reproduce?

1. Deploy a Databricks workspace using the provided templates with a fully_private VPC configuration (no NAT gateway / Internet Gateway).
2. Check to see that DBFS is enabled in the workspace/cluster configuration.
3. Create and start a new cluster with default settings.
4. Observe that the cluster fails to start with the error: “Spark Driver was down due to misconfiguration … SparkMisconfiguration: java.lang.UnsupportedOperationException: scanEntries is not supported in LocalStubWrapper.”

Environment

• AWS
• Databricks Runtime: 17.3 LTS (includes Apache Spark 4.0.0, Scala 2.13)
• Preferred Node Type: md-fleet.xlarge

Checklist

• I searched existing issues first
• No sensitive information included