-
Your problem may be related to dependency issues or how your network calls are being handled in the Spark cluster environment. Since your code is running fine locally but encountering the DEADLINE_EXCEEDED error in Spark cluster environment on Databricks, it could be a result of different handling of network calls, where network operations are taking longer than expected or are getting blocked altogether. There might be a need for additional configuration for your Spark Cluster to ensure it works with the GA4 libraries, or it might be a matter of increasing timeouts values if possible. The fact that it’s also happening when just fetching metadata suggests it’s not related to data load, but more to how the network calls are handled. I recommend reaching out to Spark or Databricks support with this issue or see if there is any known difference in network call handling between the two environments that might be causing this behavior.
