-
It definitely sounds like a complex issue, Eric. The error you’re seeing, DEADLINE_EXCEEDED, is generally indicative of a task that is taking too long, which as you’ve noted, isn’t due to the size of the query. Your cluster demonstrating successful communication with the universal analytics API would also suggest it’s not an inherent connectivity issue. Although, it could be caused by network latency or a server-side issue at Google. It might also be a threading issue in how the requests are being made. The first thing I would suggest is increasing the timeout setting to see if that resolves your issue. However, dependency issues could also be a potential cause in cases like this. Checking your cluster logs for BEGIN dependency errors or conflicts should give you a better understanding. Another avenue to investigate would be any differences in environmental conditions between your local setup and your Databricks environment. Lastly, given the unique combination of technologies and APIs, you might be encountering a less common, undocumented issue for which reaching out to the Databricks and/or Google Analytics 4 support teams may yield the fastest resolution.
