I'm using an private server to submit some jobs on databricks using databricks-connect. Where is host Spark driver ? On private server and it just sends jobs to databricks worker ? I'm able to see .show() results directly on my server, what does it mean ? Am I using client mode, and spark driver stay on private server ?
Thank you
If you use Databricks Connect, the machine where it's installed is the Spark Driver, and Spark Executors are in your Data Plane.