How to to trigger a Databricks job from another Databricks job?
I'm currently working on a project where I have two distinct jobs on Databricks. The second job is dependent on the results of the first one.
I am wondering if there is a way to automatically trigger the second job once the first one has completed successfully. Ideally, I would like to accomplish this directly within Databricks without the need for an external scheduling or orchestration tool. Has anyone been able to implement this type of setup or know if it's possible?
Solution
Databricks is now rolling out the new functionality, called "Job as a Task" that allows to trigger another job as a task in a workflow. Documentation isn't updated yet, but you may see it in the UI.
- Select "Run Job" when adding a new task:
- Select specific job to execute as a task:
- Printing secret value in Databricks
- Spark struct to case class conversion issues with getAs[T]
- how to set "api-version" dynamically in fs.azure.account.oauth2.msi.endpoint
- How to start a standalone cluster using pyspark?
- DESCRIBE TABLE see which columns are NOT NULL
- Spark sending LIMIT to SQL Server on display function
- Remove all records which are duplicate in spark dataframe
- Encoder for Row Type Spark Datasets
- Getting "Cannot broadcast the table that is larger than 8GB error" - Clarification
- KerberosAuthException in Argo for using PySpark
- Can you increase the spark shell output character limit
- What is the difference between Spark Standalone, YARN and local mode?
- How to find out the amount of memory pyspark has from iPython interface?
- Un-persisting all dataframes in (py)spark
- forming list of columns after groupByKey or groupBy
- convert Rdd to dataframe
- How to batch up items from a PySpark DataFrame
- How to load IPython shell with PySpark
- pyspark trimming all fields bydefault while writing into csv in python
- java.lang.NoClassDefFoundError: org/apache/hadoop/fs/StorageStatistics
- Spark Executor STDOUT to Kubernetes STDOUT
- How to add a new column when writing to a Delta table?
- How to parse nested JSON objects in Spark SQL?
- Reading Avro File in Spark
- How to cache dataframes that will be retained after spark session terminates
- Spark error message suggests checkpointing, but the error persists even after I add it
- How to configure Ray cluster to utilize the Full Capacity of Databricks Cluster
- Expanding dataframe column into multiple columns
- Why do columns change to nullable in Apache Spark SQL?
- How to split parquet files into many partitions in Spark?