azureazure-databricksdatabricks-cli

databricks cli to update job shcedule

I have configured the Databricks cli locally and able to connect to the Azure Databricks cluster. Link reference used for my workstation - git

$ databricks jobs list --profile dev

Say if I wanted to update only the schedule (cron expression) to a specific job which is already deployed in workspace, i don't see any option to do it using databricks CLI.

Note: In my case the the jobs are created using job definition json, is used to create the jobs in the cluster. This json doesn't have the schedule info to start with.

Is there are any options available to update only schedule, after the job is created or deployed in the workspace?

There is an option to run the command immediately, databricks jobs run-now.

The REST API configuration https://docs.databricks.com/dev-tools/api/latest/jobs.html#operation/JobsCreate

Solution

  • The Databricks jobs CLI supports calls to two versions of the Databricks Jobs REST API: versions 2.1 and 2.0. Version 2.1 adds support for orchestration of jobs with multiple tasks; see Jobs with multiple tasks and Jobs API updates.

    Going through the MS docs I see, you can use the update request to change an existing job.

    databricks jobs update --job-id 246 --json-file update-job.json

    schedule information block in the input json file.

    "schedule": {
      "quartz_cron_expression": "0 0 0 * * ?",
      "timezone_id": "US/Pacific",
      "pause_status": "UNPAUSED"
    }

    Refer: MS DOC - An example JSON document representing a single-task format job for API 2.0:

    {
  "job_id": 27,
  "settings": {
    "name": "Example notebook",
    "existing_cluster_id": "1201-my-cluster",
    "libraries": [
      {
        "jar": "dbfs:/FileStore/jars/spark_examples.jar"
      }
    ],
    "email_notifications": {},
    "timeout_seconds": 0,
    "schedule": {
      "quartz_cron_expression": "0 0 0 * * ?",
      "timezone_id": "US/Pacific",
      "pause_status": "UNPAUSED"
    },
    "notebook_task": {
      "notebook_path": "/notebooks/example-notebook",
      "revision_timestamp": 0
    },
    "max_concurrent_runs": 1,
    "format": "SINGLE_TASK"
  },
  "created_time": 1504128821443,
  "creator_user_name": "user@databricks.com"
}