kubernetescronluigi

Pass date command as parameter to kubernetes cronjob


I want to execute data migration through Kubernetes cronjob using Luigi pipeline. My luigi task receives --start parameter which I want to pass through cronjob command.

apiVersion: batch/v1beta1
kind: CronJob
metadata:
  name: migration
spec:
  schedule: "0 0 */1 * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: spark
          containers:
          - name: migratecronjob
            image: latest-image
            imagePullPolicy: Always
            env:
              - name: TAG
                value: latest-image
              - name: MIGRATION_TAG
                value: migration-v05
            command:
              -  "luigi"
              - "--module" 
              - "module.task" 
              - "Migrate10Days"
              - "--start"
              - $(date +%Y-%m-%dT%H)
              - "--workers"
              - "10"
          restartPolicy: OnFailure

The cronjob cannot recognize the $(date +%Y-%m-%dT%H) as a bash script and pass this command as string to the luigi task.


Solution

  • I am not sure what are you going to archive, but this should work:

          - command:
            - sh
            - -c
            - 'exec luigi --module module.task Migrate10Days --start $(date +%Y-%m-%dT%H) --workers --workers'