I want to execute data migration through Kubernetes cronjob using Luigi pipeline. My luigi task receives --start
parameter which I want to pass through cronjob command.
apiVersion: batch/v1beta1
kind: CronJob
metadata:
name: migration
spec:
schedule: "0 0 */1 * *"
jobTemplate:
spec:
template:
spec:
serviceAccountName: spark
containers:
- name: migratecronjob
image: latest-image
imagePullPolicy: Always
env:
- name: TAG
value: latest-image
- name: MIGRATION_TAG
value: migration-v05
command:
- "luigi"
- "--module"
- "module.task"
- "Migrate10Days"
- "--start"
- $(date +%Y-%m-%dT%H)
- "--workers"
- "10"
restartPolicy: OnFailure
The cronjob cannot recognize the $(date +%Y-%m-%dT%H)
as a bash script and pass this command as string to the luigi task.
I am not sure what are you going to archive, but this should work:
- command:
- sh
- -c
- 'exec luigi --module module.task Migrate10Days --start $(date +%Y-%m-%dT%H) --workers --workers'