In bash script, I can append & at the end of the command to make it run in the background such as:
top &
However, when I put that in a Azkaban job, such as:
type=command
command=top &
It will give the error message as:
29-04-2015 23:53:08 PDT start ERROR - invalid option or syntax: &
29-04-2015 23:53:08 PDT start ERROR - top usage: top
29-04-2015 23:53:08 PDT start ERROR - [-a | -d | -e | -c ]
29-04-2015 23:53:08 PDT start ERROR - [-F | -f]
29-04-2015 23:53:08 PDT start ERROR - [-h]
29-04-2015 23:53:08 PDT start ERROR - [-i ]
29-04-2015 23:53:08 PDT start ERROR - [-l ]
29-04-2015 23:53:08 PDT start ERROR - [-ncols ]
29-04-2015 23:53:08 PDT start ERROR - [-o ] [-O ]
29-04-2015 23:53:08 PDT start ERROR - [-R | -r]
29-04-2015 23:53:08 PDT start ERROR - [-S]
29-04-2015 23:53:08 PDT start ERROR - [-s ]
29-04-2015 23:53:08 PDT start ERROR - [-n ]
29-04-2015 23:53:08 PDT start ERROR - [-stats ]
29-04-2015 23:53:08 PDT start ERROR - [-pid ]
29-04-2015 23:53:08 PDT start INFO - Process completed unsuccessfully in 0 seconds.
29-04-2015 23:53:08 PDT start ERROR - [-user ]
29-04-2015 23:53:08 PDT start ERROR - [-U ]
29-04-2015 23:53:08 PDT start ERROR - [-u]
29-04-2015 23:53:08 PDT start ERROR -
29-04-2015 23:53:08 PDT start ERROR - Job run failed!
I tried to escape & to write the command as "top \&", but it would still report the same error. I googled for hours and couldn't find a solution, does anyone know how to run commands in the background with Azkaban?
Most probably, Azkaban executes the commands directly via OS calls, not through a shell - however, the & syntax is only understood by Bash itself. Try placing your command inside a Bash script:
#!/bin/bash
top &
and schedule
bash -c "your_script.sh"
to be run instead, or execute
bash -c "top &"
directly.
By the way, top won't run in the background without setting up a dumb terminal first. This is addressed e.g. here.