How to pass a file as command line argument to spark job in Oozie workflow? My spark job is expecting a file as command line argument, but when I am pass that file in the workflow as /file/location it is not picking up that file.
I got one workaround, if we put the file in a custom-directory in ozzie shared library with a few additional change in job.properties
- oozie.use.system.libpath=true
- oozie.action.sharelib.for.spark=spark,custom-directory
- oozie.libpath=true
Then we need to update the shared lib using below command:
oozie admin -auth SIMPLE -sharelibupdate
After that we can directly pick up the file by just using the name of the file, which we placed in custom-directory, in the oozie workflow.