I am trying to run a hive script file with beeline command invoked in my python script like this:
!beeline -u "specified the connection" -f query.sql
This query is on hdfs, and a "!hdfs dfs -ls" command shows that the file is in fact there but beeline cannot find this. The connection works well and I can use beeline to execute simple queries with "-e" prefix but I wish to run a whole script. I am a beginner in using both beeline and working on hdfs, so I am sorry for not phrasing this issue properly. Could anyone suggest a solution?
The script file should be in the local filesystem where you are executing the beeline command not in HDFS location.
Post Hive version 4.0.0-alpha-1, HIVE-21511 allows beeline -e to use script file stored in other Filesystem apart from local filesystem, if you provide the complete absolute path. ex. hdfs://hacluster1/usr/query.sql, By default it will still try to find the query file in local filesystem only if the path isn't absolute.
For hive versions before that you need to keep the query file in the local filesystem only and then try.