hadoopapache-sparkhortonworks-data-platform

Get HDP version through Spark

We installed a new Spark version so all folders name are named similar to:

ls /etc/hadoop/
2.6.4.0-91  conf  conf.backup

and from spark-submit we get

spark-submit --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.2.0.2.6.4.0-91
      /_/

What I need is to identify the spark version - 2.6.4.0-91. We can do:

ls /etc/hadoop/ | grep [0-9]
2.6.4.0-91

but this approach is not so good (because other folder name with number could be there, etc.)

By spark-submit --version we got

version 2.2.0.2.6.4.0-91

while we need only the version - 2.6.4.0-91 , ( without 2.2.0 )

Any ides how to got the version - without (2.2.0) from spark-submit --version?

Solution

  • since hadoop is installed on the linux machine,

    as:

    rpm -qa |grep hadoop | grep client
hadoop_2_6_4_0_91-client-2.7.3.2.6.4.0-91.x86_64

    then I can use hdp-select command in order to get the required HDP version 

     hdp-select |  grep hadoop-client | awk '{print $3}'
 2.6.4.0-91

    or:

    hdp-select | grep hadoop-client
hadoop-client - 2.6.4.0-91