hadoopapache-sparkhivespark-thriftserver

Spark thrift server unable to start


I am running spark 1.5.2 thrift server with Hive-1.2.1 on secured yarn-2.7.2 in windows using below command

spark-submit --class org.apache.spark.sql.hive.thriftserver.HiveThriftServer2 --master yarn-client "C:\Spark\lib\spark-hive-thriftserver_2.10-1.5.2.jar"

It stopped with below exception,

16/04/11 12:31:00 INFO AbstractService: Service:HiveServer2 is started.
16/04/11 12:31:00 INFO HiveThriftServer2: HiveThriftServer2 started
16/04/11 12:31:00 ERROR ThriftCLIService: Error starting HiveServer2: could not start ThriftBinaryCLIService
org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address hostname1/192.168.65.7:10000.
    at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:109)
    at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:91)
    at org.apache.thrift.transport.TServerSocket.<init>(TServerSocket.java:87)
    at org.apache.hive.service.auth.HiveAuthFactory.getServerSocket(HiveAuthFactory.java:241)
    at org.apache.hive.service.cli.thrift.ThriftBinaryCLIService.run(ThriftBinaryCLIService.java:66)
    at java.lang.Thread.run(Thread.java:744)
16/04/11 12:31:00 INFO HiveServer2: Shutting down HiveServer2
16/04/11 12:31:00 INFO AbstractService: Service:ThriftBinaryCLIService is stopped.

How to solve this.

Thanks.


Solution

  • Possible cause of the problem is that the port 10000 is already in use (as mentioned in your comment that Hiveserver is already running, which uses by default the port 10000).
    You could change it (to 10005 for example) when running thrift server.

    I would recommend that you start the thrift server as follow:

    $cd $SPARK_HOME
    $./sbin/start-thriftserver.sh --hiveconf hive.server2.thrift.port=10005 --master  yarn-client
    

    Please refer to the documentation here