azureapache-sparkazure-hub

Spark Deployment in Azure cloud


Is it possible to deploy spark code in Azure cloud without the yarn component? thanks in advance


Solution

  • I don't think it is possible to deploy HDInsight cluster without YARN.Refer to the HDInsight documentation

    YARN is the resource manager for Hadoop. Is there any particular reason you would not want to use YARN while working with HDInsight Spark cluster?

    If you want to use the standalone mode, you can modify the location of the master url while submitting the job using Spark-submit command.

    I have some examples in my repo with Spark-submit both in local mode and on HDInsight cluster https://github.com/NileshGule/learning-spark

    You can refer to