apache-sparkhadoophadoop-yarnhortonworks-data-platformhdp

How to run Spark 3.0.0 on HDP (Horthonworks)?

Is there a way to run a Spark 3.0 on HDP3 (Horthonworks)? I'm aware that there is always a standalone option, but I would like to configure YARN as a scheduler.

Solution

  • CDP 7.1.3 onwards, you can use Spark3.

    https://docs.cloudera.com/cdp-private-cloud-base/7.1.3/cds-3/topics/spark-spark-3-overview.html

    Installation Steps:

    https://docs.cloudera.com/cdp-private-cloud-base/7.1.3/cds-3/topics/spark-install-spark-3-parcel.html