pythonpython-3.xazureazure-hdinsight

How to upgrade Python version on Azure HDInsight


We need to use an external library which only runs on Python 3.6 or higher. Unfortunately, HDInsight only has Python 3.5 installed by default. Is there a way to upgrade the minor version for Python on HDInsight?

The official Azure documentation does not mention about it.

https://learn.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-python-package-installation


Solution

  • As stated in Microsoft Q&A:

    It's not recommended as using non-cluster built-in Python versions are unsupported scenario.

    WARNING!: HDInsight cluster depends on the built-in Python environment - Python 3.5. Directly installing custom packages in those default built-in environments may cause unexpected library version changes. And break the cluster further.

    If you want to install, you can use Python 3.6, change “python=3.5” in this command to python=3.6, and follow the rest steps in the document works.

    sudo /usr/bin/anaconda/bin/conda create --prefix /usr/bin/anaconda/envs/py36new python=3.6 anaconda –yes