So, I am using spark 3.1.3 on standalone mode with 16 workers I have done all the necessary steps described in the Spark documentation but the native libraries still are not working. I have included the below dependency in the pom file:
<dependency>
<groupId>com.github.fommil.netlib</groupId>
<artifactId>all</artifactId>
<version>1.1.2</version>
<type>pom</type>
</dependency>
and I have done the installations:
sudo apt-get install libgfortran5
sudo apt-get install libatlas3-base libopenblas-base
sudo update-alternatives --config libblas.so
sudo update-alternatives --config libblas.so.3
sudo update-alternatives --config liblapack.so
sudo update-alternatives --config liblapack.so.3
Altough I am getting "update-alternatives: error: no alternatives for..." for all above alternatives.
I am still getting the WARN from Spark:
WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS
And if I try: System.out.println(BLAS.getInstance().getClass().getName()); I get:
com.github.fommil.netlib.F2jBLAS
As you can I am using libgfortran5 because libgfortran is not available directly in Ubuntu 20.04. Is that the reason? Wouldn't libgfortran5 cover the previous versions?
Problem solved by installing gcc-6-base and libgfortran3 from here https://gist.github.com/sakethramanujam/faf5b677b6505437dbdd82170ac55322 on all workers and master.