[SOLVED] Apache Spark - OpenBlas and Lapack are not working in Ubuntu 20.04

Apache Spark - OpenBlas and Lapack are not working in Ubuntu 20.04

So, I am using spark 3.1.3 on standalone mode with 16 workers I have done all the necessary steps described in the Spark documentation but the native libraries still are not working. I have included the below dependency in the pom file:

<dependency>
   <groupId>com.github.fommil.netlib</groupId>
   <artifactId>all</artifactId>
   <version>1.1.2</version>
   <type>pom</type>
</dependency>

and I have done the installations:

sudo apt-get install libgfortran5
sudo apt-get install libatlas3-base libopenblas-base
sudo update-alternatives --config libblas.so
sudo update-alternatives --config libblas.so.3
sudo update-alternatives --config liblapack.so
sudo update-alternatives --config liblapack.so.3

Altough I am getting "update-alternatives: error: no alternatives for..." for all above alternatives.

I am still getting the WARN from Spark:

WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeSystemBLAS
WARN BLAS: Failed to load implementation from: com.github.fommil.netlib.NativeRefBLAS

And if I try: System.out.println(BLAS.getInstance().getClass().getName()); I get:

com.github.fommil.netlib.F2jBLAS

As you can I am using libgfortran5 because libgfortran is not available directly in Ubuntu 20.04. Is that the reason? Wouldn't libgfortran5 cover the previous versions?

Solution