I installed miniconda3 on one linux account, then I created an environment py37, installed all the needed packages and was able to launch ipython from the second account and import the package I wanted to import: hail. For that I changed all of the permissions in the folder with miniconda3 to 777. Somehow, the command when run on the second account gets stuck, but when executed on the initial one where miniconda3 is installed, it runs successfully:
import hail as hl
---> mt = hl.balding_nichols_model(n_populations=3, n_samples=50, n_variants=100)
mt.count()
The middle command gets stuck. No error, it is just not returning. When I run hl.balding_nichols_model on the original account, it is also giving me a warning (but runs successfully, giving the result in mt.count()):
WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
And the thing is that on another account where the command fails I have Hadoop mounted. So, I have a slight suspicion it is somehow related. I am totally stuck, so would appreciate any suggestions. I need to do it this way - installing miniconda3 on one account, then accessing it from the other - because on the first account I have terabytes of data free, but on the second - 4Gb and it can't be further expanded unfortunately. With miniconda3 I would quickly go over the limits.
Additional info regarding the actual software can be found here:
https://hail.is/docs/0.2/getting_started.html#installation
Update
I installed separately python37 on the conda present on the node and somehow it does not work either, so it is not the problem of permissions, and the issue now is limited to that particular linux account. I installed spark2.4 but that did not fix the issue. So, the middle command in the python script gets stuck and I do not know even how to get the log output, what is going on there.
The answer to the 'stuck' issue can be found here:
https://discuss.hail.is/t/spark-2-4-4-gets-stuck-in-initialization-phase/1178
I asked it on Hail forum and then replied myself there after we fixed the issue. It turned out to be space issue: hadoop and spark logs should be redirected to different places when you do not have enough of space at the partition you are working on.