How to interpret and fix 'shared object not found' error when importing library in R on MacOS in a conda environment?

I am attempting to import an R package from a specific R path in a conda environment, but I keep receiving the following error:

Error: package or namespace load failed for ‘marginaleffects’ in library.dynam(lib, package, package.lib):
shared object ‘data_table.so’ not found

I am struggling to understand not just how to fix this error, but also what this error means or where it is coming from.

Specifically, I am deploying a web application with the Python-based streamlit framework and am using the package rpy2 to interface from an existing R codebase to Python. I use conda/mamba as the package manager on my machine. I have installed all R packages using their conda-forge distribution as possible. There are several R packages that are not available, so I have installed them in R to the path of my desired library with with: install.packages("marginaleffects", dep = TRUE, lib = "/Users/myuser/mambaforge/envs/myenv/lib/R/library", verbose = TRUE). The package then appears to exist in the directory .../myenv/lib/R/library, in installed.packages(), and in RStudio's list of packages.

However, when I attempt to load the package with library(marginaleffects, lib.loc="/Users/myuser/mambaforge/envs/myenv/lib/R/library") I receive the error noted above. Note that if I import this Packages in an R code run in the streamlit app, it raises a slightly different error relating to a .dylib shared object, instead of an '.so` shared object.

RRuntimeError: Error: package or namespace load failed for ‘marginaleffects’ in library.dynam(lib, package, package.lib): shared object ‘marginaleffects.dylib’ not found

I have the vaguest sense this has something to do with a compiler error. I have also been unsuccessful in installing the package through the rpy2 framework, receiving a long and fairly inscrutable error message.

EDIT

The code I have run so far for reproducibility on a MacOS (Ventura, same issue happened on Big Sur):

zsh> mamba create -n myenv r-essentials r-base python=3.8
zsh> mamba activate myenv
zsh> mamba install -c r rpy2 
zsh> mamba install -c conda-forge pandas
zsh> mamba install -c conda-forge streamlit

R> install.packages("clarify", dep = TRUE, lib = "/Users/myname/mambaforge/envs/myenv/lib/R/library", verbose = TRUE)
R> install.packages("marginaleffects", dep = TRUE, lib = "/Users/myname/mambaforge/envs/myenv/lib/R/library", verbose = TRUE)
R> library(marginaleffects, lib.loc="/Users/myuser/mambaforge/envs/myenv/lib/R/library")

Error: package or namespace load failed for ‘marginaleffects’ in library.dynam(lib, package, package.lib):
shared object ‘data_table.so’ not found

EDIT2

I am running osx-64.

Solution

I can recreate the issue on osx-arm64 platform (note that everything works fine for osx-64):

so-rpy2-clarify.yaml

name: so-rpy2-clarify
channels:
  - conda-forge  # only Conda Forge
  - nodefaults   # ignore user config
dependencies:
  ## Python
  - python=3.8
  - rpy2
  - pandas
  - streamlit

  ## R
  - r-base=4.3  # always define R version
  - r-clarify
  - r-essentials
  - r-marginaleffects

which running with:

mamba env create -f so-rpy2-clarify.yaml

gives:

Could not solve for environment specs
The following package could not be installed
└─ r-clarify is not installable because it requires
   └─ r-mvnfast, which does not exist (perhaps a missing channel).

This package isn't yet being built for osx-arm64, but we can put in a request (see the documentation on "Apple Silicon Builds").

Let's also check to make sure nothing else also is needed, since the conflict reporting usually just reports the first problem. We use mamba repoquery for this:

mamba repoquery depends -c conda-forge -p osx-arm64 r-clarify

Executing the query r-clarify

conda-forge/osx-arm64                                       Using cache
conda-forge/noarch                                          Using cache


 Name                        Version Build             Channel    
───────────────────────────────────────────────────────────────────
 r-clarify                   0.1.3   r42hc72bb7e_0     conda-forge
 r-rlang                     0.4.11  r40h6f62c66_0     conda-forge
 r-ggplot2                   3.1.0   r351h6115d3f_1000 conda-forge
 r-pbapply                   1.3_4   r351h6115d3f_1000 conda-forge
 r-mvnfast >>> NOT FOUND <<<                                      
 r-base                      4.2.3   heabe65b_0        conda-forge
 r-chk                       0.9.0   r43hc72bb7e_1     conda-forge
 r-insight                   0.19.3  r43hc72bb7e_0     conda-forge
 r-marginaleffects           0.9.0   r42hc72bb7e_0     conda-forge

Looks like r-mvnfast is the only missing dependency.