pythonwindowsdllpyd

Discover missing module using command-line ("DLL load failed" error)

On Windows, when we try to import a .pyd file, and a DLL that the .pyd depends on cannot be found, we get this traceback:

Traceback (most recent call last):
  ...
ImportError: DLL load failed: The specified module could not be found.

When this happens, often one has to resort to a graphical tool like Dependencies to figure out what is the name of the missing module.

How can I obtain the missing module name via the command-line?

Context: often we get this error in CI, and it would be easier to login via SSH to find out the missing module name, rather than having to log via GUI.

Solution

  • First, let's chose a concrete example: NumPy's _multiarray_umath*.pyd (from Python 3.9 (pc064)). Note that I'll be reusing this console:

    [cfati@CFATI-5510-0:e:\Work\Dev\StackOverflow\q074877580]> sopr.bat
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###

[prompt]>
[prompt]> "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Scripts\python.exe"
Python 3.9.9 (tags/v3.9.9:ccb0e6a, Nov 15 2021, 18:08:50) [MSC v.1929 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.
>>>
>>> import os
>>>
>>> os.getpid()
12788
>>>
>>> from numpy.core import _multiarray_umath as _mu
>>>
>>> _mu
<module 'numpy.core._multiarray_umath' from 'e:\\Work\\Dev\\VEnvs\\py_pc064_03.09_test0\\lib\\site-packages\\numpy\\core\\_multiarray_umath.cp39-win_amd64.pyd'>
>>> ^Z


[prompt]>
[prompt]> :: Backup %PATH%
[prompt]> set _PATH=%PATH%

    To make things as generic as possible, that .pyd depends on a custom .dll (OpenBLAS):

    Img0

    Here's a snapshot of the above (Python) process:

    Img1

    Notice where the dependent .dll was loaded from (2 rows below our (selected) .pyd).

    Now, back to the question: there are a bunch of tools that can do that.
    But it's important to mention that no matter what tool you use, will (most likely) depend on the PATH environment variable contents (in the 1st image, the dependent .dll (and others) was not found). Check [MS.Learn]: Dynamic-Link Library Search Order for more details about .dlls.

    As a note, since (some) tools generate a lot of output, I'll be filtering it out (using commands like FindStr (Grep)), only showing the relevant parts, in order to avoid filling the answer with junk.

    1. [GitHub]: lucasg/Dependencies

    Besides the GUI application that you mentioned (DependenciesGui.exe), there's a command line tool as well next to it: Dependencies.exe:

    [prompt]>
[prompt]> :: Restore %PATH%
[prompt]> set PATH=%_PATH%

[prompt]>
[prompt]> "f:\Install\pc064\LucasG\DependencyWalkerPolitistTexan\Version\Dependencies.exe" -h
Dependencies.exe : command line tool for dumping dependencies and various utilities.

Usage : Dependencies.exe [OPTIONS] <FILE>

Options :
  -h -help : display this help
  -json : activate json output.
  -cache : load and use binary cache in order to prevent dll file locking.
  -depth : limit recursion depth when analysing loaded modules or dependency chain. Default value is infinite.
  -apisets : dump the system's ApiSet schema (api set dll -> host dll)
  -apisetsdll : dump the ApiSet schema from apisetschema <FILE> (api set dll -> host dll)
  -knowndll : dump all the system's known dlls (x86 and x64)
  -manifest : dump <FILE> embedded manifest, if it exists.
  -sxsentries : dump all of <FILE>'s sxs dependencies.
  -imports : dump <FILE> imports
  -exports : dump <FILE> exports
  -modules : dump <FILE> resolved modules
  -chain : dump <FILE> whole dependency chain

[prompt]>
[prompt]> "f:\Install\pc064\LucasG\DependencyWalkerPolitistTexan\Version\Dependencies.exe" -modules "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Lib\site-packages\numpy\core\_multiarray_umath.cp39-win_amd64.pyd" | findstr "libopenblas"
[NOT_FOUND] libopenblas.FB5AE2TYXYH2IJRDKGDGQ3XBKLKTF43H.gfortran-win_amd64.dll :

[prompt]>
[prompt]> set PATH=%_PATH%;e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Lib\site-packages\numpy\.libs

[prompt]>
[prompt]> "f:\Install\pc064\LucasG\DependencyWalkerPolitistTexan\Version\Dependencies.exe" -modules "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Lib\site-packages\numpy\core\_multiarray_umath.cp39-win_amd64.pyd" | findstr "libopenblas"
[Environment] libopenblas.FB5AE2TYXYH2IJRDKGDGQ3XBKLKTF43H.gfortran-win_amd64.dll : e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Lib\site-packages\numpy\.libs\libopenblas.FB5AE2TYXYH2IJRDKGDGQ3XBKLKTF43H.gfortran-win_amd64.dll

    Side note - as seen in [SO]: How to run a fortran skript with ctypes? (@CristiFati's answer), sometimes (for some reason unknown to me) it doesn't show the exports (the GUI, at least).

    2. Dependency Walker

    Although it's no longer maintained, it's a very nice tool and before Dependencies it was the best I could find. I also used it for [SO]: How to build a DLL version of libjpeg 9b? (@CristiFati's answer) (somewhere at the end).
    The drawback is that it's spitting the output in a file, so an additional step is required:

    [prompt]>
[prompt]> :: Restore %PATH%
[prompt]> set PATH=%_PATH%

[prompt]>
[prompt]> dir /b

[prompt]>
[prompt]> :: Help not available in console (/? will open GUI)
[prompt]>
[prompt]> "c:\Install\pc064\Depends\DependencyWalkerPolitistTexan\Version\depends.exe" /c /ot:_mu0.txt "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Lib\site-packages\numpy\core\_multiarray_umath.cp39-win_amd64.pyd"

[prompt]> type _mu0.txt | findstr -i "libopenblas"
     [ ? ] LIBOPENBLAS.FB5AE2TYXYH2IJRDKGDGQ3XBKLKTF43H.GFORTRAN-WIN_AMD64.DLL
[ ? ]  LIBOPENBLAS.FB5AE2TYXYH2IJRDKGDGQ3XBKLKTF43H.GFORTRAN-WIN_AMD64.DLL                                       Error opening file. The system cannot find the file specified (2).

[prompt]>
[prompt]> set PATH=%_PATH%;e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Lib\site-packages\numpy\.libs

[prompt]>
[prompt]> "c:\Install\pc064\Depends\DependencyWalkerPolitistTexan\Version\depends.exe" /c /ot:_mu1.txt "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Lib\site-packages\numpy\core\_multiarray_umath.cp39-win_amd64.pyd"

[prompt]> type _mu1.txt | findstr -i "libopenblas"
     [  6] e:\work\dev\venvs\py_pc064_03.09_test0\lib\site-packages\numpy\.libs\LIBOPENBLAS.FB5AE2TYXYH2IJRDKGDGQ3XBKLKTF43H.GFORTRAN-WIN_AMD64.DLL
[  6]  e:\work\dev\venvs\py_pc064_03.09_test0\lib\site-packages\numpy\.libs\LIBOPENBLAS.FB5AE2TYXYH2IJRDKGDGQ3XBKLKTF43H.GFORTRAN-WIN_AMD64.DLL  2022/11/30 14:57  2022/11/20 00:44  35,695,412  A      0x0220BC27     0x0220BC27     x64  Console    None        0x00000000622C0000  Unknown      0x01E88000    Not Loaded  N/A              N/A              0.0        2.30        4.0     5.2

    3. [MS.Learn]: DUMPBIN Reference

    Part of VStudio. I am only listing it as a reference, because it can display a .dll dependents, but not whether they can be loaded (and if yes, where from):

    [prompt]>
[prompt]> :: Restore %PATH%
[prompt]> set PATH=%_PATH%

[prompt]>
[prompt]> "c:\Install\pc032\Microsoft\VisualStudioCommunity\2019\VC\Auxiliary\Build\vcvarsall.bat" x64 > nul

[prompt]>
[prompt]> dumpbin /DEPENDENTS "e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Lib\site-packages\numpy\core\_multiarray_umath.cp39-win_amd64.pyd"
Microsoft (R) COFF/PE Dumper Version 14.29.30147.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Dump of file e:\Work\Dev\VEnvs\py_pc064_03.09_test0\Lib\site-packages\numpy\core\_multiarray_umath.cp39-win_amd64.pyd

File Type: DLL

  Image has the following dependencies:

    libopenblas.FB5AE2TYXYH2IJRDKGDGQ3XBKLKTF43H.gfortran-win_amd64.dll
    python39.dll
    KERNEL32.dll
    VCRUNTIME140.dll
    api-ms-win-crt-math-l1-1-0.dll
    api-ms-win-crt-heap-l1-1-0.dll
    api-ms-win-crt-stdio-l1-1-0.dll
    api-ms-win-crt-string-l1-1-0.dll
    api-ms-win-crt-environment-l1-1-0.dll
    api-ms-win-crt-runtime-l1-1-0.dll
    api-ms-win-crt-convert-l1-1-0.dll
    api-ms-win-crt-time-l1-1-0.dll
    api-ms-win-crt-utility-l1-1-0.dll
    api-ms-win-crt-locale-l1-1-0.dll

  Summary

       40000 .data
       18000 .pdata
       64000 .rdata
        3000 .reloc
        1000 .rsrc
      1F3000 .text

    4. Nix emulators

    Invoke [Man7]: LDD(1).
    I guess this can be a favorite, since it's leaning towards Nix world (where these kind of things are easier) and you also mentioned SSH connection.
    I'll be exemplifying on MSYS2, but same thing is achievable from others (Cygwin, maybe MinGW, ...).

    [cfati@cfati-5510-0:/e/Work/Dev/StackOverflow/q074877580]> ~/sopr.sh
### Set shorter prompt to better fit when pasted in StackOverflow (or other) pages ###

[064bit prompt]>
[064bit prompt]> uname -a
MSYS_NT-10.0-19045 cfati-5510-0 3.4.3-dirty.x86_64 2022-12-19 20:20 UTC x86_64 Msys
[064bit prompt]>
[064bit prompt]> _PATH="${PATH}"
[064bit prompt]>
[064bit prompt]> ls "/e/Work/Dev/VEnvs/py_pc064_03.09_test0/Lib/site-packages/numpy/core/_multiarray_umath.cp39-win_amd64.pyd"
/e/Work/Dev/VEnvs/py_pc064_03.09_test0/Lib/site-packages/numpy/core/_multiarray_umath.cp39-win_amd64.pyd
[064bit prompt]>
[064bit prompt]> ldd "/e/Work/Dev/VEnvs/py_pc064_03.09_test0/Lib/site-packages/numpy/core/_multiarray_umath.cp39-win_amd64.pyd"
ldd: /e/Work/Dev/VEnvs/py_pc064_03.09_test0/Lib/site-packages/numpy/core/_multiarray_umath.cp39-win_amd64.pyd: Bad file descriptor
[064bit prompt]>
[064bit prompt]> # Change extension
[064bit prompt]> cp "/e/Work/Dev/VEnvs/py_pc064_03.09_test0/Lib/site-packages/numpy/core/_multiarray_umath.cp39-win_amd64.pyd" ./_mu.dll
[064bit prompt]> ls
_mu.dll  _mu0.txt  _mu1.txt
[064bit prompt]> file _mu.dll
_mu.dll: PE32+ executable (DLL) (GUI) x86-64, for MS Windows
[064bit prompt]>
[064bit prompt]> ldd _mu.dll
        ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll (0x7ff8ba930000)
        KERNEL32.DLL => /c/WINDOWS/System32/KERNEL32.DLL (0x7ff8ba320000)
        KERNELBASE.dll => /c/WINDOWS/System32/KERNELBASE.dll (0x7ff8b8070000)
        msvcrt.dll => /c/WINDOWS/System32/msvcrt.dll (0x7ff8b8f40000)
        _mu.dll => /e/Work/Dev/StackOverflow/q074877580/_mu.dll (0x7ff86ab50000)
        ucrtbase.dll => /c/Windows/System32/ucrtbase.dll (0x7ff8b8840000)
        vcruntime140.dll => /c/Windows/System32/vcruntime140.dll (0x7ff8a0980000)
        libopenblas.FB5AE2TYXYH2IJRDKGDGQ3XBKLKTF43H.gfortran-win_amd64.dll => not found
        python39.dll => not found
        api-ms-win-crt-math-l1-1-0.dll => not found
        api-ms-win-crt-heap-l1-1-0.dll => not found
        api-ms-win-crt-stdio-l1-1-0.dll => not found
        api-ms-win-crt-string-l1-1-0.dll => not found
        api-ms-win-crt-environment-l1-1-0.dll => not found
        api-ms-win-crt-runtime-l1-1-0.dll => not found
        api-ms-win-crt-convert-l1-1-0.dll => not found
        api-ms-win-crt-time-l1-1-0.dll => not found
        api-ms-win-crt-utility-l1-1-0.dll => not found
        api-ms-win-crt-locale-l1-1-0.dll => not found
[064bit prompt]>
[064bit prompt]> PATH="${_PATH}:/e/Work/Dev/VEnvs/py_pc064_03.09_test0/Lib/site-packages/numpy/.libs"
[064bit prompt]>
[064bit prompt]> ldd _mu.dll
        ntdll.dll => /c/WINDOWS/SYSTEM32/ntdll.dll (0x7ff8ba930000)
        KERNEL32.DLL => /c/WINDOWS/System32/KERNEL32.DLL (0x7ff8ba320000)
        KERNELBASE.dll => /c/WINDOWS/System32/KERNELBASE.dll (0x7ff8b8070000)
        msvcrt.dll => /c/WINDOWS/System32/msvcrt.dll (0x7ff8b8f40000)
        _mu.dll => /e/Work/Dev/StackOverflow/q074877580/_mu.dll (0x7ff86ab50000)
        ucrtbase.dll => /c/Windows/System32/ucrtbase.dll (0x7ff8b8840000)
        vcruntime140.dll => /c/Windows/System32/vcruntime140.dll (0x7ff8a0980000)
        libopenblas.FB5AE2TYXYH2IJRDKGDGQ3XBKLKTF43H.gfortran-win_amd64.dll => /e/Work/Dev/VEnvs/py_pc064_03.09_test0/Lib/site-packages/numpy/.libs/libopenblas.FB5AE2TYXYH2IJRDKGDGQ3XBKLKTF43H.gfortran-win_amd64.dll (0x622c0000)
        user32.dll => /c/Windows/System32/user32.dll (0x7ff8b92f0000)
        win32u.dll => /c/Windows/System32/win32u.dll (0x7ff8b8940000)
        gdi32.dll => /c/Windows/System32/gdi32.dll (0x7ff8b94a0000)
        gdi32full.dll => /c/Windows/System32/gdi32full.dll (0x7ff8b8400000)
        msvcp_win.dll => /c/Windows/System32/msvcp_win.dll (0x7ff8b8610000)
        python39.dll => not found
        api-ms-win-crt-math-l1-1-0.dll => not found
        api-ms-win-crt-heap-l1-1-0.dll => not found
        api-ms-win-crt-stdio-l1-1-0.dll => not found
        api-ms-win-crt-string-l1-1-0.dll => not found
        api-ms-win-crt-environment-l1-1-0.dll => not found
        api-ms-win-crt-runtime-l1-1-0.dll => not found
        api-ms-win-crt-convert-l1-1-0.dll => not found
        api-ms-win-crt-time-l1-1-0.dll => not found
        api-ms-win-crt-utility-l1-1-0.dll => not found
        api-ms-win-crt-locale-l1-1-0.dll => not found

    Of course, there can be more tools that I am not aware of (or if .dlls coming from .NET are involved).

    Related (more or less):