continuous-integrationjuliagithub-actionspycall

Github Actions CI: making PyCall aware of Julia ENV variables

I want to setup a Github action ci.yml file that installs Python and Julia and then builds and tests a package using PyCall. PyCall in this case should make use of the previously installed Python version and not it's default julia-specific Python version it comes with it's miniconda.

My stripped down ci.yml looks like the following so far:

name: CI
on:
  - push
  - pull_request

test:
    name: Julia ${{ matrix.version }} - ${{ matrix.os }} - ${{ matrix.arch }} - ${{ github.event_name }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        version:
          - '1.5'
          - 'nightly'
        python: [3.7]
        # python-path: [RUNNER_TOOL_CACHE/Python/*]
        os:
          - macOS-latest
         
        arch:
          - x64
steps:
      - uses: actions/checkout@v2
      - name: Set up Python ${{ matrix.python }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python }}
      - name: Display Python version
        run: python -c "import sys; print(sys.version)"

- name: Install dependencies šŸ”§
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

# Julia Tasks
- uses: julia-actions/setup-julia@v1
- name: Setup Julia
  with:
    version: ${{ matrix.version }}
    arch: ${{ matrix.arch }}

- name: Set ENV variables for šŸ šŸ“ž
        run: |
          echo ENV["PYTHON"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV
          echo ENV["PYCALL_JL_RUNTIME_PYTHONHOME"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV
          echo ENV["PYCALL_JL_RUNTIME"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV

- name: Build Julia pkg
        run: |
          echo $ENV["PYTHON"]
          echo $ENV["PYCALL_JL_RUNTIME_PYTHONHOME"]
          echo $ENV["PYCALL_JL_RUNTIME"]
      - uses: julia-actions/julia-buildpkg@v1
              
      - uses: julia-actions/julia-runtest@v1
      - name: Test Julia

It runs fine until the julia-actions/julia-runtest@v1 and the reason is clear so far, because PyCall uses it's own miniconda environment and Python installation and the python-package I try to load in my pkg.jl with PyCall, is installed in the system Python I run the CI with.

I do not want to use the internal PyCall -Julia specific- Python version, because I want to be able to test different Python versions via Matrix call.

I am able to pass the environment variables via Github Actions suggested way like:

- name: Set ENV Variables for šŸ šŸ“ž
        run: |
          echo ENV["PYTHON"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV
          echo ENV["PYCALL_JL_RUNTIME_PYTHONHOME"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV
          echo ENV["PYCALL_JL_RUNTIME"] = "${{ env.pythonLocation }}" >> $GITHUB_ENV

which outputs:

Run echo $ENV["PYTHON"]
  echo $ENV["PYTHON"]
  echo $ENV["PYCALL_JL_RUNTIME_PYTHONHOME"]
  echo $ENV["PYCALL_JL_RUNTIME"]
  shell: /bin/bash -e {0}
  env:
    pythonLocation: /Users/runner/hostedtoolcache/Python/3.7.9/x64
    ENV[PYTHON] :  /Users/runner/hostedtoolcache/Python/3.7.9/x64
    ENV[PYCALL_JL_RUNTIME_PYTHONHOME] :  /Users/runner/hostedtoolcache/Python/3.7.9/x64
    ENV[PYCALL_JL_RUNTIME] :  /Users/runner/hostedtoolcache/Python/3.7.9/x64
[PYTHON]
[PYCALL_JL_RUNTIME_PYTHONHOME]
[PYCALL_JL_RUNTIME]

This is suggested by the PyCall Documentation

However, PyCall still uses:

ā”Œ Info: Using the Python distribution in the Conda package by default.
ā”” To use a different Python version, set ENV["PYTHON"]="pythoncommand" and re-run Pkg.build("PyCall").
[ Info: Running `conda install -q -y numpy` in root environment
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

# All requested packages already installed.

[ Info: PyCall is using /Users/runner/.julia/conda/3/bin/python (Python 3.8.5) at /Users/runner/.julia/conda/3/bin/python, libpython = /Users/runner/.julia/conda/3/lib/libpython3.8.dylib
[ Info: /Users/runner/.julia/packages/PyCall/BcTLp/deps/deps.jl has not changed
[ Info: /Users/runner/.julia/prefs/PyCall has not changed

Can I pass a startup.jl file in the julia-actions/setup-julia@v1?

How can I make PyCall be aware of these ENV variables?

Edit: Added env to julia-actions/julia-buildpkg@v1 and rebuilt PyCall

It gives me a permission error:

- uses: julia-actions/julia-buildpkg@v1
        env:
          PYTHON : ${{ env.pythonLocation }}
          PYCALL_JL_RUNTIME_PYTHONHOME: ${{ env.pythonLocation }}
          PYCALL_JL_RUNTIME: ${{ env.pythonLocation }}
      - run: |
          julia 
          using Pkg
          Pkg.build("PyCall")

     Building PyCall ā†’ `~/.julia/scratchspaces/44cfe95a-1eb2-52ea-b672-e2afdf69b78f/b6dff5fa725eff4f775f472acd86756d6e31fb02/build.log`
ā”Œ Info: No system-wide Python was found; got the following error:
ā”‚ Base.IOError("could not spawn setenv(`/Users/runner/hostedtoolcache/Python/3.7.9/x64 -c \"import distutils.sysconfig; print(distutils.sysconfig.get_config_var('VERSION'))\"`,[\"PATH=/Users/runner/hostedtoolcache/julia/nightly/x64/bin:/Users/runner/hostedtoolcache/Python/3.7.9/x64/bin:/Users/runner/hostedtoolcache/Python/3.7.9/x64:/usr/local/opt/pipx_bin:/Users/runner/.cargo/bin:/usr/local/lib/ruby/gems/2.7.0/bin:/usr/local/opt/ruby/bin:/usr/local/opt/curl/bin:/usr/local/bin:/usr/local/sbin:/Users/runner/bin:/Users/runner/.yarn/bin:/usr/local/go/bin:/Users/runner/Library/Android/sdk/tools:/Users/runner/Library/Android/sdk/platform-tools:/Users/runner/Library/Android/sdk/ndk-bundle:/Library/Frameworks/Mono.framework/Versions/Current/Commands:/usr/bin:/bin:/usr/sbin:/sbin:/Users/runner/.dotnet/tools:/Users/runner/.ghcup/bin:/Users/runner/hostedtoolcache/stack/2.5.1/x64\", \"GITHUB_RUN_NUMBER=90\", \"PERFLOG_LOCATION_SETTING=RUNNER_PERFLOG\", \"NUNIT3_PATH=/Library/Developer/nunit/3.6.0\", \"NVM_CD_FLAGS=\", \"GITHUB_ACTOR=mkschulze\", \"ANDROID_NDK_18R_PATH=/Users/runner/Library/Android/sdk/ndk/18.1.5063045\", \"ENV[PYCALL_JL_RUNTIME_PYTHONHOME] = /Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"USER=runner\", \"PYCALL_JL_RUNTIME_PYTHONHOME=/Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"CONDA=/usr/local/miniconda\", \"VCPKG_INSTALLATION_ROOT=/usr/local/share/vcpkg\", \"GITHUB_SHA=a08b0b9021e3138eb34ff298f2ccc0498713e627\", \"JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home\", \"GITHUB_API_URL=https://api.github.com\", \"GITHUB_ACTIONS=true\", \"ANDROID_SDK_ROOT=/Users/runner/Library/Android/sdk\", \"GITHUB_ENV=/Users/runner/work/_temp/_runner_file_commands/set_env_e539860f-4c69-4d29-9773-3ad63e77ebdf\", \"SHELL=/bin/bash\", \"GITHUB_ACTION_PATH=/Users/runner/work/_actions/julia-actions/julia-buildpkg/v1\", \"RUNNER_PERFLOG=/usr/local/opt/runner/perflog\", \"JAVA_HOME_12_X64=/Library/Java/JavaVirtualMachines/adoptopenjdk-12.jdk/Contents/Home\", \"EDGEWEBDRIVER=/usr/local/share/edge_driver\", \"TMPDIR=/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/\", \"GITHUB_RUN_ID=390315562\", \"ImageOS=macos1015\", \"NVM_DIR=/Users/runner/.nvm\", \"GITHUB_HEAD_REF=\", \"GITHUB_RETENTION_DAYS=90\", \"GITHUB_SERVER_URL=https://github.com\", \"GITHUB_JOB=test\", \"RUNNER_TOOL_CACHE=/Users/runner/hostedtoolcache\", \"RUNNER_TRACKING_ID=github_bab8cac0-d4b7-441b-9529-36f2faf19750\", \"HOMEBREW_NO_AUTO_UPDATE=1\", \"CHROMEWEBDRIVER=/usr/local/Caskroom/chromedriver/86.0.4240.22\", \"GITHUB_WORKFLOW=CI\", \"GITHUB_ACTION_REPOSITORY=julia-actions/julia-buildpkg\", \"PYCALL_JL_RUNTIME=/Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"pythonLocation=/Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"HOME=/Users/runner\", \"GITHUB_ACTION=julia-actionsjulia-buildpkg\", \"ENV[PYTHON] = /Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"PYTHONIOENCODING=UTF-8\", \"JAVA_HOME_8_X64=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home\", \"GITHUB_EVENT_PATH=/Users/runner/work/_temp/_github_workflow/event.json\", \"XPC_FLAGS=0x0\", \"PYTHON=/Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"XCODE_10_DEVELOPER_DIR=/Applications/Xcode_10.3.app/Contents/Developer\", \"XPC_SERVICE_NAME=0\", \"XCODE_12_DEVELOPER_DIR=/Applications/Xcode_12.2.app/Contents/Developer\", \"GITHUB_REF=refs/heads/py-cli\", \"XCODE_11_DEVELOPER_DIR=/Applications/Xcode_11.7.app/Contents/Developer\", \"__CF_USER_TEXT_ENCODING=0x1F5:0:0\", \"LC_ALL=en_US.UTF-8\", \"GITHUB_REPOSITORY=Humans-of-Julia/GraknClient.jl\", \"LC_CTYPE=en_US.UTF-8\", \"ImageVersion=20201115.1\", \"LANG=en_US.UTF-8\", \"GITHUB_GRAPHQL_URL=https://api.github.com/graphql\", \"ENV[PYCALL_JL_RUNTIME] = /Users/runner/hostedtoolcache/Python/3.7.9/x64\", \"SHLVL=2\", \"LOGNAME=runner\", \"RUNNER_WORKSPACE=/Users/runner/work/GraknClient.jl\", \"SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.yxv10u7SRe/Listeners\", \"DOTNET_MULTILEVEL_LOOKUP=0\", \"HOMEBREW_CASK_OPTS=--no-quarantine\", \"GITHUB_BASE_REF=\", \"_=/Users/runner/hostedtoolcache/julia/nightly/x64/bin/julia\", \"GITHUB_ACTION_REF=v1\", \"DOTNET_ROOT=/Users/runner/.dotnet\", \"PIPX_HOME=/usr/local/opt/pipx\", \"***", \"JAVA_HOME_13_X64=/Library/Java/JavaVirtualMachines/adoptopenjdk-13.jdk/Contents/Home\", \"GITHUB_WORKSPACE=/Users/runner/work/GraknClient.jl/GraknClient.jl\", \"NUNIT_BASE_PATH=/Library/Developer/nunit\", \"ANDROID_HOME=/Users/runner/Library/Android/sdk\", \"CI=true\", \"POWERSHELL_DISTRIBUTION_CHANNEL=GitHub-Actions-macos1015\", \"GECKOWEBDRIVER=/usr/local/opt/geckodriver/bin\", \"RCT_NO_LAUNCH_PACKAGER=1\", \"GITHUB_PATH=/Users/runner/work/_temp/_runner_file_commands/add_path_e539860f-4c69-4d29-9773-3ad63e77ebdf\", \"RUNNER_OS=macOS\", \"JAVA_HOME_7_X64=/Library/Java/JavaVirtualMachines/zulu-7.jdk/Contents/Home\", \"JULIA_LOAD_PATH=@:/var/folders/24/8k48jl6d249_n_qfxwsl6xvm0000gn/T/jl_XGe9fC\", \"PIPX_BIN_DIR=/usr/local/opt/pipx_bin\", \"BOOTSTRAP_HASKELL_NONINTERACTIVE=1\", \"JAVA_HOME_14_X64=/Library/Java/JavaVirtualMachines/adoptopenjdk-14.jdk/Contents/Home\", \"JAVA_HOME_11_X64=/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home\", \"RUNNER_TEMP=/Users/runner/work/_temp\", \"GITHUB_REPOSITORY_OWNER=Humans-of-Julia\", \"GITHUB_EVENT_NAME=push\", \"OPENBLAS_MAIN_FREE=1\", \"ANDROID_NDK_HOME=/Users/runner/Library/Android/sdk/ndk-bundle\", \"AGENT_TOOLSDIRECTORY=/Users/runner/hostedtoolcache\"])
: permission denied (EACCES)", -13)
ā”” using the Python distribution in the Conda package
[ Info: Running `conda install -q -y numpy` in root environment
Collecting package metadata (current_repodata.json): ...working... done
Solving environment: ...working... done

# All requested packages already installed.
Solution

  • Ok, got it running now. Setting this PYTHON ENV to the exact Python binary and at least on macOS and Ubuntu x64 it works.

    macOS / linux solution:

    - uses: julia-actions/julia-buildpkg@v1
        env:
          PYTHON : "${{ env.pythonLocation }}/bin/python"

    Windows solution:

    - uses: julia-actions/julia-buildpkg@v1
        env:
          PYTHON : "${{ env.pythonLocation }}/python.exe"

    Solution for all:

    - uses: julia-actions/julia-buildpkg@v1
        env:
          PYTHON : python

    here is a full working ci.yml that passes on macOS and Ubuntu x64:

    name: CI
on:
  - push
  - pull_request

jobs:
  test:
    name: Julia ${{ matrix.version }} - ${{ matrix.os }} - ${{ matrix.arch }} - ${{ github.event_name }}
    runs-on: ${{ matrix.os }}
    strategy:
      fail-fast: false
      matrix:
        version:
          - '1.5'
          - 'nightly'
        python: [3.7]
        os:
          - ubuntu-latest
          - macOS-latest
        arch:
          - x64
    steps:
      - uses: actions/checkout@v2
      - name: Set up šŸ ${{ matrix.python }}
        uses: actions/setup-python@v2
        with:
          python-version: ${{ matrix.python }}

      - name: Display Python version
        run: python -c "import sys; print(sys.version)"

      - name: Set ENV Variables for šŸ šŸ“ž
        run: echo ENV["PYTHON"] = "${{ env.pythonLocation }}/bin/python" >> $GITHUB_ENV

      - name: Install dependencies šŸ”§
        run: |
          python -m pip install --upgrade pip
          pip install -r requirements.txt

      # Julia Tasks
      - uses: julia-actions/setup-julia@v1
        with:
          version: ${{ matrix.version }}
          arch: ${{ matrix.arch }}
        
      - uses: actions/cache@v2
        env:
          cache-name: cache-artifacts
        with:
          path: ~/.julia/artifacts
          key: ${{ runner.os }}-test-${{ env.cache-name }}-${{ hashFiles('**/Project.toml') }}
          restore-keys: |
            ${{ runner.os }}-test-${{ env.cache-name }}-
            ${{ runner.os }}-test-
            ${{ runner.os }}-

      
      - uses: julia-actions/julia-buildpkg@v1
        env:
          PYTHON : "${{ env.pythonLocation }}/bin/python"
            
      - uses: julia-actions/julia-runtest@v1
      - uses: julia-actions/julia-processcoverage@v1
      - uses: codecov/codecov-action@v1
        with:
          file: lcov.info