artificial-intelligencehuggingface-transformerslarge-language-model

Error during the compilation of the tokenizers package when trying to install transformers 4.27


pip install -v transformers==4.27.1

....

error: `cargo rustc --lib --message-format=json-render-diagnostics --manifest-path Cargo.toml --release -v --features pyo3/extension-module --crate-type cdylib -- -C 'link-args=-undefined dynamic_lookup -Wl,-install_name,@rpath/tokenizers.cpython-312-darwin.so'` failed with code 101
  error: subprocess-exited-with-error
  
  × Building wheel for tokenizers (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  full command: /Users/dragonfang/****/venv_agi/bin/python3.12 /Users/dragonfang/****/venv_agi/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py build_wheel /var/folders/ll/9dtz3vg150vfv8t75ppq_nr00000gn/T/tmpymy6ke0d
  cwd: /private/var/folders/ll/9dtz3vg150vfv8t75ppq_nr00000gn/T/pip-install-tz2dgt67/tokenizers_11ac58d2069c4ec1985eae0d4528f0ec
  Building wheel for tokenizers (pyproject.toml) ... error
  ERROR: Failed building wheel for tokenizers
Failed to build tokenizers
ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (tokenizers)

How to resolve this issue?


Supplementary:

I have tried some methods from the two links above, but none of them have solved my problem.


Solution

  • ChatGPT suggested

    You can try using Python 3.10 or 3.11 to see if the issue is resolved

    Since my Python version was 3.12.5, I downgraded to 3.11.9 and re-ran pip install. This successfully resolved the problem.

    I've noticed that Gemini 1.5 Flash only suggests me to update Rust and Cargo, while GPT-4o mini additionally mentions the issue of Python version. I have been using Gemini before, it seems I should compare these two models more in the future.