When importing get_tokenizer from torchtext:
from torchtext.data.utils import get_tokenizer
it throws this error:
Traceback (most recent call last):
File "D:\PythonProjects\FakeNews\train.py", line 4, in <module>
from torchtext.data.utils import get_tokenizer
File "D:\PythonProjects\FakeNews\venv\Lib\site-packages\torchtext\__init__.py", line 18, in <module>
from torchtext import _extension # noqa: F401
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "D:\PythonProjects\FakeNews\venv\Lib\site-packages\torchtext\_extension.py", line 64, in <module>
_init_extension()
File "D:\PythonProjects\FakeNews\venv\Lib\site-packages\torchtext\_extension.py", line 58, in _init_extension
_load_lib("libtorchtext")
File "D:\PythonProjects\FakeNews\venv\Lib\site-packages\torchtext\_extension.py", line 50, in _load_lib
torch.ops.load_library(path)
File "D:\PythonProjects\FakeNews\venv\Lib\site-packages\torch\_ops.py", line 1295, in load_library
ctypes.CDLL(path)
File "C:\Users\Owner\AppData\Local\Programs\Python\Python312\Lib\ctypes\__init__.py", line 379, in __init__
self._handle = _dlopen(self._name, mode)
^^^^^^^^^^^^^^^^^^^^^^^^^
OSError: [WinError 127] The specified procedure could not be found
I'm using torch 2.4.0 and torchtext 0.18.0, which should be compatible.
Reinstalling and --upgrade did not work. Rerunning just throws the same error. No forums I've found say anything helpful either.
Turns out I was using PyTorch for CUDA 12.4 but torchtext only supports CUDA 12.1, so I had to change the compute platform.
TLDR; torchtext 0.18.0 and pytorch 2.4.0 work together on CUDA 12.1 but not 12.4