torch_ext.so cannot be found error on Macbook macOS #229
Comments
|
try to run the same code from command line, errors become different I also tried to add the path of ninja and python in my conda environment: |
Closed
|
after i execute ❯ pip install torchtext
Collecting torchtext
Downloading torchtext-0.16.2-cp39-cp39-macosx_11_0_arm64.whl.metadata (7.5 kB)
Requirement already satisfied: tqdm in /Users/cdove/mambaforge/envs/torchdrug/lib/python3.9/site-packages (from torchtext) (4.66.1)
Collecting requests (from torchtext)
Downloading requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
Collecting torch==2.1.2 (from torchtext)
Downloading torch-2.1.2-cp39-none-macosx_11_0_arm64.whl.metadata (25 kB)
Requirement already satisfied: numpy in /Users/cdove/mambaforge/envs/torchdrug/lib/python3.9/site-packages (from torchtext) (1.26.3)
Collecting torchdata==0.7.1 (from torchtext)
Downloading torchdata-0.7.1-cp39-cp39-macosx_11_0_arm64.whl.metadata (13 kB)
Collecting filelock (from torch==2.1.2->torchtext)
Downloading filelock-3.13.1-py3-none-any.whl.metadata (2.8 kB)
Requirement already satisfied: typing-extensions in /Users/cdove/mambaforge/envs/torchdrug/lib/python3.9/site-packages (from torch==2.1.2->torchtext) (4.7.1)
Collecting sympy (from torch==2.1.2->torchtext)
Downloading sympy-1.12-py3-none-any.whl (5.7 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 5.7/5.7 MB 8.6 MB/s eta 0:00:00
Requirement already satisfied: networkx in /Users/cdove/mambaforge/envs/torchdrug/lib/python3.9/site-packages (from torch==2.1.2->torchtext) (3.2.1)
Requirement already satisfied: jinja2 in /Users/cdove/mambaforge/envs/torchdrug/lib/python3.9/site-packages (from torch==2.1.2->torchtext) (3.1.2)
Collecting fsspec (from torch==2.1.2->torchtext)
Downloading fsspec-2023.12.2-py3-none-any.whl.metadata (6.8 kB)
Collecting urllib3>=1.25 (from torchdata==0.7.1->torchtext)
Downloading urllib3-2.1.0-py3-none-any.whl.metadata (6.4 kB)
Collecting charset-normalizer<4,>=2 (from requests->torchtext)
Downloading charset_normalizer-3.3.2-cp39-cp39-macosx_11_0_arm64.whl.metadata (33 kB)
Collecting idna<4,>=2.5 (from requests->torchtext)
Downloading idna-3.6-py3-none-any.whl.metadata (9.9 kB)
Collecting certifi>=2017.4.17 (from requests->torchtext)
Downloading certifi-2023.11.17-py3-none-any.whl.metadata (2.2 kB)
Requirement already satisfied: MarkupSafe>=2.0 in /Users/cdove/mambaforge/envs/torchdrug/lib/python3.9/site-packages (from jinja2->torch==2.1.2->torchtext) (2.1.3)
Collecting mpmath>=0.19 (from sympy->torch==2.1.2->torchtext)
Downloading mpmath-1.3.0-py3-none-any.whl (536 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 536.2/536.2 kB 45.2 MB/s eta 0:00:00
Downloading torchtext-0.16.2-cp39-cp39-macosx_11_0_arm64.whl (2.1 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.1/2.1 MB 16.8 MB/s eta 0:00:00
Downloading torch-2.1.2-cp39-none-macosx_11_0_arm64.whl (59.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 59.6/59.6 MB 15.8 MB/s eta 0:00:00
Downloading torchdata-0.7.1-cp39-cp39-macosx_11_0_arm64.whl (4.8 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.8/4.8 MB 23.7 MB/s eta 0:00:00
Downloading requests-2.31.0-py3-none-any.whl (62 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 62.6/62.6 kB 2.5 MB/s eta 0:00:00
Downloading certifi-2023.11.17-py3-none-any.whl (162 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 162.5/162.5 kB 14.5 MB/s eta 0:00:00
Downloading charset_normalizer-3.3.2-cp39-cp39-macosx_11_0_arm64.whl (120 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 120.4/120.4 kB 9.2 MB/s eta 0:00:00
Downloading idna-3.6-py3-none-any.whl (61 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 61.6/61.6 kB 3.5 MB/s eta 0:00:00
Downloading urllib3-2.1.0-py3-none-any.whl (104 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 104.6/104.6 kB 8.7 MB/s eta 0:00:00
Downloading filelock-3.13.1-py3-none-any.whl (11 kB)
Downloading fsspec-2023.12.2-py3-none-any.whl (168 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 169.0/169.0 kB 16.9 MB/s eta 0:00:00
Installing collected packages: mpmath, urllib3, sympy, idna, fsspec, filelock, charset-normalizer, certifi, torch, requests, torchdata, torchtext
Attempting uninstall: torch
Found existing installation: torch 1.13.0
Uninstalling torch-1.13.0:
Successfully uninstalled torch-1.13.0
Successfully installed certifi-2023.11.17 charset-normalizer-3.3.2 filelock-3.13.1 fsspec-2023.12.2 idna-3.6 mpmath-1.3.0 requests-2.31.0 sympy-1.12 torch-2.1.2 torchdata-0.7.1 torchtext-0.16.2 urllib3-2.1.0
(torchdrug)
░▒▓ …/Development/test/torchdrug 20:18
❯ pip install torch==1.13.0
Collecting torch==1.13.0
Using cached torch-1.13.0-cp39-none-macosx_11_0_arm64.whl (55.7 MB)
Requirement already satisfied: typing-extensions in /Users/cdove/mambaforge/envs/torchdrug/lib/python3.9/site-packages (from torch==1.13.0) (4.7.1)
Installing collected packages: torch
Attempting uninstall: torch
Found existing installation: torch 2.1.2
Uninstalling torch-2.1.2:
Successfully uninstalled torch-2.1.2
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
torchdata 0.7.1 requires torch>=2, but you have torch 1.13.0 which is incompatible.
torchtext 0.16.2 requires torch==2.1.2, but you have torch 1.13.0 which is incompatible.
Successfully installed torch-1.13.0 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
when I run:
solver.train(num_epoch=10)in following code:optimizer = torch.optim.Adam(task.parameters(), lr=1e-3) solver = core.Engine(task, train_set, valid_set, test_set, optimizer, gpus=None, batch_size=1024) # solver.train(num_epoch=10) solver.evaluate("valid")(I am using cpu instead of gpus, since there is no cuba support on macbook, wondering if torchdrug support mps or not)
I get this error at first run, seems related to ninja:
File "/Users/Guo/.conda/envs/torchdrug/lib/python3.10/site-packages/torch/utils/cpp_extension.py", line 1916, in _run_ninja_build raise RuntimeError(message) from e RuntimeError: Error building extension 'torch_ext': [1/2] c++ -MMD -MF torch_ext.o.d -DTORCH_EXTENSION_NAME=torch_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_clang\" -DPYBIND11_STDLIB=\"_libcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1002\" -isystem /Users/Guo/.conda/envs/torchdrug/lib/python3.10/site-packages/torch/include -isystem /Users/Guo/.conda/envs/torchdrug/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /Users/Guo/.conda/envs/torchdrug/lib/python3.10/site-packages/torch/include/TH -isystem /Users/Guo/.conda/envs/torchdrug/lib/python3.10/site-packages/torch/include/THC -isystem /Users/Guo/.conda/envs/torchdrug/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -Ofast -fopenmp -DAT_PARALLEL_OPENMP -c /Users/Guo/.conda/envs/torchdrug/lib/python3.10/site-packages/torchdrug/utils/extension/torch_ext.cpp -o torch_ext.o FAILED: torch_ext.o c++ -MMD -MF torch_ext.o.d -DTORCH_EXTENSION_NAME=torch_ext -DTORCH_API_INCLUDE_EXTENSION_H -DPYBIND11_COMPILER_TYPE=\"_clang\" -DPYBIND11_STDLIB=\"_libcpp\" -DPYBIND11_BUILD_ABI=\"_cxxabi1002\" -isystem /Users/Guo/.conda/envs/torchdrug/lib/python3.10/site-packages/torch/include -isystem /Users/Guo/.conda/envs/torchdrug/lib/python3.10/site-packages/torch/include/torch/csrc/api/include -isystem /Users/Guo/.conda/envs/torchdrug/lib/python3.10/site-packages/torch/include/TH -isystem /Users/Guo/.conda/envs/torchdrug/lib/python3.10/site-packages/torch/include/THC -isystem /Users/Guo/.conda/envs/torchdrug/include/python3.10 -D_GLIBCXX_USE_CXX11_ABI=0 -fPIC -std=c++14 -Ofast -fopenmp -DAT_PARALLEL_OPENMP -c /Users/Guo/.conda/envs/torchdrug/lib/python3.10/site-packages/torchdrug/utils/extension/torch_ext.cpp -o torch_ext.o clang: error: unsupported option '-fopenmp' ninja: build stopped: subcommand failed.if I run:
solver.train(num_epoch=10)again the error change to:File "<frozen importlib._bootstrap>", line 571, in module_from_spec File "<frozen importlib._bootstrap_external>", line 1176, in create_module File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed ImportError: dlopen(/Users/Guo/Library/Caches/torch_extensions/py310_cpu/torch_ext_0/torch_ext.so, 0x0002): tried: '/Users/Guo/Library/Caches/torch_extensions/py310_cpu/torch_ext_0/torch_ext.so' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Users/Guo/Library/Caches/torch_extensions/py310_cpu/torch_ext_0/torch_ext.so' (no such file), '/Users/Guo/Library/Caches/torch_extensions/py310_cpu/torch_ext_0/torch_ext.so' (no such file)which seems related to "torch_extensions" "cpu" and "torch_ext.so"
tried the solution in issue #8 :
rm -rf /home/your_user_name/.cache/torch_extensionsin fact I cannot find torch_extensions under this foldertried to add the path of ninja in code:
ninja_dir = "/Users/Guo/.conda/envs/torchdrug/bin/ninja" if ninja_dir not in os.environ['PATH']: os.environ['PATH'] = f"{ninja_dir}:{os.environ['PATH']}"but does not solve the ninja issue, not sure if two problems are related or not
other info:
python version: 3.10.8
torchdrug installation:
under a conda environment
pip install ninja pip install torch==1.13.0 pip install git+https://github.com/rusty1s/pytorch_scatter.git pip install git+https://github.com/rusty1s/pytorch_cluster.git pip install torchdrugThe text was updated successfully, but these errors were encountered: