Sklearn is unstable on Apple Silicon

Hi,

I installed skearn successfully and ran the MINIST toy example successfully.

then I started to run my project. The finning thing everything seems good at the start point (at least no ImportError occurs). but when I made some changes of my code and try to run all cells (I use jupyter lab) again, ImportError occurs.....

ImportError: dlopen(/Users/a/miniforge3/lib/python3.9/site-packages/scipy/spatial/qhull.cpython-39-darwin.so, 0x0002): Library not loaded: @rpath/liblapack.3.dylib

  Referenced from: /Users/a/miniforge3/lib/python3.9/site-packages/scipy/spatial/qhull.cpython-39-darwin.so

  Reason: tried: '/Users/a/miniforge3/lib/liblapack.3.dylib' (no such file), '/Users/a/miniforge3/lib/liblapack.3.dylib' (no such file), '/Users/a/miniforge3/lib/python3.9/site-packages/scipy/spatial/../../../../liblapack.3.dylib' (no such file), '/Users/a/miniforge3/lib/liblapack.3.dylib' (no such file), '/Users/a/miniforge3/lib/liblapack.3.dylib' (no such file), '/Users/a/miniforge3/lib/python3.9/site-packages/scipy/spatial/../../../../liblapack.3.dylib' (no such file), '/Users/a/miniforge3/lib/liblapack.3.dylib' (no such file), '/Users/a/miniforge3/bin/../lib/liblapack.3.dylib' (no such file), '/Users/a/miniforge3/lib/liblapack.3.dylib' (no such file), '/Users/a/miniforge3/bin/../lib/liblapack.3.dylib' (no such file), '/usr/local/lib/liblapack.3.dylib' (no such file), '/usr/lib/liblapack.3.dylib' (no such file)

then I have to uninstall scipy, sklearn, etc and reinstall all of them. and my code can be ran again.....

Magically I hate to say, anyone knows how to permanently solve this problem? make skearn more stable?

Post not yet marked as solved Up vote post of dkjdjdfdskln Down vote post of dkjdjdfdskln
6.2k views
Posted by
dkjdjdfdskln
Reply
Add a Comment

Replies

Hi,

Can you paste pip list when it runs successfully vs when it fails ? Also in your Jupyter cells, are you installing any packages or changing PYTHONPATH? If you could provide a small repro case, that might help give some idea.

Posted by
Frameworks Engineer
Up vote reply of Frameworks Engineer Down vote reply of Frameworks Engineer

  • Hi,

    Thank you for your answer. please see the following pip list. I do not sure if I change the PYTHONPATH, but I know it can be fixed if I reinstalled it.

    —
    dkjdjdfdskln

  • I meet the same problem, if you have any method to fix it, please tell us. thanks you!

    —
    Xuehan_Tang
Add a Comment

Here is the pip list. and I cannot run python -c "import sklearn" on my terminal.

Package                  Version

absl-py                  0.10.0

aiohttp                  3.7.4.post0

anyio                    2.2.0

appnope                  0.1.2

argon2-cffi              20.1.0

astroid                  2.6.6

astunparse               1.6.3

async-generator          1.10

async-timeout            3.0.1

attrs                    21.2.0

autopep8                 1.5.6

Babel                    2.9.1

backcall                 0.2.0

beniget                  0.3.0

biopython                1.79

bleach                   4.0.0

blinker                  1.4

Bottleneck               1.3.2

brotlipy                 0.7.0

cached-property          1.5.2

cachetools               4.2.2

certifi                  2021.5.30

cffi                     1.14.5

chardet                  4.0.0

charset-normalizer       2.0.4

clang                    5.0

click                    8.0.3

cloudpickle              2.0.0

conda                    4.10.3

conda-package-handling   1.7.3

cryptography             3.4.7

cycler                   0.11.0

Cython                   0.29.24

decorator                5.1.0

defusedxml               0.7.1

deprecation              2.1.0

dill                     0.3.4

entrypoints              0.3

et-xmlfile               1.1.0

flake8                   3.9.0

flatbuffers              1.12

future                   0.18.2

gast                     0.4.0

google-auth              1.33.0

google-auth-oauthlib     0.4.1

google-pasta             0.2.0

googleapis-common-protos 1.53.0

grpcio                   1.38.1

gym                      0.21.0

h5py                     3.1.0

idna                     3.2

importlib-metadata       4.8.1

ipykernel                5.3.4

ipython                  7.26.0

ipython-genutils         0.2.0

isort                    5.9.3

jdcal                    1.4.1

jedi                     0.17.2

Jinja2                   3.0.2

joblib                   1.1.0

json5                    0.9.6

jsonschema               3.2.0

jupyter-client           7.0.1

jupyter-core             4.7.1

jupyter-http-over-ws     0.0.8

jupyter-lsp              1.5.0

jupyter-packaging        0.10.4

jupyter-server           1.6.4

jupyterlab               3.1.7

jupyterlab-lsp           3.9.1

jupyterlab-pygments      0.1.2

jupyterlab-server        2.8.2

keras                    2.6.0

Keras-Preprocessing      1.1.2

kiwisolver               1.3.2

lazy-object-proxy        1.6.0

Markdown                 3.3.4

MarkupSafe               2.0.1

matplotlib               3.4.2

matplotlib-inline        0.1.2

mccabe                   0.6.1

mistune                  0.8.4

msgpack                  1.0.2

msgpack-numpy            0.4.7.1

multidict                5.1.0

nbclassic                0.2.6

nbclient                 0.5.3

nbconvert                6.1.0

nbformat                 5.1.3

nest-asyncio             1.5.1

networkx                 2.6.3

notebook                 6.4.5

numexpr                  2.7.3

numpy                    1.19.5

oauthlib                 3.1.1

olefile                  0.46

openpyxl                 3.0.5

opt-einsum               3.3.0

packaging                21.0

pandas                   1.3.1

pandocfilters            1.4.3

parso                    0.7.0

pexpect                  4.8.0

pickleshare              0.7.5

Pillow                   8.4.0

pip                      21.3

pluggy                   0.13.1

ply                      3.11

prometheus-client        0.11.0

promise                  2.3

prompt-toolkit           3.0.20

protobuf                 3.17.2

ptyprocess               0.7.0

pyasn1                   0.4.8

pyasn1-modules           0.2.8

pybind11                 2.8.1

pycodestyle              2.6.0

pycosat                  0.6.3

pycparser                2.20

pydocstyle               6.1.1

pyflakes                 2.2.0

pyglet                   1.5.21

Pygments                 2.10.0

PyJWT                    2.1.0

pylint                   2.9.6

pyOpenSSL                21.0.0

pyparsing                3.0.4

pyrsistent               0.18.0

PySocks                  1.7.1

python-dateutil          2.8.2

python-jsonrpc-server    0.4.0

python-language-server   0.36.2

pythran                  0.9.11

pytz                     2021.3

pyzmq                    20.0.0

requests                 2.26.0

requests-oauthlib        1.3.0

rope                     0.19.0

rsa                      4.7.2

ruamel-yaml-conda        0.15.100

scikit-learn             1.0.1

scipy                    1.8.0.dev0+1921.c30fefc

seaborn                  0.11.2

Send2Trash               1.8.0

setuptools               52.0.0.post20210825

six                      1.15.0

sniffio                  1.2.0

snowballstemmer          2.1.0

tensorboard              2.6.0

tensorboard-data-server  0.6.0

tensorboard-plugin-wit   1.6.0

tensorflow-datasets      4.4.0

tensorflow-estimator     2.7.0

tensorflow-macos         2.6.0

tensorflow-metadata      1.4.0

tensorflow-metal         0.2.0

Tensorforce              0.6.5

termcolor                1.1.0

terminado                0.9.4

testpath                 0.5.0

threadpoolctl            3.0.0

toml                     0.10.2

tomlkit                  0.7.0

tornado                  6.1

tqdm                     4.62.3

traitlets                5.1.0

typing-extensions        3.7.4.3

ujson                    4.0.2

urllib3                  1.26.7

wcwidth                  0.2.5

webencodings             0.5.1

Werkzeug                 2.0.2

wheel                    0.35.1

wrapt                    1.12.1

yapf                     0.31.0

yarl                     1.6.3

zipp                     3.6.0

Posted by
dkjdjdfdskln
Up vote reply of dkjdjdfdskln Down vote reply of dkjdjdfdskln

  • pls. I installed a seaboard and after that sklearn failed to be imported.

    —
    dkjdjdfdskln
Add a Comment

I reinstalled the sklearn and scipy, after that sklearn can be imported correctly.

Package                  Version

absl-py                  0.10.0

aiohttp                  3.7.4.post0

anyio                    2.2.0

appnope                  0.1.2

argon2-cffi              20.1.0

astroid                  2.6.6

astunparse               1.6.3

async-generator          1.10

async-timeout            3.0.1

attrs                    21.2.0

autopep8                 1.5.6

Babel                    2.9.1

backcall                 0.2.0

beniget                  0.3.0

biopython                1.79

bleach                   4.0.0

blinker                  1.4

Bottleneck               1.3.2

brotlipy                 0.7.0

cached-property          1.5.2

cachetools               4.2.2

certifi                  2021.5.30

cffi                     1.14.5

chardet                  4.0.0

charset-normalizer       2.0.4

clang                    5.0

click                    8.0.3

cloudpickle              2.0.0

conda                    4.10.3

conda-package-handling   1.7.3

cryptography             3.4.7

cycler                   0.11.0

Cython                   0.29.24

decorator                5.1.0

defusedxml               0.7.1

deprecation              2.1.0

dill                     0.3.4

entrypoints              0.3

et-xmlfile               1.1.0

flake8                   3.9.0

flatbuffers              1.12

future                   0.18.2

gast                     0.4.0

google-auth              1.33.0

google-auth-oauthlib     0.4.1

google-pasta             0.2.0

googleapis-common-protos 1.53.0

grpcio                   1.38.1

gym                      0.21.0

h5py                     3.1.0

idna                     3.2

importlib-metadata       4.8.1

ipykernel                5.3.4

ipython                  7.26.0

ipython-genutils         0.2.0

isort                    5.9.3

jdcal                    1.4.1

jedi                     0.17.2

Jinja2                   3.0.2

joblib                   1.1.0

json5                    0.9.6

jsonschema               3.2.0

jupyter-client           7.0.1

jupyter-core             4.7.1

jupyter-http-over-ws     0.0.8

jupyter-lsp              1.5.0

jupyter-packaging        0.10.4

jupyter-server           1.6.4

jupyterlab               3.1.7

jupyterlab-lsp           3.9.1

jupyterlab-pygments      0.1.2

jupyterlab-server        2.8.2

keras                    2.6.0

Keras-Preprocessing      1.1.2

kiwisolver               1.3.2

lazy-object-proxy        1.6.0

Markdown                 3.3.4

MarkupSafe               2.0.1

matplotlib               3.4.2

matplotlib-inline        0.1.2

mccabe                   0.6.1

mistune                  0.8.4

msgpack                  1.0.2

msgpack-numpy            0.4.7.1

multidict                5.1.0

nbclassic                0.2.6

nbclient                 0.5.3

nbconvert                6.1.0

nbformat                 5.1.3

nest-asyncio             1.5.1

networkx                 2.6.3

notebook                 6.4.5

numexpr                  2.7.3

numpy                    1.19.5

oauthlib                 3.1.1

olefile                  0.46

openpyxl                 3.0.5

opt-einsum               3.3.0

packaging                21.0

pandas                   1.3.1

pandocfilters            1.4.3

parso                    0.7.0

pexpect                  4.8.0

pickleshare              0.7.5

Pillow                   8.4.0

pip                      21.3

pluggy                   0.13.1

ply                      3.11

prometheus-client        0.11.0

promise                  2.3

prompt-toolkit           3.0.20

protobuf                 3.17.2

ptyprocess               0.7.0

pyasn1                   0.4.8

pyasn1-modules           0.2.8

pybind11                 2.8.1

pycodestyle              2.6.0

pycosat                  0.6.3

pycparser                2.20

pydocstyle               6.1.1

pyflakes                 2.2.0

pyglet                   1.5.21

Pygments                 2.10.0

PyJWT                    2.1.0

pylint                   2.9.6

pyOpenSSL                21.0.0

pyparsing                3.0.4

pyrsistent               0.18.0

PySocks                  1.7.1

python-dateutil          2.8.2

python-jsonrpc-server    0.4.0

python-language-server   0.36.2

pythran                  0.9.11

pytz                     2021.3

pyzmq                    20.0.0

requests                 2.26.0

requests-oauthlib        1.3.0

rope                     0.19.0

rsa                      4.7.2

ruamel-yaml-conda        0.15.100

scikit-learn             1.0.1

scipy                    1.7.1

seaborn                  0.11.2

Send2Trash               1.8.0

setuptools               52.0.0.post20210825

six                      1.15.0

sniffio                  1.2.0

snowballstemmer          2.1.0

tensorboard              2.6.0

tensorboard-data-server  0.6.0

tensorboard-plugin-wit   1.6.0

tensorflow-datasets      4.4.0

tensorflow-estimator     2.7.0

tensorflow-macos         2.6.0

tensorflow-metadata      1.4.0

tensorflow-metal         0.2.0

Tensorforce              0.6.5

termcolor                1.1.0

terminado                0.9.4

testpath                 0.5.0

threadpoolctl            3.0.0

toml                     0.10.2

tomlkit                  0.7.0

tornado                  6.1

tqdm                     4.62.3

traitlets                5.1.0

typing-extensions        3.7.4.3

ujson                    4.0.2

urllib3                  1.26.7

wcwidth                  0.2.5

webencodings             0.5.1

Werkzeug                 2.0.2

wheel                    0.35.1

wrapt                    1.12.1

yapf                     0.31.0

yarl                     1.6.3

zipp                     3.6.0

Posted by
dkjdjdfdskln
Up vote reply of dkjdjdfdskln Down vote reply of dkjdjdfdskln
Add a Comment

did u fix it mistake? i meet the same alert message。how to fix it？

`ImportError: dlopen(/Users/mecilmeng/miniforge3/envs/tf/lib/python3.9/site-packages/scipy/spatial/qhull.cpython-39-darwin.so, 0x0002): Library not loaded: @rpath/liblapack.3.dylib Referenced from: /Users/mecilmeng/miniforge3/envs/tf/lib/python3.9/site-packages/scipy/spatial/qhull.cpython-39-darwin.so Reason: tried: '/Users/mecilmeng/miniforge3/envs/tf/lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/python3.9/site-packages/scipy/spatial/../../../../liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/lib/python3.9/site-packages/scipy/spatial/../../../../liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/bin/../lib/liblapack.3.dylib' (no such file), '/Users/mecilmeng/miniforge3/envs/tf/bin/../lib/liblapack.3.dylib' (no such file), '/usr/local/lib/liblapack.3.dylib' (no such file), '/usr/lib/liblapack.3.dylib' (no such file)

Posted by
mecil9
Up vote reply of mecil9 Down vote reply of mecil9

  • I also meet the same problem... and the same alert message. if anyone can fix it, please tell me how. thanks so much!

    —
    Xuehan_Tang

  • ImportError: dlopen(/opt/homebrew/Caskroom/miniforge/base/envs/tf_apple/lib/python3.8/site-packages/scipy/spatial/_qhull.cpython-38-darwin.so, 0x0002): Library not loaded: @rpath/liblapack.3.dylib Referenced from: /opt/homebrew/Caskroom/miniforge/base/envs/tf_apple/lib/python3.8/site-packages/scipy/spatial/_qhull.cpython-38-darwin.so Reason: tried: '/opt/homebrew/Caskroom/miniforge/base/envs/tf_apple/lib/python3.8/site-packages/scipy/spatial/../../../../liblapack.3.dylib' (no such file), '/opt/homebrew/Caskroom/miniforge/base/envs/tf_apple/lib/python3.8/site-packages/scipy/spatial/../../../../liblapack.3.dylib' (no such file), '/opt/homebrew/Caskroom/miniforge/base/envs/tf_apple/lib/liblapack.3.dylib' (no such file), '/opt/homebrew/Caskroom/miniforge/base/envs/tf_apple/bin/../lib/liblapack.3.dylib' (no such file), '/opt/homebrew/Caskroom/miniforge/base/envs/tf_apple/lib/liblapack.3.dylib' (no such file), '/opt/homebrew/Caskroom/miniforge/base/envs/tf_apple/bin/../lib/liblapack.3.dylib' (no such file), '/usr/local/lib/liblapack.3.dylib' (no such file), '/usr/lib/liblapack.3.dylib' (no such file)

    —
    Xuehan_Tang
Add a Comment

Hi. I've been suffering from the same problem but I finally found a simple solution which worked for me.

try this one:

pip install --upgrade --force-reinstall scikit-learn

I tested on Miniforge environment(macOS 12.2.1, M1 Air) and problem solved.

Posted by
enebin
Post not yet marked as solved Up vote reply of enebin Down vote reply of enebin

  • It works for me! thank you so much! However, I find that the outputs are different on mac m1 and windows when I am using the same tensorflow code, and I have no idea about that. You help me fix the problem about scikit-learn at least! thank you!

    —
    Xuehan_Tang

  • It works for me ! Thank you !

    —
    Yuankang

  • Worked for me thanks.

    —
    niladriray

NOTE: This answer involves copying dilyb(s) that may not be compatible all versions of sklearn. I highly recommend trying to re-install the packages first. If you continue to fail and have reached a point where you are willing to try anything, the following steps may resolve the issue.

I was able to resolve this by manually copying over a copy of the necessary dylib(s) out of a homebrew installation. First: I ran brew install lapack [NOTE: For me, this installed version (3.11), you may end up with a different version after running the command] Then I identified all the relevant files in of /opt/homebrew/Cellar/lapack/3.11/lib/*:

liblapacke.dylib
libblas.3.11.0.dylib
libblas.3.dylib
libblas.dylib
liblapack.3.11.0.dylib
liblapack.3.dylib
liblapack.dylib
liblapacke.3.11.0.dylib
liblapacke.3.dylib

Identify a search path that I all my python packages would search. I'm using miniconda for my python environment management, for me it was: ~/miniconda3/envs/maxEnt/lib/.

Step 1: Delete all existing files with a matching name in this lib folder.

Step 2: Copy the files referenced above.

Step 3: Update all the symlinks (aliases) in these files [.dylib points to 3.11.0.dylib; .3.dylib points to 3.11.0.dylib].

Posted by
fasmb24
Up vote reply of fasmb24 Down vote reply of fasmb24
Add a Comment