Exception Type: EXC_CRASH (SIGABRT) Exception Codes: 0x0000000000000000, 0x0000000000000000 Exception Note: EXC_CORPSE_NOTIFY

[I am running Top2vec on Big Sur 11.6 with tensorflow-macos and tensorflow-metal. Python crashed ...

linkText

Crashed Thread:        0  Dispatch queue: metal gpu stream

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Application Specific Information:
/System/Volumes/Data/SWE/macOS/BuildRoots/38cf1d983f/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MetalPerformanceShaders-124.6.1/MPSCore/Utility/MPSCommandBufferImageCache.mm:1386: failed assertion `Failed to allocate private MTLBuffer for size 421888000

Crash Log
Up vote post of dbl001 Down vote post of dbl001
1.9k views
Posted by
dbl001
Reply
Add a Comment

Replies

Hi dbl001, Can you share the script where we can reproduce this issue? Also we would recommend you update to MacOS 12.0.

Posted by
Frameworks Engineer
Up vote reply of Frameworks Engineer Down vote reply of Frameworks Engineer

  • I installed MacOS 12.0 Beta and attempted to train the model with tensorflow_macos and tensorflow_metal 2.6. Training the model consumed ~110GB DRAM before I killed the kernel (the iMac has 128GB). I tried reducing the number of input rows in the training set to: 18806 rows × 5 columns, but it still runs out of memory. Also, I can't tell from the Activity monitor whether the GPU is running.

    —
    dbl001
Add a Comment

I installed the latest versions of tensorflow-macos and tensorflow-metal on OS X 11.6. Now, it no longer prints out that it's using metal or my AMD GPU.

% ipython
In [3]: import tensorflow
No supported GPU was found.

I installed the latest versions from PyPi into my existing tensorflow-metal virtual environement with:

% pip install tensorflow-macos==2.6.0
% pip install tensorflow-metal=0.2.0

What's changed? Do I need to recreate the tensorflow-metal virtual environment from scratch?

 % pip show tensorflow-metal
Name: tensorflow-metal
Version: 0.2.0
Summary: TensorFlow acceleration for Mac GPUs.
Home-page: https://developer.apple.com/metal/tensorflow-plugin/
Author: 
Author-email: 
License: MIT License. Copyright © 2020-2021 Apple Inc. All rights reserved.
Location: /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages
Requires: wheel, six
Required-by: 
(tensorflow-metal) (base) davidlaxer@x86_64-apple-darwin13 Top2Vec % pip show tensorflow-macos
Name: tensorflow-macos
Version: 2.6.0
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.

% pip show
tensorboard                2.6.0
tensorboard-data-server    0.6.1
tensorboard-plugin-profile 2.5.0
tensorboard-plugin-wit     1.8.0
tensorflow                 2.6.0
tensorflow-consciousness   0.1
tensorflow-datasets        4.3.0
tensorflow-determinism     0.3.0
tensorflow-estimator       2.6.0
tensorflow-gan             2.1.0
tensorflow-hub             0.12.0
tensorflow-macos           2.6.0
tensorflow-metadata        1.1.0
tensorflow-metal           0.2.0
tensorflow-probability     0.13.0
tensorflow-similarity      0.13.45
tensorflow-text            2.6.0
Posted by
dbl001
Up vote reply of dbl001 Down vote reply of dbl001

  • How do I send you a Jupyter notebook?

    —
    dbl001

  • Also, why doesn't the GPU run when Tensorflow is in Eager Execution mode? In the prior version of 'tensorflow-metal', I had to disable Eager Mode for the GPU to run:

    tf.compat.v1.enable_v2_behavior() from tensorflow.python.framework.ops import disable_eager_execution disable_eager_execution()
    —
    dbl001

  • You can find the notebook in the notebook directory in this repository:

    https://github.com/ddangelov/Top2Vec

    —
    dbl001
Add a Comment

im facing the same issue with tensorflow

Posted by
saeedmaktek
Up vote reply of saeedmaktek Down vote reply of saeedmaktek

  • I have the same problem with PyTorch. iMac 27" 2020 with Mac OS Ventura 13.2.

    /AppleInternal/Library/BuildRoots/c651a45f-806e-11ed-a221-7ef33c48bc85/Library/Caches/com.apple.xbs/Sources/MetalPerformanceShaders/MPSCore/Utility/MPSCommandBufferImageCache.mm:1411: failed assertion `Failed to allocate private MTLBuffer for size 78643200

    The monitor shows 12,3GB used of 16 installed when it crashes

    —
    agilll
Add a Comment