[MPSGraph adamUpdateWithLearningRateTensor:beta1Tensor:beta2Tensor:epsilonTensor:beta1PowerTensor:beta2PowerTensor:valuesTensor:momentumTensor:velocityTensor:gradientTensor:name:]: unrecognized selector sent to instance 0x600000eede10

I am running tensorflow-macos and tensorflow-metal version 2.6 on Monterey Beta (21A5543b) on an iMac 27" 2021 with an AMD Radeon GPU.

I got the following error training the model VariationalDeepSemanticHashing e.g.

2021-10-09 13:05:14.521286: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.

2021-10-09 13:05:27.092823: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled.

2021-10-09 13:05:27.153 python[6315:1459657] -[MPSGraph adamUpdateWithLearningRateTensor:beta1Tensor:beta2Tensor:epsilonTensor:beta1PowerTensor:beta2PowerTensor:valuesTensor:momentumTensor:velocityTensor:gradientTensor:name:]: unrecognized selector sent to instance 0x600000eede10

[I 2021-10-09 13:05:28.157 ServerApp] AsyncIOLoopKernelRestarter: restarting kernel (1/5), keep random ports

kernel d25e6066-74f7-4b4a-b5e7-b2911e7501d9 restarted

https://github.com/unsuthee/VariationalDeepSemanticHashing/blob/master/Run_Experiment_Unsupervised.ipynb

Here's the repository:

https://github.com/unsuthee/VariationalDeepSemanticHashing

Up vote post of dbl001 Down vote post of dbl001
6.7k views
Posted by
dbl001
Reply

  • This code reproduces the crash ```In [2]: import tensorflow as tf ...: ...: mnist = tf.keras.datasets.mnist ...: ...: (x_train, y_train), (x_test, y_test) = mnist.load_data() ...: x_train, x_test = x_train / 255.0, x_test / 255.0 ...: ...: model = tf.keras.models.Sequential([ ...: tf.keras.layers.Flatten(input_shape=(28, 28)), ...: tf.keras.layers.Dense(128, activation='relu'), ...: tf.keras.layers.Dropout(0.2), ...: tf.keras.layers.Dense(10) ...: ]) ...: ...: predictions = model(x_train[:1]).numpy() ...: tf.nn.softmax(predictions).numpy() ...: ...: loss_fn = tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True ...: ) ...: ...: loss_fn(y_train[:1], predictions).numpy() ...: ...: model.compile(optimizer = 'adam', loss = loss_fn) ...: model.fit(x_train, y_train, epochs=100) Epoch 1/100 2021-10-10 10:50:53.503460: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:112] Plugin optimizer for device_type GPU is enabled. 2021-10-10 10:50:53.527 python[25080:3485800] -[MPSGraph adamUpdateWithLearningRateTensor:beta1Tensor:beta2Tensor:epsilonTensor:beta1PowerTensor:beta2PowerTensor:valuesTensor:momentumTensor:velocityTensor:gradientTensor:name:]: unrecognized selector sent to instance 0x6000037975a0 zsh: segmentation fault ipython

    —
    dbl001

  • This code reproduces the crash ...

    —
    dbl001

  • I have the same problem here. Python: 3.0.10 macOS: monterey 12.0.1 macbook pro 2018 with Radeon Pro 560X 4 GB

    But before I upgrade to Monterey, The scripts works just fine with Big Sur.

    —
    qinyuhang
Add a Comment

Replies

I have a similar issue with kernel crashing with Adam:

Model: "sequential"

Layer (type) Output Shape Param #

masking (Masking) (None, 1, 28) 0

layer1 (Bidirectional) (None, 1, 128) 47616

dropout (Dropout) (None, 1, 128) 0

layer2 (Bidirectional) (None, 1, 128) 98816

dropout_1 (Dropout) (None, 1, 128) 0

layer3 (Bidirectional) (None, 128) 98816

Output (Dense) (None, 1) 129

================================================================= Total params: 245,377 Trainable params: 245,377 Non-trainable params: 0

Epoch 1/3000 2022-03-29 13:26:54.863499: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.

**2022-03-29 13:26:57.201 python[66880:3338950] -[MPSGraph adamUpdateWithLearningRateTensor:beta1Tensor:beta2Tensor:epsilonTensor:beta1PowerTensor:beta2PowerTensor:valuesTensor:momentumTensor:velocityTensor:gradientTensor:name:]: unrecognized selector sent to instance 0x600041707c60 ** zsh: segmentation fault python test.py

Posted by
raymondjiii
Up vote reply of raymondjiii Down vote reply of raymondjiii
Add a Comment

I tried swapping Adam for RMSprop - still get an error - but now it's a floating point error.

2022-03-29 13:35:35.219059: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:305] Could not identify NUMA node of platform GPU ID 0, defaulting to 0. Your kernel may not have been built with NUMA support. 2022-03-29 13:35:35.219398: I tensorflow/core/common_runtime/pluggable_device/pluggable_device_factory.cc:271] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 0 MB memory) -> physical PluggableDevice (device: 0, name: METAL, pci bus id: ) Model: "sequential"

Layer (type) Output Shape Param #

masking (Masking) (None, 1, 28) 0

layer1 (Bidirectional) (None, 1, 128) 47616

dropout (Dropout) (None, 1, 128) 0

layer2 (Bidirectional) (None, 1, 128) 98816

dropout_1 (Dropout) (None, 1, 128) 0

layer3 (Bidirectional) (None, 128) 98816

Output (Dense) (None, 1) 129

================================================================= Total params: 245,377 Trainable params: 245,377 Non-trainable params: 0

Epoch 1/3000 2022-03-29 13:35:51.773828: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2022-03-29 13:35:55.049448: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2022-03-29 13:35:55.409149: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2022-03-29 13:35:58.358459: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2022-03-29 13:35:58.544457: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2022-03-29 13:35:58.809214: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2022-03-29 13:35:58.989445: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2022-03-29 13:36:01.117681: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled. 2022-03-29 13:36:01.338093: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.

zsh: floating point exception python test.py

Posted by
raymondjiii
Up vote reply of raymondjiii Down vote reply of raymondjiii
Add a Comment

I have that exact same problem:

[MPSGraph adamUpdateWithLearningRateTensor:beta1Tensor:beta2Tensor:epsilonTensor:beta1PowerTensor:beta2PowerTensor:valuesTensor:momentumTensor:velocityTensor:gradientTensor:name:]: unrecognized selector sent to instance 0x60002836b9c0

I tried switching Adam to RMSprop and then I get floating point exception.

Posted by
raymondjiii
Up vote reply of raymondjiii Down vote reply of raymondjiii
Add a Comment

I encountered a similar problem on an iMac with a 3.6GHz quad-core Intel Core i7 and Radeon Pro 560 4 GB. Adam crashes repeatedly. I tried using tfa.optimizer.RectifiedAdam instead. It works (and uses the GPU!) but it is significantly slower than Adam on the CPU. The HuggingFace glue/mrpc fine-tuning example takes ~15min per epoch on the CPU, but over an hour on the GPU.

Posted by
zenkat
Up vote reply of zenkat Down vote reply of zenkat
Add a Comment