Deep Learning Example Chapter 11 Part 2 - Sequence Model Crashed OS X 12.3.1

Question

dbl001 OP

Created Apr ’22

Replies 1

Boosts 0

Participants 2

I am running on an iMac 27" 2020 with an AMD Radeon Pro 5700 XT GPU.

  
% pip show tensorflow-macos             
WARNING: Ignoring invalid distribution -umpy (/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages)
Name: tensorflow-macos
Version: 2.8.0
Summary: TensorFlow is an open source machine learning framework for everyone.
Home-page: https://www.tensorflow.org/
Author: Google Inc.
Author-email: packages@tensorflow.org
License: Apache 2.0
Location: /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages
Requires: absl-py, astunparse, flatbuffers, gast, google-pasta, grpcio, h5py, keras, keras-preprocessing, libclang, numpy, opt-einsum, protobuf, setuptools, six, tensorboard, termcolor, tf-estimator-nightly, typing-extensions, wrapt
Required-by: 
(tensorflow-metal) (base) davidlaxer@x86_64-apple-darwin13 deep-learning-with-python-notebooks % pip show tensorflow-metal
WARNING: Ignoring invalid distribution -umpy (/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages)
Name: tensorflow-metal
Version: 0.4.0
Summary: TensorFlow acceleration for Mac GPUs.
Home-page: https://developer.apple.com/metal/tensorflow-plugin/
Author: 
Author-email: 
License: MIT License. Copyright © 2020-2021 Apple Inc. All rights reserved.
Location: /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages
Requires: six, wheel
Required-by:

Here's the code in a Jupyter Notebook:

https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/chapter11_part02_sequence-models.ipynb

Monterey crashed in Epoch 1 of training:

 callbacks = [
    keras.callbacks.ModelCheckpoint("one_hot_bidir_lstm.keras",
                                    save_best_only=True)
]
model.fit(int_train_ds, validation_data=int_val_ds, epochs=10, callbacks=callbacks)
model = keras.models.load_model("one_hot_bidir_lstm.keras")
print(f"Test acc: {model.evaluate(int_test_ds)[1]:.3f}")

 Epoch 1/10
2022-04-01 08:55:17.903921: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2022-04-01 08:55:18.427409: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2022-04-01 08:55:18.463345: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2022-04-01 08:55:32.254473: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
2022-04-01 08:55:32.276648: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
  3/625 [..............................] - ETA: 2:03:10 - loss: nan - accuracy: 0.5000

Kernel Panic

Boost

Answer 1

Frameworks Engineer OP

Apple

Apr ’22

Hi @dbl001,

Thanks for providing the link to the notebook as well as description of the system that this happens on. I'll try to reproduce this issue next and figure out what is causing the crash you are seeing. I'll update here once I have a conclusion on the issue.

0


	% pip show tensorflow-macos
	WARNING: Ignoring invalid distribution -umpy (/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages)
	Name: tensorflow-macos
	Version: 2.8.0
	Summary: TensorFlow is an open source machine learning framework for everyone.
	Home-page: https://www.tensorflow.org/
	Author: Google Inc.
	Author-email: packages@tensorflow.org
	License: Apache 2.0
	Location: /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages
	Requires: absl-py, astunparse, flatbuffers, gast, google-pasta, grpcio, h5py, keras, keras-preprocessing, libclang, numpy, opt-einsum, protobuf, setuptools, six, tensorboard, termcolor, tf-estimator-nightly, typing-extensions, wrapt
	Required-by:
	(tensorflow-metal) (base) davidlaxer@x86_64-apple-darwin13 deep-learning-with-python-notebooks % pip show tensorflow-metal
	WARNING: Ignoring invalid distribution -umpy (/Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages)
	Name: tensorflow-metal
	Version: 0.4.0
	Summary: TensorFlow acceleration for Mac GPUs.
	Home-page: https://developer.apple.com/metal/tensorflow-plugin/
	Author:
	Author-email:
	License: MIT License. Copyright © 2020-2021 Apple Inc. All rights reserved.
	Location: /Users/davidlaxer/tensorflow-metal/lib/python3.8/site-packages
	Requires: six, wheel
	Required-by:

	callbacks = [
	keras.callbacks.ModelCheckpoint("one_hot_bidir_lstm.keras",
	save_best_only=True)
	]
	model.fit(int_train_ds, validation_data=int_val_ds, epochs=10, callbacks=callbacks)
	model = keras.models.load_model("one_hot_bidir_lstm.keras")
	print(f"Test acc: {model.evaluate(int_test_ds)[1]:.3f}")

	Epoch 1/10
	2022-04-01 08:55:17.903921: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
	2022-04-01 08:55:18.427409: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
	2022-04-01 08:55:18.463345: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
	2022-04-01 08:55:32.254473: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
	2022-04-01 08:55:32.276648: I tensorflow/core/grappler/optimizers/custom_graph_optimizer_registry.cc:113] Plugin optimizer for device_type GPU is enabled.
	3/625 [..............................] - ETA: 2:03:10 - loss: nan - accuracy: 0.5000