Tensorflow-macos 2.8.0 Optimizers bugs

Question

Created Mar ’22

Replies 4

Boosts 0

Views 1.3k

Participants 2

tensorflow-macos=2.8.0 is bugged in the optimizers. I often get this kind of errors especially during hyperparams optimization

Always incompatible shape after same operation inside the optimizer (adam,nadam,sgd ..)

Macbook m1 max, OS 12.2.2.1. No metal installed because i dont need GPU ( in this case is really slow, since small batch size)

Here one of the many

"/Users/partiallycomplex/code/cr-autoencoder/.venv/lib/python3.9/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler return fn(*args, **kwargs) File "/Users/partiallycomplex/code/cr-autoencoder/.venv/lib/python3.9/site-packages/keras/engine/training.py", line 1384, in fit tmp_logs = self.train_function(iterator) File "/Users/partiallycomplex/code/cr-autoencoder/.venv/lib/python3.9/site-packages/keras/engine/training.py", line 1021, in train_function return step_function(self, iterator) File "/Users/partiallycomplex/code/cr-autoencoder/.venv/lib/python3.9/site-packages/keras/engine/training.py", line 1010, in step_function outputs = model.distribute_strategy.run(run_step, args=(data,)) File "/Users/partiallycomplex/code/cr-autoencoder/.venv/lib/python3.9/site-packages/keras/engine/training.py", line 1000, in run_step outputs = model.train_step(data) File "/Users/partiallycomplex/code/cr-autoencoder/.venv/lib/python3.9/site-packages/keras/engine/training.py", line 863, in train_step self.optimizer.minimize(loss, self.trainable_variables, tape=tape) File "/Users/partiallycomplex/code/cr-autoencoder/.venv/lib/python3.9/site-packages/keras/optimizer_v2/optimizer_v2.py", line 532, in minimize return self.apply_gradients(grads_and_vars, name=name) File "/Users/partiallycomplex/code/cr-autoencoder/.venv/lib/python3.9/site-packages/keras/optimizer_v2/optimizer_v2.py", line 671, in apply_gradients return tf.internal.distribute.interim.maybe_merge_call( File "/Users/partiallycomplex/code/cr-autoencoder/.venv/lib/python3.9/site-packages/keras/optimizer_v2/optimizer_v2.py", line 716, in _distributed_apply update_op = distribution.extended.update( File "/Users/partiallycomplex/code/cr-autoencoder/.venv/lib/python3.9/site-packages/keras/optimizer_v2/optimizer_v2.py", line 699, in apply_grad_to_update_var update_op = self._resource_apply_dense(grad, var, **apply_kwargs) File "/Users/partiallycomplex/code/cr-autoencoder/.venv/lib/python3.9/site-packages/keras/optimizer_v2/nadam.py", line 158, in _resource_apply_dense m_t_bar = (coefficients['one_minus_m_t'] * g_prime + Node: 'Nadam/Nadam/update_5/add_2' Incompatible shapes: [8] vs. [0] [[{{node Nadam/Nadam/update_5/add_2}}]] [Op:__inference_train_function_40987417]

Boost

Answer 1

Frameworks Engineer OP

Apple

Mar ’22

Hi @gk76

Thanks for reporting this! Based on just the error printout I can't really say what is going on here. This is especially interesting if it is happening with just the tensorflow-macos without the plugin. Would you happen to have a minimal script that would enable us to reproduce this issue?

Based on a quick look though the error seems to rise from the Keras package so it might be an issue on their side as well but I would need a script that allows me to reproduce this issue to say for certain.

0

Answer 2

gk76 OP

Mar ’22

Hi same code does not crash on mac intel. Never. I dont think is a problem from their side, I opened a ticket but the code I sent them doesnt crash on x32_x86 platform

attached the code. I cant upload some input file, because is not allowed to upload .zip

example.py

0

Answer 3

gk76 OP

Mar ’22

Hi, have you had time to work on the ticket? best gk

0

Answer 4

gk76 OP

Mar ’22

It seems that disabling eager execution do not produce the crashes in any hyperparameter optimization ( It was always crashing in any tf hyperparameter optimization, with any package: optuna, hyperopt, ..)

tf.compat.v1.disable_v2_behavior()

0