Liquid glass - precent automatic color changes

I have an app that displays a MapView. While I am in light mode everything is fine. I can scroll around the map and my overlays (made by UIVisualEffectView containing an UIGlassEffect) stay light and look well! As soon as I change my phone to dark mode, depending on what's underneath the buttons (a light residential area or darker wooded areas) some of my buttons change color. But not all, only where it's supposedly lighter or darker underneath. This makes my whole UI look strange. Some buttons bright, some dark. Is there a way to lock a "color" or interfaceStyle to the effects-view? In light mode everything is fine, but in dark mode it just looks super strange.

Liquid glass - precent automatic color changes
 
 
Q