Reboot also helped here. "Have you tried turning it off and on again?"

Indeed I'm running fewer software processes on the Mac Mini, so that might be a contributing factor as well.

There seem to be no solutions except not to run other apps while listening to music. The only real solution is for Apple to fix their shit.

TabasKo, I don't seem to have Core Audio Driver running as an Intel process, but CarbonComponentScannerXPC seems to be constantly there.

Yes, I think I was too optimistic in this discovery. It could be that the these particular Intel processes I'm running cause more memory/CPU pressure than native "Apple" processes and therefore cause audio issues more quickly since they seem to be correlated but for sure they are not the whole story, as you say. It certainly is maddening and I'd really love to hear a serious comment from Apple addressing this issue.

Unfortunately it does not seem to be limited to this particular process but to all Intel type processes running.

Btw, unfortunately the above fix didn't help.

TabasKo, I hear you. When a new expensive supposedly high-quality laptop craps out on a basic function like USB audio, it makes me feel I've been scammed.