We had the same Apple ID association issue.
For internal testers, I was able to resolve it by completely removing the user from App Store Connect, then re-adding the user and adding them again as a TestFlight tester.
Unfortunately, this approach does not work for external testers. Removing the tester from the app/TestFlight and sending a new invitation does not reset the Apple ID association, and the tester still receives an error saying the invitation cannot be accepted because the email address does not match.
Has anyone found a reliable workaround for external testers in this situation?