It seams that is is an issue generally with Xcode 11 as I have tried 11.3, 11.4, and 11.5 and it still has the same issue.
I have checked that two factor is activated and it is. I have also tried my brothers account and it doesn't work either.
My main issue is it works fine with Xcode 12 beta 3 which properly asks for the two factor pin. I developed an app with Xcode 12 beta and need to go build it with Xcode 11 for release and can't.