App Rejected for requiring phone number at sign-up

Can you collect phone numbers following sign-in with apple?

I am creating a firebase backed app and offering users the option to sign in with third party apps such as facebook and google. According to Apple, this means i must also include an option to sign-in with Apple; however my app is getting rejected in the review process for attempting to grab more information from the user after they authenticate with apple. I get that sign-in with apple is supposed to be a one click solution to getting the user interacting with the core of the app; however, we require the user's phone number for app functionality. Is there any way to get the app approved in this situation while still getting the user's phone number at sign-up? I do not see an option to grab the number from the sign-in with apple credential like you can for name.