UIImagePickerController shows all photos even if I choose "Select Photos.." option

UIImagePickerController runs out-of-process since iOS 11. There is no need to prompt for Photos Library access before showing the picker on iOS 11 and above, unless you really need to work with the selected PHAssets.

As outlined in this years session, there are some considerations for using the picker with Limited Library:

The picker will still show the entire Photos Library and all photos and videos can be selected by the user. No matter what the users selects in the picker, the PHAssets you can access will not change.

There is no support to only show assets in the picker that you app has access to when in Limited Library mode.
Please feel free to provide feedback and outline the use case where your app would need to further restrict the selection.

Thank you for your support!

Is that mean, we do not need to ask permission to access photos library and I can use photos/videos in my app without taking user permission?

If I go with this approach, will this create any problem when I will submit my app for review process before it is live on AppStore? Is there any chance my app will get rejected as I am accessing photos library without user permission?

When running on iOS 11 or above, your app don't need to ask for Photos Library permission before presenting UIImagePickerController (and assuming the app doesn't use PhotoKit/doesn't access UIImagePickerControllerPHAsset from the result).
Forcing users to give your app full Photos Library access before presenting UIImagePickerController could be considered an anti-pattern.