Sorry for the cross-post but it's now two days in and this isn't fixed.
If you try to use Xcode 16.3b3 with visionOS, it won't download the visionOS SDK, gives a 'network error' so you can't use the latest beta for Apple Vision Pro.
visionOS 2.4b4 SDK Missing / Cannot Be Downloaded
Looks like this may now be fixed.
can confirm. as of now (March 20) download is OK - fixed.
visionOS 2.4 beta 4 (22O5230a) SDK + visionOS 2.4 beta 4 (22O5231b) Simulator (Installed)