It is possible that this is related to the security policies enforced by App Transport Security (ATS). If this is the case, it would explain the behavior as ATS is being enforced when the app is installed via TestFlight, but ATS is not enforced when installed via the ad hoc method.
ATS places security requirements on the servers that your app connects to. If the server does not meet the requirements, ATS will not allow the app to communicate with that server. For example, if the websockets server you are connecting to is just plain HTTP, the app will not be able to connect to it when ATS is enforced.
It is the responsibility of the developer to make sure servers under your control are updated to meet the security requirements. If the server is not under your control, there are options to allow exceptions. (App Review will make you justify the exceptions.)
Here is documentation on ATS:
https://developer.apple.com/library/content/documentation/General/Reference/InfoPlistKeyReference/Articles/CocoaKeys.html#//apple_ref/doc/uid/TP40009251-SW33
Try following the instructions in the section "Debugging ATS Connections". That should help you to determine if ATS is actually the root cause for your issue.
https://developer.apple.com/library/content/documentation/General/Reference/InfoPlistKeyReference/Articles/CocoaKeys.html#//apple_ref/doc/uid/TP40009251-SW40