Error Domain=NSURLErrorDomain Code=-1004 “Could not connect to the server.” on trying to shift from https to http on same app session

We have similar servers on two types of hardware devices, which we need to connect to using iOS application. Currently we are using Alamofire 4.9.1 for all the API requests.
Previously we supported only http endpoints on our servers, but recently we have moved to https on latest version of the devices.The app needs to support both.
The issue that we are facing is, when we connect to a Device X(which supports https), and then try to connect to Device Y(which only support http), we are getting this error

Code Block [Error] GET 'http://172.30.1.1/file.xml' [27.7252 s]:
Error Domain=NSURLErrorDomain Code=-1004 "Could not connect to the server." UserInfo={​​​​​​​NSUnderlyingError=0x281f2fde0 {​​​​​​​Error Domain=kCFErrorDomainCFNetwork Code=-1004 "(null)" UserInfo={​​​​​​​_kCFStreamErrorCodeKey=61, _kCFStreamErrorDomainKey=1}​​​​​​​}​​​​​​​, NSErrorFailingURLStringKey=https://172.30.1.1/file.xml, NSErrorFailingURLKey=https://172.30.1.1/file.xml, _kCFStreamErrorDomainKey=1, _kCFStreamErrorCodeKey=61, NSLocalizedDescription=Could not connect to the server.}​​​​​​​

Connecting to device Y(http) directly without first connecting to Device X(https) works fine, and after the error if we kill the app we are able to connect to Device Y(http). This happens even though we are using different SessionManagers in Alamofire.
Is this because we are using the same ip address? If we check the error NSErrorFailingURLStringKey, it is still https although the url we are calling is http.
I am pasting the relevant code here

Code Block private var manager: Alamofire.SessionManager = {
        let configuration = URLSessionConfiguration.default
        let manager = Alamofire.SessionManager(configuration: configuration)
        manager.retrier = RetryHandler()
        return manager
    }()
    private var manager2: Alamofire.SessionManager = {
        var serverTrustPolicies: [String: ServerTrustPolicy] = [
            "172.30.1.1": .disableEvaluation
        ]
        let config = URLSessionConfiguration.ephemeral
       
        /*Adding TrustManager to Allow All Certificates*/
        let manager = Alamofire.SessionManager(configuration: config, serverTrustPolicyManager: ServerTrustPolicyManager(policies: serverTrustPolicies))
        manager.adapter = RequestInterceptor()
        manager.retrier = RequestInterceptor()
        
        return manager
    }()

Things we have already tried :

1. Use different UrlSessionConfiguration

2. Set config.urlCache = nil