NEDNSProxyProvider: network troubles on iOS 14

I see that you are opening the remote side of the connection first, have you tried opening the local side of the flow, reading the datagrams, and then creating remote connections with NWConnection? Something like the following:

To perform this action with a local endpoint, you could implement something like the following:

1. Initialize your managing class with your localEndpoint, log, original flow, and worker queue. Start your connection.

2. Open the local flow with the localEndpoint.

3. Build your outbound copier because the local flow was opened first. So this is the flow reading from the local side of the connection. In this read you will get connection data for 1 or more outgoing datagrams that you will need to build and manage NWConnections for. I would recommend even a separate outbound datagram classes just for these NWConnections. The other side to number 3 would be the writer that will write data from the local datagrams to each one of the outbound connections, read from local flow, and write on remote connection.

4. The next part is the inbound copier from the remote connection. This is very similar to how TCP was done, but now you need to define an inbound copier for each one of your datagram connections so each one of these connections can receive data and write it to the local flow.

5. From there you would close each of the datagram connections on finish or error just as you normally would with cancel, closeReadWithError, and closeWriteWithError.

I also discuss this topic here: https://developer.apple.com/forums/thread/678464?page=1#671531022

Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

Thanks Matt. I rewrote the code according to your recommendations. The idea to encapsulate NWConnection is great. However, it still does not work (with the same symptoms). Any ideas?

import NetworkExtension


class DNSProxyProvider: NEDNSProxyProvider {
  private var connections = Set<UdpConnection>()
   
  override init() {
    super.init()
  }

  override func startProxy(options: [String: Any]? = nil,
               completionHandler: @escaping (Error?) -> Void) {
    completionHandler(nil)
  }

  override func stopProxy(with reason: NEProviderStopReason,
              completionHandler: @escaping () -> Void) {
    completionHandler()
  }

  override func sleep(completionHandler: @escaping () -> Void) {
    completionHandler()
  }

  override func wake() {}
   
  override func handleNewFlow(_ flow: NEAppProxyFlow) -> Bool {
    if let tcpFlow = flow as? NEAppProxyTCPFlow {
      NSLog("MyDebug: TCP connection")
    } else if let udpFlow = flow as? NEAppProxyUDPFlow {
      NSLog("MyDebug: UDP connection")
      openFlow(flow: udpFlow)
    }
    return true
  }

  private func openFlow(flow: NEAppProxyUDPFlow) {
    guard let local = flow.localEndpoint as? NWHostEndpoint else {
      NSLog("MyDebug: localEndpoint failed")
      return
    }
    flow.open(withLocalEndpoint: local) { opnErr in
      if let e = opnErr {
        NSLog("MyDebug: open error - \(e.localizedDescription)")
      } else {
        NSLog("MyDebug: open - ok")
        self.readFlow(flow: flow)
      }
    }
  }
   
  private func readFlow(flow: NEAppProxyUDPFlow) {
    flow.readDatagrams { (datagrams, endpoints, readError) in
      if let e = readError {
        NSLog("MyDebug: read error - \(e.localizedDescription)")
        return
      }
       
      let datas = self.extractReadData(datagrams: datagrams)
      for packet in datas {
        let udpConnection = UdpConnection(
          for: packet,
          from: flow,
          onReceive: self.onReceive(d:isComplete:flow:connection:))
        self.connections.insert(udpConnection)
        udpConnection.start()
      }
    }
  }
   
  private func extractReadData(datagrams: [Data]?) -> [Data] {
    if let datas = datagrams, !datas.isEmpty {
      NSLog("MyDebug: read - ok")
      return datas
    } else {
      NSLog("MyDebug: read - datagrams is empty or null")
      return []
    }
  }
     
  private func onReceive(d: Data,
              isComplete: Bool,
              flow: NEAppProxyUDPFlow,
              connection: UdpConnection) {
    connections.remove(connection)
    flow.writeDatagrams([d], sentBy: [flow.localEndpoint!]) { writeError in
      if let e = writeError {
        NSLog("MyDebug: write error - \(e.localizedDescription)")
      } else {
        NSLog("MyDebug: write - ok")
      }
    }
  }
}

import NetworkExtension


class UdpConnection: Hashable {
  static func == (lhs: UdpConnection, rhs: UdpConnection) -> Bool {
    return ObjectIdentifier(lhs) == ObjectIdentifier(rhs)
  }
  var hashValue: Int {
    return ObjectIdentifier(self).hashValue
  }
   
  private let onReceive: (_ d: Data, _ isComplete: Bool, _ flow: NEAppProxyUDPFlow, _ connection: UdpConnection) -> Void
  private let flow: NEAppProxyUDPFlow
  private let connection: NWConnection
   
  init(for packet: Data,
     from flow: NEAppProxyUDPFlow,
     onReceive: @escaping (_ d: Data, _ isComplete: Bool, _ flow: NEAppProxyUDPFlow, _ connection: UdpConnection) -> Void) {
    self.onReceive = onReceive
    self.flow = flow
    connection = NWConnection(host: "8.8.8.8", port: 53, using: .udp)
    connection.stateUpdateHandler = { state in
      self.connectionHandler(state: state, packet: packet)
    }
    start()
  }
   
  func start() {
    connection.start(queue: .global())
  }
   
  private func connectionHandler(state: NWConnection.State,
                  packet: Data) {
    switch state {
    case .ready:
      NSLog("MyDebug: connectionHandler ready")
      connection.send(content: packet,
              completion: .contentProcessed( { sendError in
        if let e = sendError {
          NSLog("MyDebug: send error - \(e.localizedDescription)")
        } else {
          NSLog("MyDebug: send - ok")
          self.receive()
        }
      }))
    case .setup:
      NSLog("MyDebug: connectionHandler setup")
    case .cancelled:
      NSLog("MyDebug: connectionHandler cancelled")
    case .preparing:
      NSLog("MyDebug: connectionHandler preparing")
    default:
      NSLog("MyDebug: connectionHandler default")
    }
  }
   
  private func receive() {
    connection.receiveMessage { [self] (data, context, isComplete, receiveError) in
      let d = self.extractReceivedData(data: data, receiveError: receiveError)
      onReceive(d, isComplete, flow, self)
    }
  }
   
  private func extractReceivedData(data: Data?,
                   receiveError: NWError?) -> Data {
    if receiveError == nil, let d = data {
      NSLog("MyDebug: receive - ok")
      return d
    } else {
      if let e = receiveError {
        NSLog("MyDebug: receive error - \(e.localizedDescription)")
      } else {
        NSLog("MyDebug: receive data - \(data)")
      }
      return Data()
    }
  }
   
}

Any ideas?

Regarding this part:

  private func readFlow(flow: NEAppProxyUDPFlow) {
    flow.readDatagrams { (datagrams, endpoints, readError) in
      if let e = readError {
        NSLog("MyDebug: read error - \(e.localizedDescription)")
        return
      }
       
      let datas = self.extractReadData(datagrams: datagrams)
      for packet in datas {
        let udpConnection = UdpConnection(
          for: packet,
          from: flow,
          onReceive: self.onReceive(d:isComplete:flow:connection:))
        self.connections.insert(udpConnection)
        udpConnection.start()
      }
    }
  }

  init(for packet: Data,
     from flow: NEAppProxyUDPFlow,
     onReceive: @escaping (_ d: Data, _ isComplete: Bool, _ flow: NEAppProxyUDPFlow, _ connection: UdpConnection) -> Void) {
    self.onReceive = onReceive
    self.flow = flow
    connection = NWConnection(host: "8.8.8.8", port: 53, using: .udp)
    connection.stateUpdateHandler = { state in
      self.connectionHandler(state: state, packet: packet)
    }
    start()
  }

You'll want to check that extractReadData is giving you usable data when [Data] is returned and then based on each datagram in the array I would have expected that you would be able to extract a Network.NWEndpoint from the datagram and open a NWConnection from there. This essentially puts your proxy in passthrough mode, but it's often a good technique to debug what is happening or how you need to adjust things.

Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com