How to write a NWProtocolFramer for Network.framework that splits streams into frames using a delimiter?

I tried to create a framer that splits a stream of ASCII bytes into frames separated by the pipe ascii character: "|".

So I made an NWProtocolFramerImplementation as seen in the attached document.
import Network

fileprivate let pipe = Character("|").asciiValue!

class PipeFramer: NWProtocolFramerImplementation {
    static let label = "Pipe framer"
    static let definition = NWProtocolFramer.Definition(implementation: PipeFramer.self)

    var minLengthUntilNextMessage = 1 {
        didSet { print("client: minLength set to", minLengthUntilNextMessage) }
    }

    required init(framer: NWProtocolFramer.Instance) {}

    func start(framer: NWProtocolFramer.Instance) -> NWProtocolFramer.StartResult { .ready }

    func handleInput(framer: NWProtocolFramer.Instance) -> Int {
        while true {
            var delimiterPosition: Int?
            _ = framer.parseInput(minimumIncompleteLength: minLengthUntilNextMessage, maximumLength: 65535) { buffer, endOfMessage in
                if let buffer = buffer {
                    print("client: parsing buffer: \"\(String(bytes: buffer, encoding: .utf8) ?? buffer.debugDescription)\"")
                    if let indexOfDelimiter = buffer.firstIndex(of: pipe) {
                        minLengthUntilNextMessage = 1
                        delimiterPosition = indexOfDelimiter
                    } else {
                        minLengthUntilNextMessage = buffer.count + 1
                    }
                } else {
                    print("client: no buffer")
                }
                return 0
            }

            if let length = delimiterPosition {
                guard framer.deliverInputNoCopy(length: length, message: .init(instance: framer), isComplete: true) else {
                    return 0
                }
                _ = framer.parseInput(minimumIncompleteLength: 1, maximumLength: 65535) { _,_ in 1 }
            } else {
                return minLengthUntilNextMessage
            }
        }
    }

    func handleOutput(framer: NWProtocolFramer.Instance, message: NWProtocolFramer.Message, messageLength: Int, isComplete: Bool) {
        try! framer.writeOutputNoCopy(length: messageLength)
        framer.writeOutput(data: [pipe])
    }

    func wakeup(framer: NWProtocolFramer.Instance) {}

    func stop(framer: NWProtocolFramer.Instance) -> Bool { return true }

    func cleanup(framer: NWProtocolFramer.Instance) { }
}

The problem is that from the moment I get a chunk that does not end with "|", the framer gets stuck on that chunk. So the other chunks that come after this incomplete chunk never fully arrive in the framer.parseInput(...) call. Because it always parses chunks of minimumIncompleteLength and hence never arrives to the point where the next "|" is.

Here is a simple reproduction of this problem:
  1. Create a TCP server

  2. Setup the server so that it sends chunks of messages when a client connects.

  3. Connect to the server (created in 1.) using the framer from above.

  4. Start receiving messages.

Code Block import Networklet client = DispatchQueue(label: "Server")let server = DispatchQueue(label: "Client")let networkParameters = NWParameters.tcpnetworkParameters.defaultProtocolStack.applicationProtocols.insert(NWProtocolFramer.Options(definition: PipeFramer.definition), at: 0)let server = try! NWListener(using: .tcp)server.newConnectionHandler = { connection in    print("server: new connection from", connection.endpoint)    print("server (client \(connection.endpoint)): state", connection.state)    connection.viabilityUpdateHandler = { viable in        print("server (client \(connection.endpoint)): state", connection.state)        if viable {            print("server: sending")            connection.send(content: "A|Be||Sea".data(using: .utf8)!, isComplete: false, completion: .idempotent)            serverQueue.asyncAfter(deadline: .now() + 5) {                print("server: sending second part")                connection.send(content: " is longer than expected|0|".data(using: .utf8)!, isComplete: true, completion: .idempotent)            }            serverQueue.asyncAfter(deadline: .now() + 8) {                print("server: sending last part")                connection.send(content: "Done|".data(using: .utf8)!, isComplete: true, completion: .idempotent)            }        }    }    connection.start(queue: serverQueue)}server.stateUpdateHandler = { state in    print("server:", state)    if state == .ready, let port = server.port {        print("server: listening on", port)    }}server.start(queue: serverQueue)let client = NWConnection(to: .hostPort(host: "localhost", port: server.port!), using: networkParameters)func receiveNext() {    client.receiveMessage { (data, context, complete, error) in        let content: String        if let data = data {            content = String(data: data, encoding: .utf8) ?? data.description        } else {            content = data?.debugDescription ?? "<no data>"        }        print("client: received \"\(content)\"", context.debugDescription, complete, error?.localizedDescription ?? "No error")        receiveNext()    }}client.stateUpdateHandler = { state in    print("client:", state)    if state == .ready {        print("client: receiving")        receiveNext()    }}client.start(queue: clientQueue)

Results in:
Code Block server: waiting(POSIXErrorCode: Network is down)server: readyserver: listening on 54894client: preparingclient: readyclient: receivingserver: new connection from ::1.53179server (client ::1.53179): state setupserver (client ::1.53179): state readyserver: sendingclient: parsing buffer: "A|BeSea"client: minLength set to 1client: parsing buffer: "BeSea"client: minLength set to 1client: parsing buffer: "|Sea"client: minLength set to 1client: parsing buffer: "Sea"client: minLength set to 4client: parsing buffer: ""client: minLength set to 1client: received "A" Optional(Network.NWConnection.ContentContext) true No errorclient: received "Be" Optional(Network.NWConnection.ContentContext) true No errorclient: received "<no data>" Optional(Network.NWConnection.ContentContext) true No errorclient: parsing buffer: "Sea"client: minLength set to 4server: sending second partclient: parsing buffer: "Sea "client: minLength set to 5client: parsing buffer: "Sea i"client: minLength set to 6server: sending last partclient: parsing buffer: "Sea is"client: minLength set to 7client: parsing buffer: "Sea is "client: minLength set to 8


Notice that the fourth and fifth message are never received by the client. How should I write the Framer so that it receives messages after an incoming incomplete chunk?
Yep, this can be difficult because you will need to save previously read data so that it can fit into the next frame. Take a look at a TCP / UDP framing implementation I wrote here on this post that was based off the framer in the TicTacToe example. This would be a great place to start with this.


Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com
Thanks for the quick response Matt. I looked at your other post but that is demonstrating a TLV (type, length, value) framer. I was talking about separating frames using a separation character (or delimiter).

you will need to save previously read data so that it can fit into the next frame

Hmm... Why should we be saving the old chunk's data while it is still available in NWProtocolFramer.Instance's parseInput(...) function. The old data (which you refer to as previously read data) is not lost (and I don't think it is marked read until you call deliverInput...(...) or return its length in parseInput(...)). The problem I am experiencing, is that the framer does not get the new data (but is stuck on the old data).

I made small GitHub repo where you can debug the problem: https://github.com/Dev1an/Simple-Framer

Could you explain how your other posts (TicTacToe & TLV) are relevant to this delimiting problem (where the message lengths are not known in advance)?

Could you explain how your other posts (TicTacToe & TLV) are relevant to this delimiting problem (where the message lengths are not known in advance)?

These are examples of how to handle frame parsing. As you mentioned you situation is unique because you do not know that length of the data you need to read and this can create some tricky situations. For more detailed help on this please open a TSI and I would be happy to help dig in.

Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

Did you ever find a solution to this? I am stuck with the same issue, where an incomplete ASCII sentence stalls the receiving client, and keep trying to pass the same data over and over again, until the connection is finally canceled.

Any help or hint would be greatly appreciate!!

How to write a NWProtocolFramer for Network.framework that splits streams into frames using a delimiter?
 
 
Q