Network Framework HTTPS Response Headers

I have been trying to set up a HTTP Client which uses the Apple Network Framework for it's connection and TLS needs. Using Objective-C calls in a C++ project. I have been playing around with the Implementation at Implementing netcat with Network Framework . After a successful TLS handshake, I send a request to the server using the following code snippet:

dispatch_data_t content = dispatch_data_create(headers.c_str(), strlen(headers.c_str()), dispatch_get_main_queue(), DISPATCH_DATA_DESTRUCTOR_DEFAULT);

    nw_connection_send(m_connectionObject, content, NW_CONNECTION_DEFAULT_MESSAGE_CONTEXT, true, ^ (nw_error_t nwError) {
        if (nwError != NULL)
        {
            auto errNumber = nw_error_get_error_code(nwError);
            logger->Error("Error sending data to [%s]:%d : Error Number :%d", request.Host(), request.Port(), errNumber);
        }
        else
        {
            logger->Debug("Sent %u byte(s) of HTTP headers", headersLen);
        }
    });

where std::string headers is a stream of HTTPS request headers. The sending operation is successful but when it comes to receiving block which starts after the above snippet, the Network framework directly produces the content offered by the destination. I need to access the server's HTTP response headers first. Is there a way to do that?

Also I need to continue using this framework and can not more to NSURLSessions stuff.

The receive block:

nw_connection_receive(m_connectionObject, 1, UINT32_MAX, ^ (dispatch_data_t content, nw_content_context_t context, bool isComplete, nw_error_t receiveError) {

        nw_retain(context);

        dispatch_block_t schedule_next_receive = ^ {
            // If the context is marked as complete, and is the final context,
            // we're read-closed.
            if (isComplete && (context == NULL || nw_content_context_get_is_final(context)))
            {
                logger->Info(
                    "HTTP Client was remotely disconnected from destination.");

                nw_connection_cancel(m_connectionObject);
            }

            // If there was no error in receiving, request more data
            if (receiveError == NULL)
            {
                ReceiveResponse(request, response, bodyParser);
            }
            nw_release(context);
        };

        if (content != NULL)
        {
            schedule_next_receive = Block_copy(schedule_next_receive);

            //Convert dispatch_data_t to std::string 
            std::string message = CreateStringWithDispatchData(content);

            //Parse the response message, false means unsuccessful parsing
            if(ProcessResponse(message))
            {
                schedule_next_receive();
            }
            else
            {
                nw_connection_cancel(m_connectionObject);
            }
        }
        else
        {
            // Content was NULL, so directly schedule the next receive
            schedule_next_receive();
        }
    });
Accepted Answer

The sending operation is successful but when it comes to receiving block which starts after the above snippet, the Network framework directly produces the content offered by the destination. I need to access the server's HTTP response headers first.

If I'm understanding your correctly then you are not receiving the HTTP response and only the response body? The first thing I would do here is try reworking the receive loop a bit with something like the following:

-(void) receivedDataOnConnection {
    __weak NetworkManager *wSelf = self;
    
    nw_connection_receive(connection, 0, 2048, ^(dispatch_data_t content, nw_content_context_t context, bool is_complete, nw_error_t receive_error) {
        
        int messageLength = 0;
        //For multiple reads, you will have to have a class wide data container.
        if (is_complete && content != nil) {
            NSData* bufferData = (NSData*)content;
            messageLength = (int)[bufferData length];
            NSString* htmlString = [NSString stringWithUTF8String:[bufferData bytes]];
            // Either do something with `htmlString` or append bufferData to multiple a mutable chunk of data for multiple reads.
        }
        
        if (receive_error != NULL) {
            // Log out the error and cancel the connection
            nw_connection_cancel(self.connection);
            return;
        }
        
        // Send a new read call to read more data if the connection is not marked as complete.
        if (!is_complete && messageLength > 0) {
            os_log_debug(self.log, "reading more data");
            [wSelf receivedDataOnConnection];
            
        } else if (is_complete) {
            // If complete, cancel the connection.
            nw_connection_cancel(self.connection);
        }
    });
}

Next, if you are still not able to receive the HTTP response, then you may need build out a parser as a first step. This is not an easy task but you could take a look at the CFHTTPMessage APIs to build the request and parse the response. The benefit here is the you could use a NSURLRequest as the HTTP request and then the response could be handled as data. The downside is that this approach is re-inventing the wheel a bit because you cannot use NSURLSession.

Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

I agree with everything in Matt’s response but I wanted to ask about the big picture. Why are you trying to create your own HTTP implementation?

This matters because creating an HTTP client is hard. Getting a basic HTTP 1.0 [1] client up and running is not too bad. Upgrading that to HTTP 1.1 is a challenge, especially when it comes to chunked encoded and the 100-continue stuff, but still feasible. Implementing HTTP/2 is much harder, and HTTP/3 is another major step beyond that.

And keep in mind that each HTTP upgrade offers significant performance benefits. The difference in performance between HTTP 1.0 and HTTP/3 is staggering.

The good news here is that NSURLSession does this all for you. But you don’t get that benefit if you do this all yourself.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] There’s no official standard for HTTP 1.0, so that in itself presents some challenges (-:

Thanks for the quick reply. I am able to see the HTTP response headers now. Sorry but the problem was some dodgy parsing code I had written for the received data string. The reason I do not want to use NSURL because this is a C++ project and I dont want to write separate objective-c source files and hook into those using C++ wrappers in my main project. If NSURL was accessible directly in a C++ project, I would have definitely done so.

The reason I do not want to use NSURL because this is a C++ project and I dont want to write separate objective-c source files and hook into those using C++ wrappers in my main project. If NSURL was accessible directly in a C++ project, I would have definitely done so.

I do agree that this does add an extra layer of code when working with a C++ based project, but using Objective-C/++ in your project will be a lot easier for you than writing your own HTTP client libraries from the ground up, in particular the HTTP parser.

Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com

Currently, I am only targeting HTTP 1.1. For that, I already have a parser, so my job becomes easier that way. But I do agree if we ever need to upgrade to newer versions, then I would have to switch to NSURLSessions. Can you kindly point me to some documentation/examples for the translation layer between obj-c and C++. I will surely give that a try. Thanks for all the help :)

Can you kindly point me to some documentation/examples for the translation layer between obj-c and C++.

The documentation here is kind of sparse, but there is some good information over in the Objective-C Automatic Reference Counting ARC docs. Since I cannot post the URL as a link, here is the URL directly:

https://clang.llvm.org/docs/AutomaticReferenceCounting.html#c-c-compatibility-for-structs-and-unions-with-non-trivial-members

Matt Eaton
DTS Engineering, CoreOS
meaton3@apple.com
Network Framework HTTPS Response Headers
 
 
Q