Post not yet marked as solved
We did further testing and we figured that if intermediate CA certificates are available on either side (part of anchor certs from client or part of trust object from server), they are simply used for establishing the trust chain to root but if any intermediate certificate is missing on both side, then evaluation fails. Here are our observations:
Given we have a Root certificate (rootCA), two intermediate certificates (iCA1 and iCA2)
Given leaf certificate is always available on both sides
If server presents rootCA as well as iCA1 and iCA2 while the client anchors rootCA as well as iCA1 and iCA2, the trust evaluation is successful.
If server presents rootCA as well as iCA1 and iCA2 while the client anchors rootCA as well as iCA1 but no iCA2, the trust evaluation is successful.
If server presents rootCA as well as iCA1 and iCA2 while the client anchors rootCA as well as iCA2 but no iCA1, the trust evaluation is successful.
If server presents only iCA2 while the client anchors rootCA as well as iCA2 but no iCA1, the trust evaluation fails since iCA1 is missing on both sides.
If server presents iCA1 and iCA2 while the client anchors rootCA only, the trust evaluation is successful.
If server presents only iCA2 while the client anchors rootCA and iCA1, the trust evaluation is successful.
Are these outcomes expected? Is this how certificate evaluation expected to behave?
Post not yet marked as solved
Thanks for reply, Matt. We do not have the certificates in any of keychains and we do not have MDM on the machine. We have also tried setting true in SecTrustSetAnchorCertificatesOnly(_:_:) so that only our certificates are used for validation. The outcome is still the same.
Thanks for the tip @eskimo. Currently we are supporting macOS 11+ but will definitely use the new API for Monterey. Here is the feedback for documentation update: FB9964722
As for the original issue, we noticed that replacement with same version only happens when developer mode is enabled (systemextensionsctl developer on), on the machine, regardless of SIP status. Once we turn off developer mode, the API starts working properly and we do not get delegate callback for replacement of same version. It directly jumps to completed result.
I am not sure why this behaviour and could not find any documentation regarding this.
I hope this information helps others in future.
Post not yet marked as solved
We opened a TSI where Matt informed us that NWProtocolWebSocket does not have HTTP stack (unlike URLSessionWebSocketTask) and can not parse HTTP responses from server. Therefore, any error returned as part of HTTP response from server during WebSocket handshake will not be available to clients.
We have opened a feedback to have ability to parse and return HTTP response in NWProtocolWebSocket: FB9878278
Post not yet marked as solved
We are not using SCDynamicStore and have very limited understanding of it. Can you let us know how can we use this to get status of utun interface? How can we get a change in status callback and is it possible to use any API to change the status from container app?
Post not yet marked as solved
In container app we are observing for both NEVPNConfigurationChange and NEVPNStatusDidChange notifications. But we do not receive any update there.
Probably unrelated but only callback or update we receive is in path update handler of NWPathMonitor but path remains satisfied with same interface.
Post not yet marked as solved
Thanks Matt. In console logs I do not see any error or unexpected log. Here is a screenshot of logs when bringing the interface down: https://ibb.co/7gz331J and here is a screenshot of when bringing it up: https://ibb.co/kyNyTGV
I would recommend controlling this behavior from either your container app or Network System Extension
As per our design, we are providing users with control from container app. However, we noticed some (geeky) users are bringing our virtual interface down, then bringing it back up using ifconfig and reporting that our packet tunnel is not working anymore.
So, if there is a way for our container app or system extension to be notified when the interface is brought down/up, we could react to it by setting the tunnel settings again and bringing up the virtual interface. Is that at all possible? If yes, how?
To be honest this doesn't surprise me because of what I mentioned above, the provider and system state are out of sync.
When bringing any other physical interface down and back up, it starts working fine in few seconds. So why this issue only with packet tunnel providers? Shouldn't system keep the state in sync like any other interface? Provider itself is not informed about this change anyways.
Finally I was able to solve this problem. Here are few things we learned and fixed:
At first we were not receiving the FIN,ACK message from server when connection was closed from server side. When we looked at server packet capture, we could see that server was sending FIN,ACK but was not receiving ACK from client. So, server was retransmitting FIN,ACK. Similarly, client was not aware of server closing the connection and when trying to send something to server was also not being ACKed by server and was being retransmitted. This was happening after 5 or more minutes of idle. We narrowed it down to a NAT issue where NAT device was losing the mapping of client IP after certain period of idle connection. So, after losing the mapping, both server and client were not able to reach each other and were in re-transmission. We fixed this by sending TCP keep-alive every one minute during idle connection.
After fixing that we started getting FIN,ACK from server but were not able to receive it in NWConnection callback. SSL_shutdown from server closes the write direction and sends a FIN,ACK to client. On client side, this is indicates by isFinal property of NWConnection.ContentContext received in receiveMessage completion handler. Our mistake was that we were expecting either data or error in the completion handler to be non-nil. Otherwise, we were discarding the call-back. But when FIN,ACK is sent by server, error and data, both are nil and isFinal in context is set. This gives the hint that server has closed the write channel and client side should also send any remaining message and close the connection. Once we implemented that, we were able to fix complete issue.
Thanks @eskimo for your help and suggestions. Let me know if you see any issue with above fixes or any improvements that can be made.
Sure, I will open a TSI next week. Meanwhile, continuing my investigation, I looked into console logs and I can see SSL3_RT_ALERT, description: close notify. Which means Mac client is receiving TLS close but is not closing the connection.
The screenshot for console log is available here: https://ibb.co/bXYrPT4
The server side calls SSL_shutdown which only closes the write direction from server side while keeping read side open.
SSL_shutdown() only closes the write direction. It is not possible to call SSL_write() after calling SSL_shutdown(). The read direction is closed by the peer.
So, as per the OpenSSL docs, server is doing right thing but NWConnection is not responding back with notify close, to finish the connection from both sides.
Hi @eskimo,
After our discussion last evening, we did some deliberations and now we are able to get FIN from server to client. However, we still notice that NWConnection does not close the connection after receiving FIN and subsequent writes are being sent to server. Is there any reason NWConnection is not closing web-socket after receiving FIN and ACKing to the FIN?
Our server has separate management of read and write web sockets. They are sending FIN after closing their write socket while keeping their read socket open for a little while longer so that client can write any last message or send FIN and close.
Could there be any reason/configuration for NWConnection to not disconnect by writing client FIN after receiving server FIN? In Windows client, we see client side FIN after server FIN and then connection closes completely.
The packet capture screenshot with server FIN can be seen https://ibb.co/M6cLz85 since image upload here was failing.
Here is the code we use for reading data continuously on web socket. We only stop reading only when we encounter an error.
func listen() {
connection?.receiveMessage { [weak self] (data, context, _, error) in
guard let self = self else {
return
}
if let data = data, !data.isEmpty, let context = context {
self.receiveMessage(data: data, context: context)
}
if let error = error {
self.reportErrorOrDisconnection(error)
} else {
self.listen()
}
}
}
Sorry, I mistyped my previous post and did not notice it till now. Here is the right version:
I am not sure I completely understood what do you mean but let me try to answer that: I add an FQDN in searchDomains and then I try to access the same exact FQDN in browser. But the DNS queries for that FQDN are not sent to my PacketTunnel.
Is this what you are asking?
Post not yet marked as solved
Are you handling this DNS query in your System Extension?
Within the SystemExtension, I am actually trying to connect to server FQDN using NWConnection(to: NWEndpoint.url(server_FQDN), using: parameters). So, I am expecting the FQDN to be resolved to IP from hosts file and connection being established. But that is not happening.
If you try to resolve the query in the context of the System Extension does this respect the hosts file?
I did not try resolving the host myself using something like getaddrinfo or CFHostStartInfoResolution. So, I am not sure about this behaviour.
Also, just FYI, I have split tunnel and a few matchDomains in my extension but they are very different from server FQDN I am using to connect to, in case this matters.
do you pickup the exact values in searchDomains?
I am not sure I completely understood what do you mean but let me try to answer that: I add an FQDN in searchDomains and then I try to access the same exact FQDN in browser. But the DNS queries for that FQDN are sent to my PacketTunnel.
Is this what you are asking?
Hi Matt, since I had never used searchDomains, I just did some testing and results are not what I expected.
I added few domains like google.com, amazon.com, www-google.com and www-amazon.com (- are dots here since www URLs are not allowed) into search domains, added nothing in matchDomains and matchDomainsNoSearch is set to true . So, I expected to get DNS queries for all the above exact domains in my NE. But that is not the case. I get no DNS queries at all.
If I add the same domains in matchDomains and nothing in searchDomains, I get all the exact as well as subdomain queries. Which is working as expected but is not something I need. I only need queries for exact domain names. So, how can that be achieved? I thought adding only to searchDomains would do that. But that is not the case.
