We have observed a per-process limitation on the number of simultaneous nw_connection_t objects in certain macOS environments. On some systems, this limit does not appear to apply, but on others the limitation is reproducible. When a process attempts to establish a large number of connections (e.g. 512+), some connections enter the nw_connection_state_waiting state and report the POSIX error “Cannot allocate memory”. These connections remain stuck indefinitely, even after other connections are deallocated and resources should theoretically be available again. This behavior severely impacts use cases such as transparent proxies implemented via the NetworkExtension framework, which intercept system-wide traffic and must open connections on behalf of all client processes. In this scenario, a per-process limit effectively becomes a system-wide limit, leading to unexpected and hard-to-diagnose network failures in client applications. So, is there any way to disable this restriction for Network Extension processes? Are there any system settings that could affect this limitation and be modified by users?
Network.Framework: Per-process simultaneous connection limit
We have observed a per-process limitation on the number of simultaneous nw_connection_t objects in certain macOS environments. On some systems, this limit does not appear to apply, but on others the limitation is reproducible. When a process attempts to establish a large number of connections (e.g. 512+), some connections enter the nw_connection_state_waiting state and report the POSIX error “Cannot allocate memory”. These connections remain stuck indefinitely, even after other connections are deallocated and resources should theoretically be available again.
The answer here is basically, yes, you're hitting a limit (which is ~512). More specifically, if you look at the thread that's getting "ENOMEM" in the system console and trace that thread "back" into the kernel, I believe you'll find that you’re getting log messages similar to this coming from the kernel:
kernel … SK[6]: flow_owner_flowadv_index_alloc nw_playground(44836) flow advisory table full: num 512 max 512
kernel … SK[6]: flow_entry_alloc failed to alloc flowadv index for flow 6E4A27CC-86A0-41FC-A51D-A1086A8E7D5B
If you're curious, you can actually see the code for this here and here.
So, is there any way to disable this restriction for Network Extension processes? Are there any system settings that could affect this limitation and be modified by users?
No, I don't think there is anything you can do to raise the limit. I think your only option is to let the system handle the flow or delay starting the flow until you have free connections.
__
Kevin Elliott
DTS Engineer, CoreOS/Hardware