Faulty, hard-to-understand XPC behavior with remote methods that have a reply-block

Assume this over-simplified @protocol I'm using for my XPC-service:

@protocol MyMinimalProtocol <NSObject>
- (void)getStatusWithReply:(void (^ _Nullable)(NSDictionary * _Nonnull))reply;
@end

The Client side would then

NSXPCConnection *connection =  [[NSXPCConnection alloc] initWithMachServiceName:myServiceLabel options:NSXPCConnectionPrivileged];

connection.remoteObjectInterface = [NSXPCInterface interfaceWithProtocol:@protocol(MyMinimalProtocol)];
connection.interruptionHandler = ^{  NSLog(@"XPC: connection - interrupted"); };
connection.invalidationHandler = ^{ NSLog(@"XPC: connection - invalidated"); };
[connection resume];
[connection.remoteObjectProxy getStatusWithReply:^(NSDictionary * response) {
        NSLog(@"XPC service status received - %@", response);
}];

So far - so good. My XPC service receives the asynchronous call, schedules it's "status gathering operation" on internal background queue, and returns. Later - when information is available, my XPC service executes the reply-block then, on the remote calling side - I see the log line with the status, as expected.

BUT!!!

If I add another different code-block argument to the method e.g.

@protocol MyMinimalProtocol <NSObject>
- (void)getStatusWithReply:(void (^ _Nullable)(NSDictionary * _Nonnull))reply andFailureBlock:(void (^ _Nullable)(NSError * _Nonnull))failureBlock;
@end

Then all hell breaks loose. Both XPC service and the client crash with hilarious crash reasons I can't decipher.

Here's "Client side" caller crash (excerpt - forgive the methods are NOT the simplified ones above)

```
-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               SomeApp [49547]
Path:                  /Library/Some/Dir/SomeApp.app/Contents/MacOS/SomeApp
Identifier:            com.myCompany.myApp
Code Type:             ARM-64 (Native)
Parent Process:        launchd [1]
User ID:               504

Date/Time:             2023-09-01 01:17:46.1415 +0300
OS Version:            macOS 12.6.8 (21G725)
Report Version:        12
Anonymous UUID:        0D13BB92-970A-1888-D656-624C48FD6184

Sleep/Wake UUID:       95E5F68C-EFD3-4E6E-8FC2-313397367BDD

Time Awake Since Boot: 160000 seconds
Time Since Wake:       6115 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BREAKPOINT (SIGTRAP)
Exception Codes:       0x0000000000000001, 0x00000001a4592080
Exception Note:        EXC_CORPSE_NOTIFY

Termination Reason:    Namespace SIGNAL, Code 5 Trace/BPT trap: 5
Terminating Process:   exc handler [49547]

Application Specific Information:
BUG IN CLIENT OF LIBPLATFORM: Trying to recursively lock an os_unfair_lock
Abort Cause 259

Kernel Triage:
VM - pmap_enter failed with resource shortage

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   libsystem_platform.dylib      	       0x1a4592080 _os_unfair_lock_recursive_abort + 36
1   libsystem_platform.dylib      	       0x1a458d174 _os_unfair_lock_lock_slow + 312
2   Foundation                    	       0x1a54e0474 -[NSXPCInterface setProtocol:] + 100
3   Foundation                    	       0x1a54ea264 +[NSXPCInterface interfaceWithProtocol:] + 52
4   ViewBridge                    	       0x1ab97fff4 __auxiliaryProxyFor_block_invoke + 84
5   ViewBridge                    	       0x1ab97fea8 auxiliaryProxyFor + 372
6   ViewBridge                    	       0x1ab97f518 _ensureAuxServiceAwareOfHostApp + 308
7   ViewBridge                    	       0x1ab986b8c __26+[NSRemoteView initialize]_block_invoke_2 + 576
8   ViewBridge                    	       0x1ab986820 withAutoreleasePoolAndExceptionProcessing + 60
9   ViewBridge                    	       0x1ab9867d8 __26+[NSRemoteView initialize]_block_invoke + 100
10  libdispatch.dylib             	       0x1a43b25f0 _dispatch_call_block_and_release + 32
11  libdispatch.dylib             	       0x1a43b41b4 _dispatch_client_callout + 20
12  libdispatch.dylib             	       0x1a43c26cc _dispatch_main_queue_drain + 928
13  libdispatch.dylib             	       0x1a43c231c _dispatch_main_queue_callback_4CF + 44
14  CoreFoundation                	       0x1a4686968 __CFRUNLOOP_IS_SERVICING_THE_MAIN_DISPATCH_QUEUE__ + 16
15  CoreFoundation                	       0x1a4643bd8 __CFRunLoopRun + 2532
16  CoreFoundation                	       0x1a4642a54 CFRunLoopRunSpecific + 600
17  HIToolbox                     	       0x1ad287338 RunCurrentEventLoopInMode + 292
18  HIToolbox                     	       0x1ad2870b4 ReceiveNextEventCommon + 564
19  HIToolbox                     	       0x1ad286e68 _BlockUntilNextEventMatchingListInModeWithFilter + 72
20  AppKit                        	       0x1a71ab4b8 _DPSNextEvent + 860
21  AppKit                        	       0x1a71a9db0 -[NSApplication(NSEvent) _nextEventMatchingEventMask:untilDate:inMode:dequeue:] + 1328
22  AppKit                        	       0x1a719bf7c -[NSApplication run] + 596
23  AppKit                        	       0x1a716d698 NSApplicationMain + 1132
24  SomeApp                   	       0x102e6bbe8 main + 416 (main.m:63)
25  dyld                          	       0x10305d08c start + 520
```

while on the "XPC Service" side, crashes like these:

```
-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               MyService [49657]
Path:                  /Library/Some/Dir/MyService
Identifier:            com.myCompany.myService
Version:               1.0a25 (1.0.0a25(1))
Code Type:             ARM-64 (Native)
Parent Process:        launchd [1]
User ID:               0

Date/Time:             2023-09-01 01:20:26.0958 +0300
OS Version:            macOS 12.6.8 (21G725)
Report Version:        12
Anonymous UUID:        0D13BB92-970A-1888-D656-624C48FD6184

Sleep/Wake UUID:       95E5F68C-EFD3-4E6E-8FC2-313397367BDD

Time Awake Since Boot: 160000 seconds
Time Since Wake:       6275 seconds

System Integrity Protection: enabled

Crashed Thread:        1  Dispatch queue: com.apple.NSXPCListener.service.com.myCompany.myService

Exception Type:        EXC_CRASH (SIGABRT)
Exception Codes:       0x0000000000000000, 0x0000000000000000
Exception Note:        EXC_CORPSE_NOTIFY

Application Specific Information:
abort() called

Application Specific Backtrace 0:
0   CoreFoundation                      0x00000001a46c5138 __exceptionPreprocess + 240
1   libobjc.A.dylib                     0x00000001a440fe04 objc_exception_throw + 60
2   Foundation                          0x00000001a54e0ebc __setProtocolMetadataWithSignature_block_invoke + 0
3   Foundation                          0x00000001a54e09c4 setProtocolMetdataWithMethods + 576
4   Foundation                          0x00000001a54e06d8 setProtocolMetadata + 264
5   Foundation                          0x00000001a54e054c -[NSXPCInterface setProtocol:] + 316
6   Foundation                          0x00000001a54ea264 +[NSXPCInterface interfaceWithProtocol:] + 52
7   ITProtector                         0x00000001042ad1a4 -[OITPreventionXPCService listener:shouldAcceptNewConnection:] + 1356
8   Foundation                          0x00000001a55ac984 service_connection_handler_make_connection + 180
9   libxpc.dylib                        0x00000001a42ad168 _xpc_connection_call_event_handler + 152
10  libxpc.dylib                        0x00000001a42abeac _xpc_connection_mach_event + 2140
11  libdispatch.dylib                   0x00000001a43b4274 _dispatch_client_callout4 + 20
12  libdispatch.dylib                   0x00000001a43d053c _dispatch_mach_msg_invoke + 464
13  libdispatch.dylib                   0x00000001a43bb784 _dispatch_lane_serial_drain + 376
14  libdispatch.dylib                   0x00000001a43d125c _dispatch_mach_invoke + 456
15  libdispatch.dylib                   0x00000001a43bb784 _dispatch_lane_serial_drain + 376
16  libdispatch.dylib                   0x00000001a43bc438 _dispatch_lane_invoke + 444
17  libdispatch.dylib                   0x00000001a43c6c98 _dispatch_workloop_worker_thread + 648
18  libsystem_pthread.dylib             0x00000001a4574360 _pthread_wqthread + 288
19  libsystem_pthread.dylib             0x00000001a4573080 start_wqthread + 8

Kernel Triage:
VM - pmap_enter failed with resource shortage
VM - pmap_enter failed with resource shortage

Thread 1 Crashed::  Dispatch queue: com.apple.NSXPCListener.service.com.myCompany.myService
0   libsystem_kernel.dylib        	       0x1a4542d78 __pthread_kill + 8
1   libsystem_pthread.dylib       	       0x1a4577ee0 pthread_kill + 288
2   libsystem_c.dylib             	       0x1a44b2340 abort + 168
3   libc++abi.dylib               	       0x1a4532b18 abort_message + 132
4   libc++abi.dylib               	       0x1a4522a54 demangling_terminate_handler() + 336
5   libobjc.A.dylib               	       0x1a4418320 _objc_terminate() + 144
6   libc++abi.dylib               	       0x1a4531eb4 std::__terminate(void (*)()) + 20
7   libc++abi.dylib               	       0x1a4531e50 std::terminate() + 64
8   libdispatch.dylib             	       0x1a43b4288 _dispatch_client_callout4 + 40
9   libdispatch.dylib             	       0x1a43d053c _dispatch_mach_msg_invoke + 464
10  libdispatch.dylib             	       0x1a43bb784 _dispatch_lane_serial_drain + 376
11  libdispatch.dylib             	       0x1a43d125c _dispatch_mach_invoke + 456
12  libdispatch.dylib             	       0x1a43bb784 _dispatch_lane_serial_drain + 376
13  libdispatch.dylib             	       0x1a43bc438 _dispatch_lane_invoke + 444
14  libdispatch.dylib             	       0x1a43c6c98 _dispatch_workloop_worker_thread + 648
15  libsystem_pthread.dylib       	       0x1a4574360 _pthread_wqthread + 288
16  libsystem_pthread.dylib       	       0x1a4573080 start_wqthread + 8
```

I wonder if there's something inherently wrong with having two code-block arguments for an XPC remote method?

Another issue. The client XPC calls are asynchronous. They return immediately. The XPC service implementing the remote-call also returns immediately - and it executes the "reply block" far (a minute!) later, on another queue.

However, if the XPC service attempts to execute the code-block MORE THAN ONCE, then the client-side code-block is only called ONCE. rest of the executions look benign in the XPC-service side - but never happen on the calling (client) side.

Any idea why? can this be overcome?

Any thoughts/ideas/references to documentation will be greatly appreciated. I couldn't find any good source on this.

Thanks.

Answered by DTS Engineer in 763699022

I wonder if there's something inherently wrong with having two code-block arguments for an XPC remote method?

Correct. NSXPCConnection requires that all protocol methods have at most one block parameter, that it be the last parameter, and that it has reply semantics. Adding extra block parameters isn’t supported.

However, if the XPC service attempts to execute the code-block MORE THAN ONCE

That’s also not supported.

You need to design your protocols with these restrictions in mind. For example:

  • You might have a single method that completes immediately with a request token.

  • And then have other methods that get the latest status for that token.

You can use an anonymous connection (via NSXPCListenerEndpoint) for this if you want.

You can also use NSProgress.

It’s also possible to set up bi-directional messaging, but that’s challenging because you want the server to be resilient in the face of a wedged client.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Accepted Answer

I wonder if there's something inherently wrong with having two code-block arguments for an XPC remote method?

Correct. NSXPCConnection requires that all protocol methods have at most one block parameter, that it be the last parameter, and that it has reply semantics. Adding extra block parameters isn’t supported.

However, if the XPC service attempts to execute the code-block MORE THAN ONCE

That’s also not supported.

You need to design your protocols with these restrictions in mind. For example:

  • You might have a single method that completes immediately with a request token.

  • And then have other methods that get the latest status for that token.

You can use an anonymous connection (via NSXPCListenerEndpoint) for this if you want.

You can also use NSProgress.

It’s also possible to set up bi-directional messaging, but that’s challenging because you want the server to be resilient in the face of a wedged client.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Just a reminder: Please reply in a reply. If you reply in the comments, I’m not notified. See tip 5 in Quinn’s Top Ten DevForums Tips.


In what doc/TechNote - can I find these facts (what's supported and what isn't)?

The current official documentation for NSXPCConnection leaves a lot to be desired )-: Please do file a bug about that.

The only doc I could find that covers this is in the Documentation Archive, namely Daemons and Services Programming Guide > Creating XPC Services > Using the Service > Designing an Interface [1], which says:

Because communication over XPC is asynchronous, all methods in the protocol must have a return type of void. If you need to return data, you can define a reply block like this … A method can have only one reply block.

The rest of that doc is also well worth a read.

However, my all-time favourite documentation for this is WWDC 2012 Session 241 Cocoa Interprocess Communication with XPC. That’s long since been removed from the Apple Developer website but, if you have a copy sequestered away somewhere, it’s well worth your time.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Which is the Apple documentation equivalent of Beware of the Leopard. (-:

Thanks again.

Indeed I missed the sentence "A method can have only one reply block." in the old archived documentation - but that same sentence also says:

A method can have only one reply block. However, because connections are bidirectional, the XPC service helper can also reply by calling methods in the interface provided by the main application, if desired.

Which brings me to the original question -- HOW do I do that? My "players" aren't an App and its helper-service, but rather independent, resilient system daemons and global agents that need communicate, in all directions. The dual XPC connection I'm maintaining now, is just cumbersome. Each "product shutdown" flow is an ugly nightmare, and recovering from one-side-crash looks different on either sides - which is ugly too. Bidirectional XPC connections look like the right thing to me.

I found and downloaded the recommended 2012 WWDC session from here: https://archive.org/download/wwdc-2012-sessions and watched it carefully twice - but there too - it is only HINTED for a split second, that the connection is bidirectional and both sides can send messages - and immediately the hint is removed from screen, and replaced by that single reply-block technique.

There is a tiny gap here I need to bridge. I already have a live NSXPCConnection, both sides agree on the same protocol, the "client side" obtains a "remote proxy" and the "server side" exports an object and assigns them to accepted incoming connection. Then client sends messages to the service.

How can the client-side create an "exported object"? can it also assign an NSXPCInterface and exported-object to its NSXPCConnection when it connects to the service? and how would the service get the proxy if it wants to call-back to the client?

If the docs say NSXPCConnection is bi-directional, SOMETHING must be said somewhere regarding the use of this feature.

As in your hinted novel - I think I'll just go on and open the "disused lavatory" carrying that "Beware of the Leopard" sign, to find the demolition order for my house :(

One things keeps me uneasy. In the full decade of NSXPCConnection, hasn't anyone need bi-directional IPC? Why is it only I'm looking for ways to do it? Design-wise, Is it something wrong with my ideas?

Setting up bidirectional XPC is ridiculously straightforward, although it took me years to realise it was this easy! (-: In the classic XPC workflow, the client sets up a connection like so:

connection.remoteObjectInterface = NSXPCInterface(with: MyClientToServerProtocol.self)

and the server like so:

self.xpcConnection.exportedInterface = NSXPCInterface(with: MyClientToServerProtocol.self)
self.xpcConnection.exportedObject = self

For the server to talk back to the client, you need to do the reverse:

  • Define MyServerToClientProtocol.

  • On the server, set remoteObjectInterface to it.

  • On the client, set exportedInterface to it and exportedObject to an object that implements it.

There are other options though:

  • You can create an anonymous XPC listener on the client and pass its endpoint over to the server. The server can then connect back to the client.

  • You can create an object that implements an XPC compatible protocol on the client and pass that to the server. The server gets a proxy for that object and can message it.

  • NSXPCConnection has NSProgress integration. [Now where is that documented… Oh, here you go…] See the NSXPCConnection support for discrete NSProgress section of Foundation Release Notes for macOS 10.13 and iOS 11.

My "players" aren't an App and its helper-service, but rather independent, resilient system daemons and global agents that need communicate, in all directions.

Endpoints can help with this. They allow you to set up connection graphs more complex than a simple N-clients-to-1-server star. For example, see the XPC Rendezvous section of XPC and App-to-App Communication.

Design-wise, Is it something wrong with my ideas?

No, but you do have to be careful when crossing trust domains. Imagine you have a server that sends a message to a client. If the server is a daemon and the client is an agent, think about what happens if the client simply stops processing requests. The requests back up in the receiver’s Mach port, but that fills up pretty quickly (the standard queue length is 5 IIRC). After that they start stacking up in the server’s address space [1]. This represents a potential denial of service attack on the server.

This problem is particularly bad on iOS because of the way that iOS suspends apps when they’re in the background. Fortunately, there aren’t many third-party folks doing XPC on iOS [2].

One solution is to issue a -scheduleSendBarrierBlock: after your server-to-client request and then refuse to issue any more until the barrier block is called.

Share and Enjoy

Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Assuming you’re using async messaging. If you use -synchronousRemoteObjectProxyWithErrorHandler: then the calling thread blocks, which would be Bad™.

[2] AFAIK the only third-party XPC support on iOS is for file providers.

Faulty, hard-to-understand XPC behavior with remote methods that have a reply-block
 
 
Q