XPC, memory allocation, and much confusion

In the old days, I would have asked “where are the NSNumber and NSString that you create in the while loop being released?”, i.e. is there an autorelease pool, and where is it being drained. But today there are more layers of magic; I don’t know if an infinite loop should be expected to be memory-friendly or not.

Oh that NSNumber was holdover from trying to debug a different thing, I kinda forgot how asynchronous queues work for a bit, ha ha.

Using leaks and the command line doesn't find anything particularly horrible either.

Did you mean to use @synchronized (self)? Or were you aiming for @autoreleasepool? Because the latter makes sense in this context but former does not.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

No, I meant @synchronized for thread-safety. It's wrapped in an outer @autoreleasepool via main. But ARC should be freeing things. And it is, honestly, but the memory use I see in top or Activity Monitor is different depending on whether I'm running it under Instruments or not. And that's the part that really confuses me.

It's wrapped in an outer @autoreleasepool via main.

Huh? But it’s running a tight loop within that code snippet your posted, which means it never gets back to main to drain the pool.

But ARC should be freeing things.

In your Swift code, sure, but there’s a bunch of Objective-C framework underlying that and there are limits to how much ARC to prevent autorelease pool traffic in that code.

I recommend you have a read of Objective-C Memory Management for Swift Programmers. Much of this will be a refresher for you, but it’s still important.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Also, if I change it to:

        while (self.stopRun == NO) {
            @autoreleasepool {
                NSNumber *objNum;
                @synchronized (self) {
                    objNum = [NSNumber numberWithUnsignedLongLong:self.count++];
                }
                NSString *objName = [NSString stringWithFormat:@"Object %@", objNum];
                TestObject *to = [TestObject name:objName];
                [proxy logDataWithEntry:to];
#endif
            }
        }

then it still gets up to 500mbytes (according to Activity Monitor) within just a few seconds. So that @autoreleasepool doesn't seem to be doing anything (since it's inside the loop).

OK, it’s good to rule that out.

Looking back at your original problems statement, I’m not surprised by the memory growth in the asynchronous case. If your XPC service runs slower than your app, messages will pile up within the Mach message port associated with the connection. However, that’s limited to just a few messages — IIRC the default is 5 — so then messages will start piling up in your app’s address space. That’s fine if you’re generating messages in short burst, but it’ll cause this problem if you generate an unbounded number of messages.

Folks usually encounter the reverse of this, where they see unbounded memory growth in the server because the client stops receiving. That can be used as a DoS attack. On the client side, you’re just DoSing yourself (-:

Try this: Every 100 messages, schedule a barrier block (using -scheduleSendBarrierBlock:) and then don’t send any more messages until the completion handler is called. Does that limit the memory growth?

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

I changed the code to

        int counter = 0;
        while (self.stopRun == NO) {
            NSNumber *objNum;
            @synchronized (self) {
                objNum = [NSNumber numberWithUnsignedLongLong:self.count++];
            }
            NSString *objName = [NSString stringWithFormat:@"Object %@", objNum];
#ifdef USE_COMPLETION
            __block TestObject *to = [TestObject name:objName];
            [proxy logDataWithEntry:to completion:^(NSString *str) {
                to = nil;
            }];
#else
            TestObject *to = [TestObject name:objName];
            [proxy logDataWithEntry:to];
            }
#endif
            if ((counter++ & 0x7ff) == 0) {
                [self.connection scheduleSendBarrierBlock:^{
                    return;
                }];
        }
    });

and there is no change in the Activity Monitor-reported memory usage.

(Amusingly, this is much worse on Apple Silicon, because it is so much faster.)

Your code isn’t waiting for the barrier completion handler to be called, so the barrier has no effect.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Ha, ok, I thought it was a synchronous call. 😄

I used a dispatch semaphore, and the memory growth is significantly slower, although it still happens.

the memory growth is significantly slower

At that point you need to start looking at a memory graph to see what’s taking up memory and who’s referencing it.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Ok, I've been experimenting off and on. My loop is this (some #if's and stuff removed); proxy is just a normal remoteObjectProxy on the XPC connection, with an error handler just for logging:

        int counter = 0;
        while (self.stopRun == NO) {
            NSNumber *objNum;
            size_t dataLength = arc4random() % 16384;
            void *dataBuffer = malloc(dataLength);
            NSData *data = nil;
            @synchronized (self) {
                objNum = [NSNumber numberWithUnsignedLongLong:self.count++];
            }
            NSString *objName = [NSString stringWithFormat:@"Object %@", objNum];
            if (dataBuffer != NULL) {
                arc4random_buf(dataBuffer, dataLength);
                data = [NSData dataWithBytesNoCopy:dataBuffer length:dataLength freeWhenDone:YES];
            }
            TestObject *to = [TestObject name:objName data:data];
            [proxy logDataWithEntry:to];
            if ((counter++ & 0x7ff) == 0) {
                dispatch_semaphore_t sempaphore = dispatch_semaphore_create(0);
                [self.connection scheduleSendBarrierBlock:^{
                    printf("In schedule block\n");
                    dispatch_semaphore_signal(sempaphore);
                    [self updateBandwidth];
                    return;
                }];
                dispatch_semaphore_wait(sempaphore, DISPATCH_TIME_FOREVER);
                printf("Done with lock");
            }

        }
    });

Some screenshots from Instruments -- the allocation graph during the runtime, and then the source code with its annotation for allocation size

In the allocation graph, once it exits out of the loop, the memory use goes back to (delightfully!) 0. Until then, however, it doesn't seem to be dong any releasing of memory. The output of both top and Activity Monitor match the allocation size and behavior.

Until! If I put an autorelease pool inside the entire contents of the loop, then... it grows, albeit much more slowly. It also runs much faster, which indicates (to me) that Instruments is interfering with it enough to change its behavior.

So, in summary: I am still deeply confused about how ARC is reaping when I use it with XPC. 😄

(Our actual application gets into hundreds of mbytes fairly quickly; I've already tried adding in a barrier call.)