Memory mapped file: "Cannot allocate memory"

Question

Hello,

I am having an hard time figuring out how memory mapped files works under iOS. Suppose that I want to read a big file, let's say 4GB. If I use memory mapped files in READ mode I should be able to get a valid pointer to a file and benefit from the page fault mechanism of virtual memory but apparently iOS can map only up to 2.5GB of data. I run the following test on an iPad Pro A1673 that has 2GB of ram and got as result:

2018-04-06 17:20:12.660888+0200 TestMemory[414:316466] Data address: 0x102224000

2018-04-06 17:20:12.662355+0200 TestMemory[414:316466] Data address: 0x112224000

2018-04-06 17:20:12.663801+0200 TestMemory[414:316466] Data address: 0x12b900000

2018-04-06 17:20:12.665235+0200 TestMemory[414:316466] Data address: 0x13b900000

2018-04-06 17:20:12.666756+0200 TestMemory[414:316466] Data address: 0x14b900000

2018-04-06 17:20:12.668202+0200 TestMemory[414:316466] Data address: 0x15b900000

2018-04-06 17:20:12.669597+0200 TestMemory[414:316466] Data address: 0x16ff24000

2018-04-06 17:20:12.739549+0200 TestMemory[414:316466] Data address: 0x1e0000000

2018-04-06 17:20:12.747604+0200 TestMemory[414:316466] Data address: 0x1f0000000

2018-04-06 17:20:12.749130+0200 TestMemory[414:316466] Data address: 0x200000000

2018-04-06 17:20:12.764753+0200 TestMemory[414:316466] NSData failed: Error Domain=NSCocoaErrorDomain Code=256 "The file “02808580-5D42-43BC-B5AA-628E3682A546” couldn’t be opened." UserInfo={NSFilePath=/var/mobile/Containers/Data/Application/5207275C-5525-47CA-AC4B-2AA07E05C1E9/Documents/02808580-5D42-43BC-B5AA-628E3682A546, NSUnderlyingError=0x1c0455270 {Error Domain=NSPOSIXErrorDomain Code=12 "Cannot allocate memory"}}

2018-04-06 17:20:12.764801+0200 TestMemory[414:316466] Mapped 2684354560

- (NSString*)createFileOfSize:(unsigned long long)size {
    NSString* uuid = [[NSUUID UUID] UUIDString];
    NSString* documentFolderPath = [[[NSFileManager defaultManager] URLsForDirectory:NSDocumentDirectory inDomains:NSUserDomainMask] lastObject].path;
    NSString* filePath = [documentFolderPath stringByAppendingPathComponent:uuid];
    NSFileHandle *fileHandle = [NSFileHandle fileHandleForWritingAtPath:filePath];
    if (fileHandle == nil) {
        [[NSFileManager defaultManager] createFileAtPath:filePath contents:nil attributes:nil];
        fileHandle = [NSFileHandle fileHandleForWritingAtPath:filePath];
    }
    [fileHandle truncateFileAtOffset:size];
    [fileHandle closeFile];
    return filePath;
}
- (void)test {
    const unsigned long long MB = 1 << 20;
    const unsigned long long GB = 1 << 30;
    unsigned long long space = 4 * GB;
    unsigned long long size = 64 * MB;
    unsigned long long fileCount = space / size;
    NSMutableArray* mapped = [NSMutableArray array];
    int i = 0;
    for (i = 0; i < fileCount; i++) {
        NSString* filePath = [self createFileOfSize:size];
        NSError* error = nil;
        NSData* data =  [NSData dataWithContentsOfFile:filePath options:NSDataReadingMappedAlways error:&error];
        if (error) {
            NSLog(@"NSData failed: %@", error);
            break;
        }
        else {
            const void* bytes = [data bytes];
            NSLog(@"Data address: %p", bytes);
            [mapped addObject:data];
        }
    }

    NSLog(@"Mapped %llu", i * size);
}

Any idea of why we have this limitation?

Thanks!

Libe

Kernel

3.2k

Posted by

libe

Reply

Add a Comment

Answer 1

This is not about physical memory but address space. iOS puts limits on the address space available to app processes, meaning your can’t map files beyond a certain size.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Posted by

eskimo

Add a Comment

Answer 2

Thank you for the infomation you provide, that realy make sense to me.

But how can i release the address space?

I use "UIGraphicsBeginImageContextWithOptions", and I got this "CGBitmapContextInfoCreate: unable to allocate 6512640 bytes for bitmap data",so I can't get the image I want, but the memory is just 170M~180M, and App doesn't crash, I just want the image.

Could you give me some advise?

Posted by

Sumbooo

Add a Comment

Answer 3

But how can i release the address space?

By freeing the address space using the deallocator associated with the allocator you used. For example:

If you mapped a file with
```
mmap
```
, you’d free the address space with
```
free
```
.
If you mapped a file with
```
NSData
```
, you’d free the address space by releasing all of the references to the resulting data object.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Posted by

eskimo

Add a Comment

Answer 4

"iOS puts limits on the address space" OMG! The only advantage of 64 bit (vs 32 bit) is/was the support for up to 16 Exa-Byte memory, no system has that much "physical" memory (of course), but at least virtual-address-space should NEVER be limited to physical-address-space.

Isn't it time to fix iOS mistakes?

(Instead of telling people that mapped files are useless)

Posted by

top-master

Add a Comment

Answer 5

While there have been some changes in this space in the 2 years since this thread was last active [1], my initial response on this thread is still accurate in general.

Isn't it time to fix iOS mistakes?

This isn’t a “mistake” but a deliberate design choice. Keep in mind that iOS shares a kernel with macOS, and macOS behaves the way that you consider to be ‘correct’. The limits your see on iOS are, like most things in engineering, a balance between constraints.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

[1] Like the com.apple.developer.kernel.increased-memory-limit entitlement.

Posted by

eskimo

Add a Comment

Answer 6

Just stumbled upon this in an app I'm working on. I set a symbolic breakpoint on malloc_error_break which ends up getting hit on the following line in my code:

   CGImageRef cgImage = CGImageSourceCreateThumbnailAtIndex(imageSource,

                                                             0,

                                                             (CFDictionaryRef)options);

The imageSource above is created via a CGImageSourceCreateWithURL function call.

I use this to load images in a table view. All works well, but after scrolling the table view for awhile it appears I hit the address space limit.

I don't use mmap directly and after I get CGImage's from above I covert them to UIImage and release it:

CFRelease(imageSource);
CFRelease(imageProperties);
 if (cgImage != NULL)

    {
        UIImage *uiIMage = [[UIImage alloc]initWithCGImage:cgImage];

        CGImageRelease(cgImage);

        return uiIMage;

    }

The returned UIImages are held in a NSCache object. Is it possible that CGImageSourceCreateThumbnailAtIndex is causing me to hit the address space limit? I don't appear to have a memory leak here so I'm not sure why my address space limit would keep accumulating as I continue scrolling the table view. The image source and CGImage are released after converting to a UIImage.

Posted by

Macho Man Randy Savage

_> I don't appear to have a memory leak here so I'm not sure why my address space limit would keep accumulating as I continue scrolling the table view. _

Looks like the returned UIImages (created with the CGImages) need to be released. They UIImages are stored in a NSCache. Unfortunately NSCache is not automatically removing objects when malloc_error_break is hit. The operating system isn't sending my app a memory warning either. I was under the impression NSCache would remove objects at the appropriate time based on certain OS conditions (like this one) automatically. I guess I should be able to resolve this by setting a count limit on the cache and/or manually emptying the cache when CGImageSourceCreateThumbnailAtIndex returns NULL. After manually emptying the cache of UIImages I'm able to load more without getting another cannot allocate memory error.

—
Macho Man Randy Savage

Add a Comment

Answer 7

I was under the impression NSCache would remove objects at the appropriate time based on certain OS conditions … automatically.

This is a recurring theme on this thread, namely that logical memory (that is, your address space) and physical memory are different things. NSCache removes items when physical memory gets short. It does not monitor your address space.

Share and Enjoy
—
Quinn “The Eskimo!” @ Developer Technical Support @ Apple
let myEmail = "eskimo" + "1" + "@" + "apple.com"

Posted by

eskimo

Add a Comment

Answer 8

Cool got it. Thanks for the reply.

I'm guessing the answer tot his question is no but I'll ask anyway. Is there any API on iOS to detect when an app's address space limit is getting close to being hit? That is, to be notified preemptively before hitting the limit so I can free some, rather than waiting for a ENOMEM to occur..and then freeing up address space?

Posted by

Macho Man Randy Savage

Add a Comment

Memory mapped file: "Cannot allocate memory"

Replies