Sandboxed Mac app can not accept more than ~5,000 files total

Maybe try logging when you start and stop access to see if there's some edge case where it doesn't properly balance?

so I'm not crazy - this approach is supposed to work, right? To be able to have more than 5,000 files dragged to the app (in total, not necessarily all at once - that's another story entirely...)

If you are absolutely certain you are balancing the security scoped start/stop calls, I'd file a bug.

5,000 in any case seems like a whole lot of bookmarks. Why don't you just ask the user for permission to the home directory on first launch, and make 1 bookmark?

"Why don't you just ask the user for permission to the home directory on first launch"

That seems like something Apple would reject pretty quickly.

Also, usually, there's less than 5,000 files inside the app at any given time, so the bookmarks are obviously cleaned up when the files are no longer needed inside my app.

So the flow would be something like this:

- Add a file to Yoink (bookmark created)

- Move the file out of Yoink (file disappears from Yoink, bookmark discarded)

- Add two files to Yoink (bookmarks created)

- Move the files out of Yoink (files disappear from Yoink, bookmarks discarded)

...

It takes a while until you get to the 5,000 file limit, but when you do, there's nothing that can be done aside from a restart to make the app work again.

I tested it by 10x dragging 500 different files to the app. 10 times, it works. After that, maybe some 200 more files can be added, but after that, the sandbox can not consume the extension or something like that (it's the error I get).

My assumption was incorrect. I found what the problem was.

To follow files around (see if they were renamed or moved to the Trash), I use dispatch_sources, and to create them, I have to open() the filepath, which is where the limitation is coming from.

Sorry about that.

>"Why don't you just ask the user for permission to the home directory on first launch"

>That seems like something Apple would reject pretty quickly.

I have a couple of apps downloaded from the Mac App Store that ask for this. They could reject you I guess, but that seems like a better approach than having to potentially create thousands of bookmarks.

So the user adds a file, you create a bookmark for it and save it to disk. On relaunch, you attempt to go through all 5000 bookmarks and and start accessing the security scoped resources on them so you can show the files in your UI. Are all 5000 files on screen at once? If your UI has some sort of 'paging' to it, you could ensure that you are only accessing X number of bookmarks at a given time.

Not sure if that is "the limit", meaning you can't have more than 5000 bookmarks or you can't access more than 5000 bookmarks at a given time...someone with the fruit badge may be able to give you an answer. I don't have your app, so I don't know how it works exactly, but I've heard of it (congrats). Seems impractical that a user would want to drag 5000 files there, but I guess it could happen (and since you are asking, I'm assuming it did).

"On relaunch, you attempt to go through all 5000 bookmarks and and start accessing the security scoped resources on them so you can show the files in your UI."

No, I only start accessing those resources when needed, so I don't access all 5000 at the launch at once, only when scrolling through for creating QL previews, update their filenames, etc. So it is pretty efficient.

"but I've heard of it (congrats)"

Thank you 🙂

"Seems impractical that a user would want to drag 5000 files there, but I guess it could happen (and since you are asking, I'm assuming it did)."

For me, it wasn't really about having 5,000 files in Yoink at once, but that over the course of the app's session, those 5,000 items could be reached altogether (over several days or weeks) and then the app would stop working (which, before I make the switch to bookmarks with the next update, is happening).

-

But again, I thought about it the wrong way. The actual culprit was not the amount of bookmarks or NSURLs added to the sandbox, but the dispatch_source for vnodes I create for each NSURL added so I can track if they're renamed or are moved to the Trash/deleted. I'll have to find a more efficient way for that.

Discarding the dispatch_sources and polling the NSURL's filePath (to see if it's in the Trash) would be an alternative, but I really, really do not want to do that.

I tried, instead of watching the files themselves if they're moved to the trash, watching the Trash and, if the Trash changes, see if it's one of Yoink's files that's in there.

It works, but only for the boot hard drive. It doesn't work for external disks' trashes, and I do not have permission to watch those inside the sandbox (I tried).

Generally speaking, it's a limitation only if 5,000 files are actually inside Yoink at one time. Then new files added fail. It's an edge-case, but it's still something I'd like to solve to be on the safe side.

Thank you for your feedback, Macho Man 😉

Two things:

• If a file is moved, renamed or deleted, that’ll trigger kqueue activity on the (immediate) parent directory. So if the files you’re trying to monitor exist in a small set of directories, you may be able to get big wins by monitoring the parents.

• If you want to monitor a large part of the file system hierarchy, FSEvents is your friend, at least in general. The mechanics of getting FSEvents to do the right thing in a sandboxed app may be tricky.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Thank you, Quinn, for your help here.

I ended up revamping the whole system so it uses less resources.

What I've been doing up until now is create a dispatch_source for every NSURL added to Yoink. Obviously, at some point, resources run out if sufficient NSURLs are inside Yoink at one given time.

What I'm doing now is this:

I only create dispatch_sources for NSURLs that are currently visible in Yoink (they're displayed as icons in an NSTableView). This saves a ton of resources.

So file names are only updated for currently visible files. Titles of files that are currently not in view are updated once they're scrolled to.

For deletion, I use FSEvents to watch all Trashes on the system (which is possible using FSEvents, but not dispatch_sources, interestingly) so I can update accordingly, even for files that are currently not in view in Yoink.

I think that's the most efficient solution to this problem.

Do FSEvents work with files in iCloud? I've had problems with this in one of my apps and instead use file coordination.

Do FSEvents work with files in iCloud?

Honestly, I don’t know how these two technologies interact. If you need a definitive answer here, I recommend you open a DTS tech support incident so that me, or one of my colleagues, can dig into it.

Share and Enjoy
—
Quinn “The Eskimo!”
Apple Developer Relations, Developer Technical Support, Core OS/Hardware

let myEmail = "eskimo" + "1" + "@apple.com"

Cool. I might have to do that. I remember trying awhile ago and it seemed that you just didn't get callbacks if the file you were watching was ubiquitous, though I guess it is possible that there was some 'gotcha' I was missing. Ended up going with file coordination in case a user did select a file in icloud, though it seems with file coordination you get notified for changes your own process makes, not sure if there is an easy way for me to opt out of that like how FSEvents gives you the kFSEventStreamCreateFlagIgnoreSelf