How to detect an auto-mounting directory and wait for it to get mounted?

Ironically, the answer is found in a 10 year old post by myself (https://stackoverflow.com/a/28663838/43615).

Basically:

• Before diving into the dir, get the ATTR_DIR_MOUNTSTATUS attribute and check the DIR_MNTSTATUS_TRIGGER flag. If it's set, it means the dir is currently not mounted but will trigger a mount once it gets read. So, if the flag is set, I only have to read the dir's contents (which will report back as being empty), and then I can immediately read again, and this time the mount will be there. I thought that there would be a delay, but apparently this happens synchronously.
• Alternatively, if I do not want to check the flag beforehand, I could also, if I find an empty dir, check if the dir is a mountpoint (DIR_MNTSTATUS_MNTPOINT flag) and if it is, re-read the dir.

Ironically, the answer is found in a 10 year old post by myself

As an additional irony, I'd just finished writing all this up when I saw your own post. Filling in some details which may be helpful...

(This was a bit surprising to me - I'd have thought that Windows or SMB would hide the fact that the share has sub-volumes, and simply show them as directories - and that's what it did in older macOS versions indeed, e.g. in High Sierra. But in Sequoia, these sub-volumes on the Windows side are mirrored on the Mac side, and they behave accordingly)

This is actually done by something called "DFS" (Distributed File System), which is actually designed to allow an organization to present a unified hierarchy across multiple servers as well as shares. The broader protocol obviously handles much more that this specific volume case and I suspect they used it here because it handles standard edge cases better* and the work had already been done.

*For example, exposing the secondary volume as a separate share means that if the remote volume disappears while being browsed, the client receives that operation as the share itself becoming unavailable instead of as a chunk of the hierarchy just "vanishing".

How do I do that? I.e. how do I tell that a dir is actually a auto-mount point and how do I wait for it to get auto-mounted before I continue to browse its contents?

That is a great question that I'm not sure of the answer to. From looking at our code, I don't think the Finder itself is actually doing this, but that it's actually being done by the SMB client itself using something called a "trigger vnode". If you're curious, you can actually see the code here.

SO, just to confirm what you're saying here:

Only once I try to enter the empty dir, listing its contents, which will return no items, an auto-mount action will get triggered, which will add the mountpoint at the path.

The problem here is that while your code is triggering the automount, the problem is that you don't actually "know" that it will occur or when it's done, so you can't go back again and "check".

Assuming that's correct, then I think you could detect this case using "NSURLIsVolumeKey" or "kCFURLIsVolumeKey", both of which correspond to "ATTR_DIR_MOUNTSTATUS" of getattrlist. That's typically used to denote actual mount points, however, it looks like it was also used to mark trigger nodes (the case here).

Do I need to use the old Disk Arbitration functions for this, somehow?

First off, Disk Arb is well aged, like a fine wine, not "old". Having been at this long enough to remember using the older, private, API (which was MISERAGBLE) and then being thrilled when the public API was introduced in 10.4, I have to defend it's honor.

But yes, I think this is what I would do:

I have used the DA framework in the part to prevent auto-mounting, so I imagine I could hook into that handler, and if I get a callback for a mount operation, I could then queue the newly mounted volume for scanning.

The issue here isn't just detecting these nodes at the point you're scanning, it's thinking through how you want to handle broader edge cases. The basic case here is "you scanned a folder which triggered an automount", but the other cases are things like:

• After your scan had "left" the directory, the user attached a new remote volume and then browsed into that directory through the Finder.

• The user (or some other app) manually mounted a new volume within the hierarchy you're scanning.

• Even in the "basic" case (you navigated in a particular directory which triggered and automount), the time that automount could take to finish is WILDLY unpredictable. The best case (local volume on a fast LAN server) is pretty "fast", but the worst case (totally different server with a very slow/high latency connection) is basically unbounded.

The broader issue here is that relying on DiskArb lets you avoid thinking about all those specific case. DiskArb will tell you if/when any mount occurs, which you can then address however you choose.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

As it currently seems to me that auto-mounting these sub-volumes on the Windows share is immediate, I'll stay with the method of simply re-reading the once-empty dir if I realize it's a mountpoint.

Again, the issue here is that you're focusing on the specific edge where the volume is already fully mounted on the Windows side. It's fine if that's all you care about, but there are other edge cases where the mount can happen well "after" the initial browse occurs.

Also, what API are you iterating the hierarchy with? One thing that could get weird here is if you're iterating very large subvolumes, you could end up getting new volumes mounting while you're still iterating a different subvolume. I honestly don't know how well our code will handle it, but I suspect it will just miss the new volume entirely*.

*Our directory enumeration APIs are recursive, so unless the mountpoint directory is "large" it will only fetch the volume once, when it first touches the directory, which will the "miss" later mounts.

But just in case - do you know if there's a DA callback that'll notify me immediately once the mountpoint gets triggered?

No, there isn't. The issue here is the "direction" the trigger vnode is going- the trigger vnode is about the VFS system telling the file system that it's being navigated "into", NOT telling user space about that node. What actually happens as a result of that trigger depend on the state of the distant file system.

I only know DARegisterDiskMountApprovalCallback, and that comes in only after the new volume has been inspected a bit, I believe.

A few different answer here:

1. As far as the broader system is concerned, there's not such thing as an "unmounted volume". That is, the definition of "a volume" is something like "a file system object that's been attached to the unified hierarchy". The issue here is that "volume" and "mounting" are inherently entangled with each other, as the only way to know whether or a given "thing" (like a device node or a share path) will ACTUALLY mount is to in fact "mount that object". DiskArb does have access to some information prior to the mount occurring, but that all that information comes form the originating mount "source". There isn't really any earlier "point" than that, at least not in the general case.

2. I haven't looked closely at exactly how this particular case plays out. I expect/assuming the mount flows through DiskArb (and could be blocked by an approval callback), but is possible to bypass DiskArb (basically, by calling mount/mount_<type>). If that occurs, I think you'd get be notified that the mount occurred but the approval callback would not occur.

3. It MIGHT be possible "know" (possibly through kqueue?) about the incoming mount "before" DiskArb, however, I don't think there would be any meaningful benefit. Basically, with a lot of work you could get the same event DiskArb is responding to, at which point you could do a lot more work to retrieve the same information DiskArb helpfully hands you. Having gone to all the work, you'd have basically recreated DiskArb except that it wouldn't be able to do things like blocking automounts*.

*DiskArb is also what ultimately calls "mount" during the automount process, so the approval callback are basically DiskArb asking clients "is everyone ok with my calling mount?". When an automount is "blocked", what actually happened is that DiskArb simply "chooses" not to call mount.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Kevin,

I'm using both a simple recursive iteration (contentsOfDirectoryAtURL) and enumeratorAtURL. In both cases, before I read the dir contents, I check if it's a volume with a trigger flag - if it is, I call opendir, readdir, closedir before I invoke one of the above FSFileManager methods again.

That way, at least in my use case, the sub vols get mounted right away by the opendir/readdir, and the next filemanager call provides me with the new volume's contents.

I only have one user out of 10000s who has reported this issue so far, for the >15 years I have my app in the public, so I consider this a "quick'n dirty" fix that works well enough for now.

I've noted down your comments, though, in case I have to work on this again.

I have a concern about performance, though: I like to be as fast as possible with the iteration, especially if I only need to look at file names. So, when calling enumeratorAtURL, I fetch no extra URL properties, and detect directories by called url.hasDirectoryPath, which turns out to be faster than checking NSURLIsDirectoryKe. If it's a dir, then I call getattrlist to get the mountpoint values. (See my "DirScanner" test project at https://files.tempel.org/Various/DirScanner.zip, which also employs this technique to avoid diving into other volumes, e.g. when browsing from "/" down.) Any idea if there's a better way?

(I also learned recently from Jim Luther that it's much faster to use the C-string "url.fileSystemRepresentation" path instead of first getting url.path and then getting its UTF8String because that skips come costly internal conversions. Same for creating URLs when the name/path comes from POSIX/BSD functions. I've updated my DirScanner code accordingly.)

I've noted down your comments, though, in case I have to work on this again.

I have a concern about performance, though: I like to be as fast as possible with the iteration, especially if I only need to look at file names.

So, my first and biggest warning here is that this kind of performance testing is excruciatingly tricky and error prone. There are so many details and edge cases that distort the results, all of which make it very hard to be entirely confident in any conclusion.

The next thing to understand here is that the fundamental performance issues and the corresponding solutions are EXACTLY the same across basically "all" of the APIs your looking at. More specifically, going back to your article here:

https://blog.tempel.org/2019/04/dir-read-performance.html

...you reference basically 3 APIs. Here are the APIs and their underlying implementation:

1. contentsOfDirectoryAtURL-> Implemented as a wrapper around enumeratorAtURL, which is wrapper around getattrlistbulk.

2. opendir()/readdir_r()-> Wrapper around getdirentries. See implementation here.

3. fts_open() && fts_read()-> Wrapper around getattrlistbulk. See implementation here.

In other words, what you're actually looking at are different implementation built on top of VERY similar syscalls. More specifically:

• getattrlistbulk-> VNOP_GETATTRLISTBULK

abstract Call down to get file attributes for many files in a directory at once.
discussion VNOP_GETATTRLISTBULK() packs a buffer  with file attributes, as if the results of many "getattrlist" calls.

• getdirentries-> VNOP_READDIR

abstract Call down to a filesystem to enumerate directory entries.
discussion VNOP_READDIR() packs a buffer with "struct dirent" directory entry representations as described
by the "getdirentries" manual page.

The key point here is that both of those use the same approach of cramming multiple entries into buffer so that they can return more data at once. The reason for this is that, all other factors being equal, the primary performance factor here is the total number of syscalls. That dynamic is what explains this:

For readdir(), fetching additional attributes (through lstat()) turns it into the slowest method.

It's starts out the fastest because, in the basic case, it's returning less data per object, so it requires fewer syscalls. However, that benefit is completely eliminated if you need ANY other data, since you then start making additional syscalls.

Moving into all the details that make this so much fun, let me actually start here:

So, when calling enumeratorAtURL, I fetch no extra URL properties, and detect directories by called url.hasDirectoryPath, which turns out to be faster than checking NSURLIsDirectoryKey.

That's extremely weird, which led me to do my own checking. As it turns out, the real mystery here isn't why hasDirectoryPath is fast, it's why adding ANY keys makes things so slow. Architecturally, the performance here should basically be "identical", as the ONLY reason "hasDirectoryPath" can work at all is that the enumeratorAtURL "told" the URL it was a directory at the point it was created. I'm seeing a similar issue with NSURLIsVolumeKey, which should also have been retrieved during bulk iteration and should have been relatively "free" to retrieve.

That leads me to here:

(See my "DirScanner" test project at https://files.tempel.org/Various/DirScanner.zip, which also employs this technique to avoid diving into other volumes, e.g. when browsing from "/" down.)

After a bit of experimentation with your code, the only conclusion I can come to is that there's a serious issue in enumeratorAtURL. The expected behavior here is that there should only be a VERY weak connection between specific attribute requested and overall performance. That's exactly what your getattrlistbulk test run showed:

getattrlistbulk():
Run 1:  4.51s, scanned: 296647, found: 0, size: 0 (not determined)
Run 2:  4.59s, scanned: 296647, found: 0, size: 0 (not determined)
Run 3:  4.57s, scanned: 296647, found: 0, size: 440110390101
 
fts() was worse, but not entirely unreasonable:
 
Run 1:  5.24s, scanned: 296647, found: 0, size: 0 (not determined)
Run 2:  5.01s, scanned: 296647, found: 0, size: 0 (not determined)
Run 3:  7.86s, scanned: 296647, found: 0, size: 440110390101
 
Unfortunately, enumeratorAtURL was 2x slower:
 
Run 1:  4.75s, scanned: 296647, found: 0, size: 0 (not determined)
Run 2:  4.68s, scanned: 296647, found: 0, size: 0 (not determined)
Run 3:  9.28s, scanned: 296647, found: 0, size: 440110390101
 

I don't know what's gone wrong but the only explanation I have is that there's a serious issue in our current implementation and the resource cache simply isn't working properly. Please file a bug on this and send me the bug number once it's filed.

Any idea if there's a better way?

Well, to be honest, given how important this performance is to your product, I think my advice would probably be to build and optimize entirely on getattrlistbulk. It's the API that's underneath "everything" else, giving you the most control and opportunity to optimize. More to the point, it's the API "underneath" every other API, so any performance difference between your implementation and other APIs is a matter of "doing what they do".

In terms of large performance gains, the other thing you might want to look at here is using multiple threads and making multiple calls into getattrlistbulk simultaneously. A few reasons for that:

• On network file systems, the VAST majority of performance loss is simply waiting on the network so the more request your sending, the faster you'll finish.

• Historically, you wanted to avoid parallel file system access because it made I/O more random (which spinning disks don't like) and the file system's locking architectures meant it often provided less benefit than you might think. Neither of those are really true anymore (at least not to the same degree).

I don't know what the "right" level of parallelism will be or how big the gains will be, but my intuitions is that they'll be significantly larger than any gain you'll see from tweaking your API usage.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Kevin, I appreciate your involvement in this, and your willingness to dive deeper into this for my (and hopefully others') education

Some of the things you explain about performance were already clear to me (I used to write file systems and a simple operating system in the long past).

Some were not clear, e.g. I have assumed that maybe some of the NS based calls were happening in a kernel space where making calls to the lower APIs would be less wasteful, or that NSURLs, which these calls return would be pooled at a more efficient level (where I only recently learned from Jim L that it was for me using NSString for paths where the fileSystemRep was the key to keeping it faster).

I also agree that some of the properties, such as "NSURLIsVolumeKey" shouldn't be costly because they should not require another call into the BSD/POSIX APIs as I'd have thought that the information is already present at the level above the VFS. And that's especially true for the NSURLIsDirectoryKey.

Architecturally, the performance here should basically be "identical", as the ONLY reason "hasDirectoryPath" can work at all is that the enumeratorAtURL "told" the URL it was a directory at the point it was created

Exactly my thought - I concluded that from logic, without looking at the source code. You're aware that NSURLs append a "/" suffix to dirs, while paths do not, right? So, I suspect that hasDirectoryPath simply checks for that trailing "/", hence it being so fast.

BTW: If you time the "find" tool, you'll see that it can be almost twice as fast as my DirScanner tests. Which surprised me. I still wonder if NSStrings conversions play a role in that, which the find tool doesn't have to worry about.

Huh, well, okay. I was already expecting that anyway, but never bothered to report it as a problem because I didn't think anyone would bother with it. I had reported, for instance, an issue with searchfs(), which has become much much slower in APFS over HFS+ (like, 6 times slower for the same amount of files, when it's an entire startup volume), and this was not addressed, as far as I can see.

And if we're talking about performance issues on the FS, the worst offender is Finder and its support frameworks. I have once run a trace of calls (e.g. with fs_usage) and I found that there's a LOT of repeat calls for the same attributes, which explains why nowadays browsing the Applications folder, especially over a network volume, is so awfully slow (it's mainly about diving into the bundles). THAT needs addressing, and it's so obviously bad - it should be obvious to anyone using a Mac for a few hours, especially if they have some experience how fast it should be (and indeed used to be!). As long as that isn't tackled, I wonder why I should make an effort to report these rather minor bottlenecks. But then again, maybe, if enumeratorAtURL is used by Finder a lot (can't dtrace anymore, can we?) and and this gets improved, maybe it'll automagically also improve browsing in Finder all over, significantly? If you think so, I'm happy to make the effort. (Feel free to email me directly if you have private comments, I don't want to start a flame war here, but rather really care about this getting better, if I can make a difference. Incidentally, I had applied for a job a few times working on FS improvement in the past but nothing come of it.)

Yes, that's indeed on my wish list, after seeing how the find tool can be so much faster (my app has now a mode where it calls the find tool and then my app just looks up the paths it reports back, and that's twice as fast). But then, I also want to improve the UI a lot and add a caching of the FS by tracking all changes and keeping a shadow directory, and that'll benefit my users more in the long run). Only so much a single programmer can accomplish :)

using multiple threads and making multiple calls into getattrlistbulk simultaneously

Yes, that's on my list as well, and yes, especially for network vols. Also, I had already planned this before SSDs became common, and I'd thought to identify which drives where on the same HDD, and only run concurrent searches on separate HDDs, to avoid excessive seeking. Though, I still have to experiment with this, because I'm not sure if the VFS would queue calls on a global level - but in recent years I got hints that this would not be the case, and I'd be able to have multiple FS calls run concurrently if they're on independent file systems, and you seem to indicate the same. Which is promising.

However, most users of my app are simple "home" users who only search on their startup disk, and there's little to parallellize there. I had, for instance, considered to pre-cache the locked system volume, but then, the time for searching that volume is fairly quick compared to the /System/Volumes/Data volume, so caching the former doesn't gain much in the overall search time once the user has accumulated lots of files (which happens eventually).

Again, thank you for not only trying to answer my questions but also exploring solutions.

I also agree that some of the properties, such as "NSURLIsVolumeKey" shouldn't be costly because they should not require another call into the BSD/POSIX APIs as I'd have thought that the information is already present at the level above the VFS. And that's especially true for the NSURLIsDirectoryKey.

No, it doesn't work that way. More specifically, there are basically two levels where "caching" can occur:

1. At the system level, where all processes have a coherent view of what they believe the file systems current state is.

2. At the process level, where processes have whatever data they last received from #1.

The key issue here is that the system level cache IS the VFS layer. The POSIX APIs operate as the bridge between those two layers and that means they minimize any kind of caching (since that would only complicate the bridge).

That is, the ony level "above" the VFS layer IS NSURL. It should be caching the data it's getting from getattrlistbulk (that's one of it's jobs) and it shouldn't be seeing the large performance drop it's seeing.

Exactly my thought - I concluded that from logic, without looking at the source code. You're aware that NSURLs append a "/" suffix to dirs, while paths do not, right? So, I suspect that hasDirectoryPath simply checks for that trailing "/", hence it being so fast.

No, that's not how it works. As the most obvious issue, keep in mind that URLs support three different path seperator's (POSIX, HFS, Windows) and that file reference URLs don't contain a "path" in the normal sense. The actual answer here is that "hasDirectoryPath" is simply returning what it was "told". That is, every NSURL/CFURLReg file path initializer either has an "isDirectory" argument or is using a type/process that will "answer" the question*.

*For example, CFURLCreateFromFSRef (FSRef's embed this in their structure) or URLByResolvingAliasFileAtURL / URLByResolvingBookmarkData (the VFS operations required to resolve the reference also tell the NSURL if it's a directory).

What CFURLHasDirectoryPath actually does is check a bitfield which is use to store the value passed in through isDirectory (or configured by one of the other initializers). The enumerator had the relevant path here and I don't know why it didn't cache it and/or why caching it causes such a large performance drop.

BTW: If you time the "find" tool, you'll see that it can be almost twice as fast as my DirScanner tests. Which surprised me.

FYI, "find" is opensource and seems to be a relatively straightforward implementation built on fts.

I still wonder if NSStrings conversions play a role in that, which the find tool doesn't have to worry about.

I think the big point to understand here is that the basic performance metric here is essentially syscalls per second. You can optimize that by reducing the time required to process the data returned by each syscall, but you could also optimize this by moving the processing off of the thread making syscall. Similarly, you can also do the same thing by using more thread to issue more syscalls.

This got me curious enough that I ended up doing a very quick and dirty pass at a "brute force" multithread of your getattrlistbulk scan. Basically:

• Modify "scan" so that every call allocates and frees it's own buffer.

• Create an NSOperationQueue and then use "maxConcurrentOperationCount" to control how many calls happen in parallel.

• Use "addOperationWithBlock" to recursively call "scan" again instead of directly recursing.

Here are the results I got in ~2 million files (APFS is fabulous for this sort of testing):

maxConcurrentOperationCount = 16:
Run 3: 16.08s, scanned: 2117642, found: 0, size: 3108295888604
 
Original Implementation:
Run 3: 39.87s, scanned: 2135507, found: 0, size: 3112826775276
 

Now, this was with basically no effort at any sort of refinement or optimization. It's allocating/freeing memory far more often than would be ideal the operation overhead is relatively high. The scanned/size numbers are also being thrown off because of threads stepping on each others data.

In a real implementation, I think you'd probably want a fixed set of threads calling "getattrlistbulk", another set of threads processing the data each call produced, and then system the pushed directory back to the syscall thread.

Huh, well, okay. I was already expecting that anyway, but never bothered to report it as a problem because I didn't think anyone would bother with it.

Bugs really do matter, particularly for an issue like this. It can sometimes take longer than anyone would like and the situation is sometimes more complicated than it seems but they really do matter.

I had reported, for instance, an issue with searchfs(), which has become much much slower in APFS over HFS+ (like, 6 times slower for the same amount of files, when it's an entire startup volume), and this was not addressed, as far as I can see.

I actually tracked this bug down and I have a question about your testing. Were you always testing on "freshly" mounted volumes (meaning, umount volume-> mount volume-> searchfs)? Or did you also test on volumes under normal use and repeatedly running searchfs?

There were some significant issues (primarily involving compressed files) but the really big performance issue was specifically tied to newly mounted volumes. The issue here when compared to HFS+ is that the HFS+ catalog file is stored as a "basically" contiguous (it can be fragmented, but even on very large volume the total fragment count tends to be fairly small) chunk of data which the system ends up essentially reading into memory at or near boot time.

APFS simply does NOT work that way for very sensible reasons (which I won't try to list), but that also means that searchfs IMMEDIATELY after boot takes a large performance hit, as the system is basically forced to stream in the entire catalog set through a long series of chained reads. Various techniques were considered to improve that, but all of them had significant downsides.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Wow, Kevin. That's excellent support, thank you so much for making the effort!

I had no idea that using concurrency in file system calls would have any such effect. You proved me wrong.

In a real implementation, I think you'd probably want a fixed set of threads calling "getattrlistbulk", another set of threads processing the data each call produced, and then system the pushed directory back to the syscall thread.

Already doing all the processing past the initial gathering of the directory contents in a separate thread, but that's clearly far from what's possible.

I actually tracked this bug down and I have a question about your testing. Were you always testing on "freshly" mounted volumes (meaning, umount volume-> mount volume-> searchfs)? Or did you also test on volumes under normal use and repeatedly running searchfs?

The use case of my app is to mainly run searches on the entire startup volume. So that's what I focused on with my tests.

I try to be aware of caching effects (I believe some OS versions can use a of the available memory for caching information, but never learned about the details - all info on this I found was fairly vague). What I did, for instance, was to run searchfs on a running High Sierra system using HFS+, then cloned the system and let it convert to APFS, booted from that and ran the same tests. Repeatedly, of course. Later ran the tests on upgraded macOS systems, showing no significant improvement.

The thing that confuses me most about searchfs performance on APFS is that when I search for something that's not findable (e.g. a unique name) both with searchfs and with a conventional method that eventually goes thru getatrrlistbulk, it didn't make much of a difference - even though the number of syscalls should be immensely different (and thereby contributing a lot to the overall time): searchfs should be a single syscalls, whereas the other method will be in the 100000s, right? That is why it could be so fast on HFS+, because all the searching happened right in the VFS, in a tight loop.

Am I misunderstanding how it's working on APFS? I always assumed it was because of the dual-tree nature of APFS, where one cannot simply look at the one tree containing the dir records, looking for matching names, but must also look up the corresponding entries in the other tree in order to figure out which volume the dir entries belong to, and this would be the bottleneck (though one could argue that simply first blindly looking for matches in the dir rec tree and only then verifying the other tree if a match is found may be faster than the other way around that I suspect it's happening now). You understand what I mean? searchfs may be optimized to either first sort out the right dir recs by making sure they belong to the target volume before checking the dir properties, or vice versa, and maybe the other way around would be usually faster (assuming one seeks to find a small set of hits). Of course, that would require more work on this code for very few gain (hardly anyone is using this anyway, I believe).

FYI, "find" is opensource and seems to be a relatively straightforward implementation built on fts.

Yes, I know, and still my fts code in DirScanner was slower. So I assumed it had to do with the use of NSURLs.

However, I must now admit that I was wrong in that. While find is still twice as fast compared to the search performance in FindAnyFile, it is now on-par with my most recent DirScanner version: After Jim Luther told me about the key to using fileSystemRepresentation instead of converting paths to NSStrings in my code, DirScanner became significantly faster. Yet, I failed to contemplate that fact when I wrote above that fts would still be 2x faster, or was too lazy (or confident) to double check.

Now, with your findings on running parallel dir reads, I have a big task ahead of me rewriting that code. I'm excited. Thank you again for all your input.

The thing that confuses me most about searchfs performance on APFS is that when I search for something that's not findable (e.g. a unique name) both with searchfs and with a conventional method that eventually goes thru getatrrlistbulk, it didn't make much of a difference - even though the number of syscalls should be immensely different (and thereby contributing a lot to the overall time): searchfs should be a single syscalls, whereas the other method will be in the 100000s, right?

So, I think there are two different situation that play out. First off, there are edge cases which just inherently push "against" AFPS. Examples include:

1. Spinning media.

2. The "immediately after mount" case I outlined above.

3. Compressed file (not sure where this stands today, but was definitely an issue in the past).

Note that 1 & 2 are not bugs but are the DIRECT consequences of design choices that underly APFS, the same design choices that make it much "better" than HFS+. Engineering is all about tradeoffs and it's isn't reasonable to compare the strong point of one file system while ignoring all of it's other weaknesses*.

*For example, HFS+ has serious issues with multithreaded access and very poor support for file cloning and snapshots.

In the second case, where you've ruled out those other factors, then I think there are actually two things going on:

1. I think HFS+ happens to be PARTICULARLY fast at this. If you're entire goal was to design a file system that was really fast at this sort of search, I think you'd end up with something a lot like HFS+. Again, a lot of that is about tradeoffs. For example, using nodeName as part of the b-tree sorting is not a good engineering choice on a modern system, as it can force small operation (like a simple file rename) to cause major shifts in the entire b-tree. However, it can certainly speed up searching*.

*Have you compared searching by comparing searches that DON'T involve the file name? I'd be curious to see how that changes HFS+ performance.

1. Specifically comparing searchfs vs getatrrlistbulk, if you end up having read the entire catalog structure off disk, then that cost can mask any performance difference.

Finally, I think it's worth thinking about why searchfs exists at all. Ignoring the case where the file system structure has some inherent benefit (like HFS+ name searching), searching basically means "retrieve specific attributes from every file and make a comparison". We already have an API optimized for that retrieval (getatrrlistbulk) and, in general, operations that block in the kernel for extended period of time are worth avoiding. However, the real benefit here isn't local filesystem, it's remote file system case where the core cost isn't the direct syscall cost, but the cost of transferring all of those individual file records.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Right - but that huge advantage was lost when Apple decided to drop AFP support and didn't add such a feature to SMB. (And this is part of my current pain having to find alternative search methods (https://findanyfile.app/fs/) for various servers, such as Spotlight, SSH login with "find" execution on the server, using Everything's http server on Windows, and so on. It was all so easy when we had AFP).

Is there even any supported network file system left that supports searchfs? Does NFS (I believe I never checked because it's so diffucult to use)?

Specifically comparing searchfs vs getatrrlistbulk, if you end up having read the entire catalog structure off disk, then that cost can mask any performance difference.

I just did a new comparison: I compared how searchfs compares vs. recursive dir scan on HFS+ vs. APFS. On SSDs, of course. Always searching for a non-existing file name (so that no extra attributes need to be collected).

A search on a HFS+ vol with 8.8M items (6.6M files, 2.2M dirs) on a fast Intel system:

• searchfs: 40s
• find: 270s

And on a HFS+ vol with 2.6M items (2M files, 600k dirs), Mac Mini 2018:

• searchfs: 11s
• find: 70s

We can see that the search locally in the FS makes a huge different here. About factor 6 each time. Probably due to the many kernel-user space transitions.

Now a search on APFS, with 1.75M items (1.5M files and 250k dirs), on an M4 Mini:

• searchfs: 13s
• find: 35s

And APFS with 490k items (360k files, 130k dirs), on an Intel Mini 2018:

• searchfs: 9s
• find: 40s

So, there's still a difference, but smaller. Which may suggest that there's still room for improvement on the searchfs part, or not.

But it seems that on APFS everything (both searchfs and readdir etc) is significantly slower compared to HFS+, which is probably due to the dual-tree nature of APFS that requires more reads from the medium. With this in mind, the difference of searchfs performance of HFS+ vs. APFS becomes smaller, and so my bug report (radar 35052030, FB5367373) isn't really an issue. I've justed close it.

I suggest we put this to rest. There's nothing to be done about this anyway.

I'd rather have someone focus on how to make the Finder be faster when browsing the /Applications folder, e.g. by avoiding needless repeat calls into the FS, because that affects everyone, and to a point where one can get impatient (just try to mount another Mac's startup volume over SMB and browse that Mac's Apps folder over the network and it's time to make a coffee).