AppleDouble, aka dotbar, files not removed via Finder

Question

alkap OP

Created Dec ’24

Replies 1

Boosts 0

Participants 2

I've having trouble deleting AppleDouble files residing on my custom filesystem through Finder.

This also affects files that use the AppleDouble naming convention, i.e. their names start with '._', but aren't AppleDoubles themselves.

dtrace output

% cat dtrace_myfs_rm_appledouble_finder.log 
  4  -> myfs_vnop_readdir                   DesktopServicesHelper-4671 -> __MACOSX
  4  <- myfs_vnop_readdir                   DesktopServicesHelper-4671 -> __MACOSX -> 0
  1  -> myfs_vnop_rmdir                     DesktopServicesHelper-4671 -> root/__MACOSX
  1  <- myfs_vnop_rmdir                     DesktopServicesHelper-4671 -> root/__MACOSX -> 66 ;ENOTEMPTY
  1  -> myfs_vnop_readdir                   DesktopServicesHelper-4671 -> __MACOSX
  0  <- myfs_vnop_readdir                   DesktopServicesHelper-4671 -> __MACOSX -> 0
  0  -> myfs_vnop_readdir                   DesktopServicesHelper-4671 -> __MACOSX
  0  <- myfs_vnop_readdir                   DesktopServicesHelper-4671 -> __MACOSX -> 0
  0  -> myfs_vnop_rmdir                     DesktopServicesHelper-4671 -> root/__MACOSX
  2  <- myfs_vnop_rmdir                     DesktopServicesHelper-4671 -> root/__MACOSX -> 66
  2  -> myfs_vnop_rmdir                     DesktopServicesHelper-4671 -> root/__MACOSX
  1  <- myfs_vnop_rmdir                     DesktopServicesHelper-4671 -> root/__MACOSX -> 66
  1  -> myfs_vnop_remove                    DesktopServicesHelper-4671 -> root/__MACOSX
  2  <- myfs_vnop_remove                    DesktopServicesHelper-4671 -> root/__MACOSX -> 2 ;ENOENT
  2  -> myfs_vnop_readdir                   DesktopServicesHelper-4671 -> __MACOSX
  1  <- myfs_vnop_readdir                   DesktopServicesHelper-4671 -> __MACOSX -> 0
  1  -> myfs_vnop_readdir                   DesktopServicesHelper-4671 -> __MACOSX
  1  <- myfs_vnop_readdir                   DesktopServicesHelper-4671 -> __MACOSX -> 0
  1  -> myfs_vnop_rmdir                     DesktopServicesHelper-4671 -> root/__MACOSX
  2  <- myfs_vnop_rmdir                     DesktopServicesHelper-4671 -> root/__MACOSX -> 66

In vnop_readdir, 'struct dent/dentry' is set up for dotbar files and written to the uio_t buffer. It's just that my vnop_remove is never called for dotbar files, and I don't understand why not.

Dotbar files are removed successfully, when deleted through command line.

For SMBClients, vnop_readdir is followed by vnop_access, followed by vnop_lookup, followed by vnop_remove of dotbar files.

SMBClient rm dotbar files dtrace output

  4  -> smbfs_vnop_rmdir                      DesktopServicesHelper-1512 -> TheRooT/__MACOSX
  4  <- smbfs_vnop_rmdir                      DesktopServicesHelper-1512 -> TheRooT/__MACOSX -> 66
  4  -> smbfs_vnop_readdir                    DesktopServicesHelper-1512 -> __MACOSX
  0  <- smbfs_vnop_readdir                    DesktopServicesHelper-1512 -> __MACOSX -> 0
  0  -> smbfs_vnop_readdir                    DesktopServicesHelper-1512 -> __MACOSX
  0  <- smbfs_vnop_readdir                    DesktopServicesHelper-1512 -> __MACOSX -> 0
  0  -> smbfs_vnop_access                     DesktopServicesHelper-1512 -> __MACOSX action: 8 ;KAUTH_VNODE_SEARCH
  0  <- smbfs_vnop_access                     DesktopServicesHelper-1512 -> __MACOSX action: 8 -> 0
  0  -> smbfs_vnop_lookup                     DesktopServicesHelper-1512 -> __MACOSX/._200mb-example-dummy-file_1.txt nameiop: 2
  0  <- smbfs_vnop_lookup                     DesktopServicesHelper-1512 -> __MACOSX/._200mb-example-dummy-file_1.txt nameiop: 2 -> 0
  0  -> smbfs_vnop_getattr                    DesktopServicesHelper-1512 -> ._200mb-example-dummy-file_1.txt
  0  <- smbfs_vnop_getattr                    DesktopServicesHelper-1512 -> ._200mb-example-dummy-file_1.txt -> 0
  0  -> smbfs_vnop_access                     DesktopServicesHelper-1512 -> ._200mb-example-dummy-file_1.txt action: 16 ;KAUTH_VNODE_DELETE
  0  <- smbfs_vnop_access                     DesktopServicesHelper-1512 -> ._200mb-example-dummy-file_1.txt action: 16 -> 0
  0  -> smbfs_vnop_getattr                    DesktopServicesHelper-1512 -> ._200mb-example-dummy-file_1.txt
  0  <- smbfs_vnop_getattr                    DesktopServicesHelper-1512 -> ._200mb-example-dummy-file_1.txt -> 0
  0  -> smbfs_vnop_remove                     DesktopServicesHelper-1512 -> __MACOSX/._200mb-example-dummy-file_1.txt
  3  <- smbfs_vnop_remove                     DesktopServicesHelper-1512 -> __MACOSX/._200mb-example-dummy-file_1.txt -> 0
  3  -> smbfs_vnop_inactive                   DesktopServicesHelper-1512 -> ._200mb-example-dummy-file_1.txt
  3  <- smbfs_vnop_inactive                   DesktopServicesHelper-1512 -> ._200mb-example-dummy-file_1.txt -> 0
  3  -> smbfs_vnop_reclaim                    DesktopServicesHelper-1512 -> ._200mb-example-dummy-file_1.txt
  3  <- smbfs_vnop_reclaim                    DesktopServicesHelper-1512 -> ._200mb-example-dummy-file_1.txt -> 0
  0    -> smbfs_vnop_rmdir                    DesktopServicesHelper-1512 -> TheRooT/__MACOSX
  0    <- smbfs_vnop_rmdir                    DesktopServicesHelper-1512 -> TheRooT/__MACOSX -> 0
  0    -> smbfs_vnop_inactive                 DesktopServicesHelper-1512 -> __MACOSX
  0    <- smbfs_vnop_inactive                 DesktopServicesHelper-1512 -> __MACOSX -> 0
  0    -> smbfs_vnop_reclaim                  DesktopServicesHelper-1512 -> __MACOSX
  0    <- smbfs_vnop_reclaim                  DesktopServicesHelper-1512 -> __MACOSX -> 0

Implementing vnop_access for my filesystem did not result in the combination of vnop_lookup and vnop_remove being called for dotbar files.

Perusing the kernel sources, I observed the following functions that might be involved, but I have not way of verifying this, as none of the functions of interest are dtrace(1)-able, rmdir_remove_orphaned_appleDouble() in particular.

rmdir_remove_orphaned_appleDouble() -> VNOP_READDIR().
rmdirat_internal() -> rmdir_remove_orphaned_appleDouble()
unlinkat()-> rmdirat_internal()
rmdir()-> rmdirat_internal()

Any pointers on how dotbar files may be removed through Finder would be greatly appreciated.

Answered by DTS Engineer in 819851022

I've having trouble deleting AppleDouble files residing on my custom filesystem through Finder.

What's the actual failure? What does the Finder return to the user? And is anything logged to the console?

This also affects files that use the AppleDouble naming convention, i.e. their names start with '._', but aren't AppleDoubles themselves.

SO, my first question here is why are you dealing with AppleDouble files at all?

The AppleDouble format exists because 25+ years ago the original data/resource fork structure was "unusual", so a flat file structure was created so that the data streams could be stored separately in single data stream file systems. However, the world has changed enormously since then. More specifically:

The resource fork itself is largely "gone". With the resource manager gone, we no longer provide any API for manipulating it and the bundle structure has replace the role the resource fork used to serve.
The systems own use of multiple forks has actually increased, through the use of "extended attributes".
Multi-fork/stream files systems are now commonplace. SMB supports them through "alternate data streams", NFS through "Extended Attribute Support". All the modern file systems I'm aware of support them as well, typically calling them "Extended Attributes".

The bottom line here is that unless you're working with something particularly old, I'd expect you to just implement "native" extended attribute support (and possibly named fork) instead of relying on the VFS layer and AppleDouble.

For SMBClients, vnop_readdir is followed by vnop_access, followed by vnop_lookup, followed by vnop_remove of dotbar files.

So, there are a number of issues here that make this kind of operation comparison very difficult to make:

The VFS system itself alters it's own behavior based on the configuration you present to it, so unless you're file system configuration matches the SMBClient in detail, there's not reason the exact call configuration will match.
The calling client (in this can, the Finder/DesktopServices) can ALSO alter it's behavior, either based on the characteristics of the volume itself OR based on the characteristic of the file returned to it. Unless you've compared those characteristics in detail you can't really be sure what's going on. As one side note here, it does appear (based on looking at our code) that the Finder is looking for SF_DATALESS and that our SMBClient may be setting that.
It can be very difficult to try and debug a "high level" case like the Finder, where you have very poor visibility into what it's actually "doing" in any given operation.

On that last point in particular, my main suggestion would be to try testing with carbon file manager, particularly FSDeleteObject and possibly FSDeleteFork (depending on the failure you're trying to understand). While those APIs don't use the same implementation as the Finder, they were implemented at roughly the same time as the Finder's original "core" implementation and generally implement similar semantics. Notably, they'll fail the delete of open files while the standard Unix APIs will not.

Perusing the kernel sources, I observed the following functions that might be involved, but I have not way of verifying this, as none of the functions of interest are dtrace(1)-able, rmdir_remove_orphaned_appleDouble() in particular.

I don't think that's a "standard" code path. My understanding of the code is that under any normal circumstance, the AppleDouble file would have already been removed before you ever reached rmdir. To put that another way, the VFS layers implementation is supposed to be treating the AppleDouble file as "part" of it's corresponding data file, moving/deleting/copying them "in sync". Having an orphaned file at all implies something went wrong much earlier, when the original file was deleted. That's the point you need to be looking at, not remove directory.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

Boost

Answer 1

DTS Engineer OP

Apple

Jan ’25

Accepted Answer

Recommended

I've having trouble deleting AppleDouble files residing on my custom filesystem through Finder.

What's the actual failure? What does the Finder return to the user? And is anything logged to the console?

This also affects files that use the AppleDouble naming convention, i.e. their names start with '._', but aren't AppleDoubles themselves.

SO, my first question here is why are you dealing with AppleDouble files at all?

The AppleDouble format exists because 25+ years ago the original data/resource fork structure was "unusual", so a flat file structure was created so that the data streams could be stored separately in single data stream file systems. However, the world has changed enormously since then. More specifically:

The resource fork itself is largely "gone". With the resource manager gone, we no longer provide any API for manipulating it and the bundle structure has replace the role the resource fork used to serve.
The systems own use of multiple forks has actually increased, through the use of "extended attributes".
Multi-fork/stream files systems are now commonplace. SMB supports them through "alternate data streams", NFS through "Extended Attribute Support". All the modern file systems I'm aware of support them as well, typically calling them "Extended Attributes".

The bottom line here is that unless you're working with something particularly old, I'd expect you to just implement "native" extended attribute support (and possibly named fork) instead of relying on the VFS layer and AppleDouble.

For SMBClients, vnop_readdir is followed by vnop_access, followed by vnop_lookup, followed by vnop_remove of dotbar files.

So, there are a number of issues here that make this kind of operation comparison very difficult to make:

The VFS system itself alters it's own behavior based on the configuration you present to it, so unless you're file system configuration matches the SMBClient in detail, there's not reason the exact call configuration will match.
The calling client (in this can, the Finder/DesktopServices) can ALSO alter it's behavior, either based on the characteristics of the volume itself OR based on the characteristic of the file returned to it. Unless you've compared those characteristics in detail you can't really be sure what's going on. As one side note here, it does appear (based on looking at our code) that the Finder is looking for SF_DATALESS and that our SMBClient may be setting that.
It can be very difficult to try and debug a "high level" case like the Finder, where you have very poor visibility into what it's actually "doing" in any given operation.

On that last point in particular, my main suggestion would be to try testing with carbon file manager, particularly FSDeleteObject and possibly FSDeleteFork (depending on the failure you're trying to understand). While those APIs don't use the same implementation as the Finder, they were implemented at roughly the same time as the Finder's original "core" implementation and generally implement similar semantics. Notably, they'll fail the delete of open files while the standard Unix APIs will not.

Perusing the kernel sources, I observed the following functions that might be involved, but I have not way of verifying this, as none of the functions of interest are dtrace(1)-able, rmdir_remove_orphaned_appleDouble() in particular.

I don't think that's a "standard" code path. My understanding of the code is that under any normal circumstance, the AppleDouble file would have already been removed before you ever reached rmdir. To put that another way, the VFS layers implementation is supposed to be treating the AppleDouble file as "part" of it's corresponding data file, moving/deleting/copying them "in sync". Having an orphaned file at all implies something went wrong much earlier, when the original file was deleted. That's the point you need to be looking at, not remove directory.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

0