Unexpected Permission denied error on file sharing volume

Hi,

I tried to replicate the situation on the server, but the newly copied application files do not have xattrs,

Copied from the client to the server? What copied the files? This particular attribute is used to track the "source" of file, so it will change (or be removed) depending on what actually interacts with the file.

In addition:

The file that I described had a value of 01 02 00 F5 28 1A 84 40 15 BA C9 when I inspected it on the server.

The contents of that data include a machine specific reference to the source app, so the data is only valid on the source machine.

However, its current value is empty. Other files in that application had the same attribute and it seems they all now have empty values.

How is the file actually being copied?

This issue here is that while the term "copy" is obviously widely used, "copying" isn't really something any file system actually defines, nor is there any coherent/standard definition of what constitutes a "copy" of a file (particularly across file systems). Ultimately, every copy engine/implementation has to make a long series of decisions, choosing what data to preserve and which it discard. I'm concerned that you're getting an empty value here because the engine is trying to blindly copy "everything", is not a very good idea (with this xattr being one example).

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

The application is copied using ditto.

Using Howard Oakley's xattred program, I examined the original application on the client. (I built this application on the client machine.) The program reports that every file and directory in the application has the com.apple.provenance attribute with identical values. More interesting is that the file Contents/runtime-x86/Contents/_CodeSignature/CodeResources has the com.apple.quarantine attribute, with the value 0083;673245b6;Safari;71F7C727-7831-4DBB-B9FA-13D14B18F09D. I'm guessing that is because I copied this directory from a downloaded JDK. The corresponding file under Contents/runtime-arm does not have this attribute. Presumably, the process by which I obtained this JDK runtime was different.

Is this the cause of the problem?

Should I delete the com.apple.quarantine attribute?

What process should I use to extract a runtime from a JDK? It seems that files in a JDK routinely have this attribute. A JDK is not an application, however, so it never gets cleared?

Using Howard Oakley's xattred program, I examined the original application on the client. (I built this application on the client machine.) The program reports that every file and directory in the application has the com.apple.provenance attribute with identical values. More interesting is that the file Contents/runtime-x86/Contents/_CodeSignature/CodeResources has the com.apple.quarantine attribute, with the value 0083;673245b6;Safari;71F7C727-7831-4DBB-B9FA-13D14B18F09D. I'm guessing that is because I copied this directory from a downloaded JDK. The corresponding file under Contents/runtime-arm does not have this attribute. Presumably, the process by which I obtained this JDK runtime was different.

All of that sounds reasonable and correct.

Is this the cause of the problem?

Almost certainly. I'm not sure sure of the exact mechanics of what happened, but it sounds like the system basically interpreted the embedded object as having been "injected" into the whataver app you were copying.

Should I delete the com.apple.quarantine attribute?

Depends on what the "I" here is. If you're acting as the transport server then, no, I don't think I would remove it. While it isn't the case here, this is a security mechanism and blindly disabling it is likely to be a mistake.

If you're building an app/executable object for broader "distribution" then, yes, that is what I would do. I'd actually strip them from ALL objects, as they're simply not necessary or relevant. FYI, this kind of issues is one of the reasons many apps are distributed inside disk images (or other "container" formats like "zip"). During distribution creation the isolated file system makes it easier to do this kind of "cleanup", after which you can then convert the disk image to read-only, ensuring it can't attach again.

What process should I use to extract a runtime from a JDK? It seems that files in a JDK routinely have this attribute. A JDK is not an application, however, so it never gets cleared?

If your creating app builds, I'd actually do the cleanup at the "end" of the creation process, not the beginning. That is, what I would do is:

• Move the app build into a controlled location (often a disk image).

• Strip all xattr's off the object using:

xattr -cr <app object path>

• Test to make sure everything works correctly.

Note that blindly stripping all xattr's like this is NOT something I'd normally recommend, as it can cause data loss or other issues (depending on exactly what the attributes are actually "for"). However, the difference here is that you're intentionally trying to create a specific object with a specific configuration, so if something breaks you're going to go back and figure out what happened and fix the issue at the "source".

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

The part that I don't understand is why an attribute designed to warn users of an untrusted application would have the side effect of preventing the application from being deleted (or at least making it more difficult). Granted that users would not delete the application but instead move it to the trash, but from a file sharing client apparently I no longer have that option.

Is it the case that I am able to delete the application on the server because Terminal has full disk access, but I cannot delete it remotely because a file sharing client does not have similar permission?

The part that I don't understand is why an attribute designed to warn users of an untrusted application would have the side effect of preventing the application from being deleted (or at least making it more difficult).

From the file system's security perspective, the "delete" operation doesn't really exist, just "read' and "write" directory. Jumping back to an earlier message, I pointed out that client and server were "seeing" different xattr state for the two objects:

Client: 
drwxr-xr-x  1 alan  staff  16384 Dec 12 11:34 Contents

Server:
drwxr-xr-x@ 2 alan  staff  68 Dec 12 11:34 Contents

What's actually going on here is that "read" access to "Contents" is being blocked (more on that in a moment). The delete failure then flows from that, both because you can't see it's contents (so you can't empty the diretory) and because the sandbox won't allow it to be modified (and write access is required to delete a directory).

Next moving to here:

Is it the case that I am able to delete the application on the server because Terminal has full disk access,

Note that the Terminal DOESN'T inherently have FDA. It often does (because heavy users give it that) but by default it does not. That detail is relevant here because part of the issue here is that under "normal" conditions, the sandbox doesn't simply block access but tell them the issue is going on and give them a path to resolve it.

I'm also not sure that the local sandbox alone would have immediately blocked access. It's possible a local process would have been allowed to just delete it. However, that leads to here:

but I cannot delete it remotely because a file sharing client does not have similar permission?

Sort of, however, I think what's being block is actually the server, not the client. In other words, in the xattr state I listed earlier, the issue isn't that the client isn't "showing" the xattr data, it's that the server never SENT it xattr data. And the only reason the server wouldn't send it is because it couldn't read it.

In terms of the server side of this, I think there are actually a few different factors at work here that create this particular failure (which is also why you don't see it all that often):

• The server is a daemon, so all it can do is "fail" (for example, it can't notify the local users).

• This is happening inside an app bundle, which the system is particularly careful to protect.

• The modification pattern you happen to have used means that it looks like one app modified the contents of an unrelated app, something the system is obviously "sensitive" to.

One final point on the "middle" case, it's possible that you could have moved object around in a way that would have bypassed the problem. Did you ever try moving "runtime-arm" into it's own directory (entirely outside the app container), then deleting the object?

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

I tried removing the extended attributes from the source app, but it did not solve the problem. I also rebooted the server and avoided opening a Finder window on the target directory, but the problem persists. The following shows actions performed on the client while connected to the remote volume.

Mac-mini:test alan$ xattr -lr /Applications/apps/VAquaManager.app
/Applications/apps/VAquaManager.app: com.apple.macl: 
Mac-mini:test alan$ ditto /Applications/apps/VAquaManager.app VAquaManager.app
Mac-mini:test alan$ xattr -lr VAquaManager.app
VAquaManager.app/Contents/app/.jpackage.xml: com.apple.FinderInfo: 
VAquaManager.app/Contents/runtime-arm/Contents/Home/.DS_Store: com.apple.FinderInfo: 
VAquaManager.app: com.apple.macl: 
Mac-mini:test alan$ rm -rf VAquaManager.app
rm: VAquaManager.app/Contents/runtime-arm/Contents/Home: Permission denied
rm: VAquaManager.app/Contents/runtime-arm/Contents: Permission denied
rm: VAquaManager.app/Contents/runtime-arm: Permission denied
rm: VAquaManager.app/Contents/runtime-x86/Contents/Home: Permission denied
rm: VAquaManager.app/Contents/runtime-x86/Contents: Permission denied
rm: VAquaManager.app/Contents/runtime-x86: Permission denied
rm: VAquaManager.app/Contents: Permission denied
rm: VAquaManager.app: Permission denied
Mac-mini:test alan$ xattr -lr VAquaManager.app
xattr: [Errno 13] Permission denied: 'VAquaManager.app/Contents/runtime-arm/Contents/Home/conf'
xattr: [Errno 13] Permission denied: 'VAquaManager.app/Contents/runtime-x86/Contents/Home/conf'

On the server, I can list VAquaManager.app with no error. There are no extended attributes on the files that were not deleted.

It seems odd that FinderInfo extended attributes were present before I tried to delete the application. That makes me wonder if the Finder is calculating sizes even when the directory is not displayed.

On the other hand, I do have the directory open in Finder on the client. Is it possible that the Finder on the client would attach extended attributes to remote files?

I closed the FInder window on the client and repeated the test to a new target directory on the server. I get the same result.

One thing I'm not sure if is when the Finder removes its FSEvents listeners.

The issue is actually here:

Mac-mini:test alan$ xattr -lr /Applications/apps/VAquaManager.app
/Applications/apps/VAquaManager.app: com.apple.macl: 

The "macl" refers to "Mandatory Access Control List". We've never formally documented this, but they're a mechanism the system uses to try and further constrain processes ability to interact with data the user has not granted them access to, particularly the ability to destroy data as well as modify it.

I don't think a standard copy would have generated a MACL*, however, you specifically did not ask for a standard copy. You used "ditto", which tries to create a more exact "duplicate" of the original object, even when those semantics restrict access that object.

*More specifically, if it did generate a MACL, that MACL would have been scoped to the creating process, which should have given unrestricted access to the newly created object.

Mac-mini:test alan$ ditto /Applications/apps/VAquaManager.app VAquaManager.app

It seems odd that FinderInfo extended attributes were present before I tried to delete the application. That makes me wonder if the Finder is calculating sizes even when the directory is not displayed.

No, that's not what's causing that. One thing to keep in mind here is that at this point macOS is old enough (25+ years) that many of it's details come from very old choices that aren't necessarily what we'd do today. The "FinderInfo" xattr isn't really the "general purpose" storage mechanism it's name would imply. It was originally created to store specific HFS+ data on other file system, primarily the "finderInfo" field of the HFSPlusCatalogFolder struct. TN1150 has some additional details about that, but the short summary is that their use to support a few details of the Finder's implementation, NOT as part of any kind of security or control mechanism.

In terms of the particular suggestion here:

That makes me wonder if the Finder is calculating sizes even when the directory is not displayed.

...the Finder does not have it's own mechanism for caching directory sizes, particularly to disk. APFS does do this, but that implementation is internal to APFS and not something that would be exposed through an xattr. I'll also note that this kind of caching is ENTIRELY about directory sizing, not individual files. There's no point caching the size of individual file.

This data was created for one of two reasons:

1. The destination file system couldn't store the metadata natively, so the system use the xattr. In other words, the data actually came from the source file, it just wasn't visible as an xattr because it was stored directly in the file record of the file system.

2. ditto created in unnecessarily, probably because an smb issue "masked" the destination volumes full capabilities.

However, the bottom line here is that I don't think

On the other hand, I do have the directory open in Finder on the client. Is it possible that the Finder on the client would attach extended attributes to remote files?

Yes, the Finder can do that. However, "com.apple.FinderInfo" is not the cause of any of what you're seeing.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

I don't think a standard copy would have generated a MACL, however, you specifically did not ask for a standard copy. You used "ditto", which tries to create a more exact "duplicate" of the original object, even when those semantics restrict access that object.

What would create a standard copy, preferably an operation that can be performed on the command line?

More specifically, if it did generate a MACL, that MACL would have been scoped to the creating process, which should have given unrestricted access to the newly created object.

I'm not sure how that would work if the the newly created object is on a remote file system. Is ditto creating the MACL or is the file sharing daemon creating it? Is the MACL scoped to the file sharing daemon or to the client Terminal process?

...the Finder does not have it's own mechanism for caching directory sizes, particularly to disk. APFS does do this, but that implementation is internal to APFS and not something that would be exposed through an xattr.

But the target volume is not APFS, it is Mac OS Extended (Journaled). The known behavior of displaying cumulative sizes for bundled applications is presumably still implemented by the Finder?

If I use xattr -cr on the server to remove the extended attributes, including the MACL, from the application on the server volume, the clinet still gets errors:

Mac-mini:test2 alan$ xattr -lr V*
xattr: [Errno 13] Permission denied: 'VAquaManager.app/Contents/runtime-arm/Contents/Home/conf'
xattr: [Errno 13] Permission denied: 'VAquaManager.app/Contents/runtime-x86/Contents/Home/conf'
rm -rf V*
rm: VAquaManager.app/Contents/runtime-arm/Contents/Home/conf: Permission denied
rm: VAquaManager.app/Contents/runtime-arm/Contents/Home: Permission denied
rm: VAquaManager.app/Contents/runtime-arm/Contents: Permission denied
rm: VAquaManager.app/Contents/runtime-arm: Permission denied
rm: VAquaManager.app/Contents/runtime-x86/Contents/Home/conf: Permission denied
rm: VAquaManager.app/Contents/runtime-x86/Contents/Home: Permission denied
rm: VAquaManager.app/Contents/runtime-x86/Contents: Permission denied
rm: VAquaManager.app/Contents/runtime-x86: Permission denied
rm: VAquaManager.app/Contents: Permission denied
rm: VAquaManager.app: Permission denied

I don't think a standard copy would have generated a MACL, however, you specifically did not ask for a standard copy. You used "ditto", which tries to create a more exact "duplicate" of the original object, even when those semantics restrict access that object. What would create a standard copy, preferably an operation that can be performed on the command line?

"cp" would be the standard solution.

More specifically, if it did generate a MACL, that MACL would have been scoped to the creating process, which should have given unrestricted access to the newly created object.

I'm not sure how that would work if the the newly created object is on a remote file system. Is ditto creating the MACL or is the file sharing daemon creating it?

My understanding of what happened here is that:

• The MACL existed on the original file.

• ditto was run on the local machine, so it duplicated the MACL.

However, if ditto was run from the server side then that introduces another complication. You can read more about this here but there is an option for server-side copying that is only partially exposed through in our public API. For complicated historical reasons, it's very likely that ditto would have called into the server side copy while cp does NOT do so. Similarly, that server side copy could perserve the MACL in a way the standard copy process would not.

Is the MACL scoped to the file sharing daemon or to the client Terminal process?

I'm not sure.

But the target volume is not APFS, it is Mac OS Extended (Journaled). The known behavior of displaying cumulative sizes for bundled applications is presumably still implemented by the Finder?

Yes, but the standard flow doesn't cache the size. The standard flow recursively calculates the size then (I believe) monitors the directory for changes (using FSEvents or kqueue) and recursively updates the size when it detects a change. None of that is written to the file system or persisted in an strong way.

If I use xattr -cr on the server to remove the extended attributes, including the MACL, from the application on the server volume, the clinet still gets errors:

Did you check that the xattr's were actually gone? And did your try the same thing running as root? You might have to give Terminal Full Disk Access to clear the values.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

cp also copies the MACL

Mac-mini:test4 alan$ pwd
/Volumes/Alan’s iMac._smb._tcp.local/test4
Mac-mini:test4 alan$ xattr -lr /Applications/apps/VAquaManager.app
/Applications/apps/VAquaManager.app: com.apple.macl: 
Mac-mini:test4 alan$ cp -pR /Applications/apps/VAquaManager.app .
Mac-mini:test4 alan$ xattr -lr VAquaManager.app
VAquaManager.app: com.apple.macl: 

Deletion from the client still fails:

Mac-mini:test4 alan$ rm -rf VAquaManager.app
rm: VAquaManager.app/Contents/runtime-arm/Contents/Home: Permission denied
rm: VAquaManager.app/Contents/runtime-arm/Contents: Permission denied
rm: VAquaManager.app/Contents/runtime-arm: Permission denied
rm: VAquaManager.app/Contents/runtime-x86/Contents/Home: Permission denied
rm: VAquaManager.app/Contents/runtime-x86/Contents: Permission denied
rm: VAquaManager.app/Contents/runtime-x86: Permission denied
rm: VAquaManager.app/Contents: Permission denied
rm: VAquaManager.app: Permission denied

Finder most definitely does cache sizes, in .DS_Store files. I discovered this a while ago, because it can cause rm -rf to fail locally.

Yes, I did check that the extended attribute was gone after I removed it (on the server using Terminal).

After restarting the client and reconnecting to the remote volume, ls and xattr continue to get Permission denied errors. After restarting the server and reconnecting to the remote volume, ls and xattr succeed without error.

It seems that the file sharing server is working with cached data that does not match the file system.

Incorrect cached data could explain why the original recursive delete failed. For example, if deleting the last file in a directory does not immediately update the cached data for the parent, then the subsequent attempt to delete the parent directory will fail because the directory appears to not be empty (based on the cached data).

I repeated the experiment using the internal SSD (APFS) on the target machine instead of the external SSD (HFS+). The results were the same, except that xattr gets one more error:

xattr: [Errno 13] Permission denied: './VAquaManager.app/Contents/runtime-x86/Contents/Home/conf'
xattr: [Errno 13] Permission denied: './VAquaManager.app/Contents/runtime-arm/Contents/Home/legal'
xattr: [Errno 13] Permission denied: './VAquaManager.app/Contents/runtime-arm/Contents/Home/conf'

I suspect the difference is due to timing.

To test the hypothesis that extended attributes are involved, I tried copying a bunch of files from /Library/Developer/CommandLineTools (not all of them). There were no extended attributes in the original files and none on the copied files.

Deleting the copies produced errors. Oddly, trying to delete a seond time produced more errors.

Mac-mini:test6 alan$ rm -rf C
rm: C/usr/lib/sourcekitd.framework/Versions/A: Permission denied
rm: C/usr/lib/sourcekitd.framework/Versions: Permission denied
rm: C/usr/lib/sourcekitd.framework: Permission denied
rm: C/usr/lib/swift/pm/llbuild: Permission denied
rm: C/usr/lib/swift/pm: Permission denied
rm: C/usr/lib/swift: Permission denied
rm: C/usr/lib: Permission denied
rm: C/usr: Permission denied
rm: C/SDKs/MacOSX15.2.sdk/usr: Permission denied
rm: C/SDKs/MacOSX15.2.sdk: Permission denied
rm: C/SDKs: Permission denied
rm: C: Permission denied

Mac-mini:test6 alan$ rm -rf C
rm: C/usr/include: Permission denied
rm: C/usr/lib/clang: Permission denied
rm: C/usr/lib/tapi: Permission denied
rm: C/usr/lib/sourcekitd.framework/Versions/A/XPCServices: Permission denied
rm: C/usr/lib/sourcekitd.framework/Versions/A: Permission denied
rm: C/usr/lib/sourcekitd.framework/Versions: Permission denied
rm: C/usr/lib/sourcekitd.framework: Permission denied
rm: C/usr/lib/sourcekitdInProc.framework: Permission denied
rm: C/usr/lib/swift/pm/llbuild/llbuild.framework: Permission denied
rm: C/usr/lib/swift/pm/llbuild: Permission denied
rm: C/usr/lib/swift/pm: Permission denied
rm: C/usr/lib/swift: Permission denied
rm: C/usr/lib: Permission denied
rm: C/usr/share: Permission denied
rm: C/usr: Permission denied
rm: C/SDKs/MacOSX15.2.sdk/usr/include: Permission denied
rm: C/SDKs/MacOSX15.2.sdk/usr: Permission denied
rm: C/SDKs/MacOSX15.2.sdk: Permission denied
rm: C/SDKs: Permission denied
rm: C: Permission denied

Mac-mini:test6 alan$ xattr -lr C
xattr: [Errno 13] Permission denied: 'C/usr/include'
xattr: [Errno 13] Permission denied: 'C/usr/lib/clang'
xattr: [Errno 13] Permission denied: 'C/usr/lib/tapi'
xattr: [Errno 13] Permission denied: 'C/usr/lib/sourcekitd.framework/Versions/A/XPCServices'
xattr: [Errno 13] Permission denied: 'C/usr/lib/sourcekitdInProc.framework'
xattr: [Errno 13] Permission denied: 'C/usr/lib/swift/pm/llbuild/llbuild.framework'
xattr: [Errno 13] Permission denied: 'C/usr/share'
xattr: [Errno 13] Permission denied: 'C/SDKs/MacOSX15.2.sdk/usr/include'

So, I've been going over everything that's been sent and there are few points I want to revisit and think about again:

If I try to list the directory on the target (14.7.1) system

Is 14.7.1 when this specifically started happening? There was a significant issue with security scoped bookmark resolution that was introduced in 14.7.1 and was only resolved in 14.7.3. I ask because of this point:

After restarting the server and reconnecting to the remote volume, ls and xattr succeed without error.

The bug above was caused by issues with the keychain not unlocking when the system expected and that could explain the behavior above. That is:

1. The server launched at boot, couldn't get access to the data it expected and was left in a broken state.

2. The server was reset, returning to normal operation.

3. (speculative) Sleep/wake issues can end up causing keychain issues as well, which could recreate #1.

In a different direction:

• Is the server machine logged into by multiple users?

• Does the volume(s) you're seeing this in have the "Ignore Ownership on this volume" checkbox checked?

That checkbox works by having the system return hard coded permission settings based on the currently logged in user, but the effects of that can be odd if one user is logged in a the console and a different user is connecting via smb.

The broader point here is that IF reseting the server resolves the issue, then the underlying issue is actually with the server's state/configuration, NOT the actual file system data.

Incorrect cached data could explain why the original recursive delete failed. For example, if deleting the last file in a directory does not immediately update the cached data for the parent, then the subsequent attempt to delete the parent directory will fail because the directory appears to not be empty (based on the cached data).

I don't follow how you think this works. The Finder simply displays data, it isn't integrated into the lower level system the way you're describing. That is, the Finder can (and does) cache data to speed up it's own display of that data, but that cached data isn't used by the broader system, particularly not the command line tool layer, much less the kernel.

Notably, the behavior here:

if deleting the last file in a directory does not immediately update the cached data for the parent,

The requirement that a directory needs to empty is something individual file systems enforce, NOT the higher level system. Similarly, that requirement is tied to vnode management, not "size" or data content.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware