`cp` ( & friends ) silent loss of extended attributes & file flags

As an example, ditto which was introduced more than 10 years ago...

This makes me feel very old. "ditto" was part of the base release of Mac OS X 10.0, which makes it 23 years old.

In any case, I want to start by commenting on this:

--extattr     Preserve extended attributes (requires --rsrc). As of Mac
                   OS X 10.5, --extattr is the default.
                   

The idea that extended attributes are something that can simply be "preserved" is dangerously naive. What's important to understand here is that extended attributes don't represent a coherent form of metadata but are instead basically a form of "surrender" on the part of file system designers.

Historically, file systems were basically designed around the assumption that a "file" consisted of:

a) A series of block that stored it's "data".

b) A discrete collection of metadata which stored information about those blocks ("name", "times", "permissions", etc....).

The file systems job was then quite simple- provide a place to store the data ("a"), define a fixed structure to hold the metadata ("b"), dream up a smart way to organize both, implement all that, and then have a nice nap in the afternoon.

Unfortunately, the problem here turned out to be the word "discrete". In practice, there has been a long history of new and brilliant idea which would be AWESOME if the file system would just track this one more piece of metadata... which then forced the file system designer to come up with yet another silly trick to store metadata it did not in fact have space for.

Extended attributes exist because file system designer final got tired of all this foolishness. They still kept "b" around (because there is a core set of metadata every file will have), but they also include a general mechanism that could be used to store "anything". Anyone with a fancy idea can now attach whatever they want to a file and the file system designer can nap in peace.

However, here is the key point in that. Extended attributes don't have any single coherent "meaning", nor do they "do" any coherent "thing". The higher level system determines what they mean and what they'll "do", which makes blindly copying them EXTREMELY problematic.

How to recover the standard behaviour --extattr working on modern macOS?

Case in point "--extattr" has NEVER simply captured "all" extended attributes. In reality, it has only ever captured the specific attributes it recognizes, since that's the ONLY safe option.

Moving the the specific case you mentioned:

drwx------@ 5 root  wheel  datavault 160 Jan 20  2024 /private/var/db/ConfigurationProfiles/Store
                           *********
        com.apple.rootless       28 

That is a datavault directory (Quinn's post here has more info). It only be accessed by specifically entitled processes, which is why nothing you've tried has been able access it. Disabling SIP will allow you to access it's contents, however, keep in mind that it's content and structure are all private, so anything you see/find could change at any time.

However, that doesn't explain this:

As a consequence, all backup scripts and applications are failing more or less silently, and provide corrupted copies of files or directories.

I'm not sure what you mean by "corrupt", but nothing stored inside a data vault should need to be backed up. It's understood that they're not generally accessible or preservable, so the system cannot rely on them persisting.

What problem(s) are these missing files actually causing?

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware

What problem(s) are these missing files actually causing?

Sorry for the late answer and a lot of unsuccessful work around this problem ( still a blocking one, only on macOS ).

Thank you for your nice answer.

It's now very difficult to make a perfect copy, of a macOS core filesystem. One on which it would be possible to restart in case of a failure within less than a few minutes, i.e. a clone copy.

It's now very difficult to make a perfect copy of a macOS core filesystem. One on which it would be possible to restart in case of a failure within less than a few minutes, i.e. a clone copy.

If you're talking about creating a bootable copy of an existing system, then yes, that's true. There are a lot of different reasons for that, but, off the top of my head, the main issues are:

(1) The actual "system" volume (what your operating system actually runs from) is NOT in fact an APFS "volume" in the normal sense. What you actually boot off of is, in fact, a snapshot of the true "main" volume. This has a few advantages:

• It provides an additional level of protection beyond simply being read-only since it is, quite literally, IMPOSSIBLE to modify through any sort of "normal" volume level interaction. Snapshots are an inherently read-only part of APFS structure, so modifying an existing snapshot would be the equivalent of creating your own (non-standard) APFS implementation. Certainly an extreme level of effort, particularly since you'd still need all of the encryption keys for that implementation to work.

• The lower level integration makes it easier/quicker to validate that the system itself is properly signed and has not been tampered with.

(2) There are a variety of security mechanisms in the system that prevent some files from being accessed by accessed by anything other than the correct/expected clients, even if the accessing process is running as root. Not only are these files difficult/impossible to copy, it's also difficult to creating these files from "scratch" is also difficult/impossible. This is primarily to prevent these kinds of files being used/abused in a way that create other, unexpected problems.

That leads back to here:

It's now very difficult to make a perfect copy of a macOS core filesystem.

I'm old enough to remember when the idea that creating a bootable copy of a system might require any sort of "special" effort or tool would have been considered slightly ridiculous. On Classic MacOS, the way you create a bootable copy of an existing system is that you pick up the folder named "System Folder"... and you copy it to the new volume. That's it. Strictly speaking, that folder didn't even need to be named "System Folder" or be in any particular location.

However, for better or worse, preserving that behavior/flexibility has simply never been a goal of Mac OS X/macOS. It's certainly become more difficult over time; however, I think it important to understand that even the original release of Mac OS X 10.0 made managing and copying the system WILDLY more complicated than Classic MacOS. Indeed, a major factor that led to the set of "volume cloning" apps that exists today was simply the expectation that you could/should be able to easily create bootable copies of an existing system.

One on which it would be possible to restart in case of a failure within less than a few minutes, i.e. a clone copy.

I don't know what your specific needs/requirements are here, but as someone who has accidentally destroyed systems more than a few times[1], my own strategy has been to shift my home directory and as much other data as possible onto its own volume[2] outside of the "normal" system hierarchy. It's far from a perfect solution, and many applications complicate this approach (requiring installation in /System, scattering components across the Data volume, etc.); however, in the "best" case, this means I can erase and reinstall the system with minimal concern about data loss, restoring my "standard" user configuration without a great deal of extra "fiddling". At times, I've also used the approach across systems by configuring two different system installs to use the same home directory for my primary user.

I don't know if this will work for your needs, but it's another option you could look at.

[1] KEXT development opens up a vast array of fun and interesting ways to break the system.

[2] The feature is not particularly well known and now requires using Directory Utility, but the system has always allowed home directories to be located at any location, not just the default "Users" directory.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware