Backing up dataless files

Our backup app runs as a LaunchDaemon, has APFS-snapshot entitlement, and attempts to read and back up dataless files/directories by calling setiopolicy_np(IOPOL_TYPE_VFS_MATERIALIZE_DATALESS_FILES, IOPOL_SCOPE_THREAD, IOPOL_MATERIALIZE_DATALESS_FILES_ON) before reading files/directories. This worked for a while but lately has been producing many ETIMEDOUT errors for our users, especially for iCloud files but sometimes for FileProvider files, e.g. Dropbox. We've attempted to wait up to 5 minutes and retry, but the ETIMEDOUT error persists. In many cases, the errors stop hours or days later.

Are you attempting to materialize the dataless file of a snapshot? That's a weird edge case where I'm honestly not sure what would happen. Do you have any more information about the files this occurred on? Size, contents, etc.? Also, how long does it take for you to receive ETIMEDOUT?

Having said that...

Why is the user able to easily materialize dataless files but our app isn't?

What is the correct approach for backing up these files/directories?

...I think the answer to both of these questions is to use FileCoordination to initiate and manage the download. I'm not sure what the exact circumstances would be, but I'd expect there to be cases where the low-level calls failed and FileProvider succeeded. The low-level calls are ultimately a compatible layer, not a full solution, so I'm sure there are cases where they fail the open simply because the alternative (blocking for the full download) wasn't really practical/viable.

What APIs can we use to determine the actual issue with a given dataless file/directory that's giving us a read error?

I'm not sure there's an API that will give you great visibility into exactly what the error is/was, but there are new NSFileManager APIs that will let you more directly manage the download process.

__
Kevin Elliott
DTS Engineer, CoreOS/Hardware