Possibly triggered CDP bug (can't re-establish Octagon Trust, ADP, facing potential fatal data loss)


this might not belong to dev forums, but rather support, the reason I'm posting here I seem to have localized problem via Apple logging, partially, and can't believe I can get any prompt answer if asked on discussions.apple.com...

ckks data inaccessible and potential data loss

what happend

I was stupid enough to delete a few key records from iCloud Keychain ("Local Items"), using official /Applications/Utilities/Keychain Access.app ( seems like ephemeral keys / identifiers, ckks, idms and stuff).

What was different this time: because the account uses ADP, I might have deleted some end-to-end encryption related material.

I was not initially too much upset, as normally "trust" can be reestablished via fresh login.

Prerequisite: my account is ADP-enabled. ckks state is waitfortrust which corresponds to what I get from Octagon CLI, later on in the post.

is@airstation ~ % /usr/sbin/ckksctl status | head -15

Global state:

CKKS state machine:   waitfortrust
Active account:       (null)
CloudKit account:     logged in
Account tracker:      <CKKSAccountStateTracker: <CKAccountInfo: accountStatus=Available, accountPartition=Prod, deviceToDeviceEncryptionAvailability=(account), hasValidCredentials=true, walrus=Could Not Determine>, hsa2: available>
Syncing Policy:       <TPSyncingPolicy: <TPPolicyVersion: 16, SHA256:/4gt8WFEXCVLYI+C+8/2MiMz6Srv0vpcvlkJ4gkepHQ=>, MacBookAir10,1, userViews: UNKNOWN>
Views from policy:    yes
Reachability:         network
Retry:                <CKKSNearFutureScheduler(zonemodifier-ckretryafter): no pending attempts
CK DeviceID:          0B643F9E-AD74-4916-84A4-D3589F0B2061
CK DeviceID Error:    (null)
Lock state:           <CKKSLockStateTracker: unlocked last:now>

Attempt to recover using trusted phone number

Obviously I just tried to login and waiting for the challenge to be sent to my trusted phone number. This didn't happen and System Settings GUI cannot handle it, doing just nothing.

Inspired by prior success: attempt using Octagon Trust CLI

Kudos to Apple for opening https://opensource.apple.com/source/Security/Security-59754.80.3/keychain/ which helped me tremendously to make sense of how it works, roughly :)

So: previously I used bottled peers data and/or escrow records and recall success in the past with this approach (using recover commands), on the other occasion, had success in a much more simple way, by

is@airstation ~ % /usr/sbin/otctl resetoctagon

While worked great before Apple introduced ADP, attempt to reset Octagon quite recently for one of my accounts resulted in banning me from reenabling ADP on the account for some 3 months, damn, I understand why you have this feature in place, but srsly, it came hard on me (didn't find workaround for it :)

... and how it failed in a way resembling actual CloudKit bug

is@airstation ~ % /usr/sbin/otctl allBottles

returns nothing, but this is probably due to I don't have any trusted devices now. Well, this should not be fatal yet, right?

Only not this

is@airstation ~ % /usr/sbin/otctl fetchAllEscrowRecords
fetching escrow records failed: Error Domain=CKErrorDomain Code=15 "CKInternalErrorDomain: 2000" UserInfo={ContainerID=com.apple.security.keychain, NSUnderlyingError=0x6000019e40f0 {Error Domain=CKInternalErrorDomain Code=2000 "(null)" UserInfo={ContainerID=com.apple.security.keychain, CKHTTPStatus=500, RequestUUID=4130264A-AD5A-4970-88EF-622667C6553B, OperationID=682A09E938434541}}, CKHTTPStatus=500, NSDebugDescription=CKInternalErrorDomain: 2000, RequestUUID=4130264A-AD5A-4970-88EF-622667C6553B, OperationID=682A09E938434541}

My reversing didn't go that far to judge if it's related to absense of trusted peers or actual bug with CloudKit.

Log messages from trustedpeershelperd:

fetchViableBottles failed with error: <CKError 0x125636640: "Server Rejected Request" (15/2000); op = 3958A36F3B166393; uuid = 9C6FB698-E677-4B7C-A323-14121908371A; container ID = "com.apple.security.keychain">

fetchEscrowRecords failed with error: <CKError 0x125636640: "Server Rejected Request" (15/2000); op = 3958A36F3B166393; uuid = 9C6FB698-E677-4B7C-A323-14121908371A; container ID = "com.apple.security.keychain">

Is it CloudKit bug? Can I hope for rescuing my data?

The sad thing about it, that due to lack of knowledge, I anticipate this can be fatal data loss. E.g. if some part of secret chain was discarded by HSM which I presume is irrecoverable....

I have only partial backup of Cloud Drive. There are Photos which are really important to me, so it would be so nice to get it back.

At least if you can answer: if I can safely try resetoctagon this occasion as well, or if disabling ADP might help.

@eskimo if there is any chance you could comment, I'd appreciate a lot.

Kind regards, Peter

More logging:

is@airstation ~ % ckksctl fetch
Error fetching: Error Domain=CKKSResultOperationError Code=1 "Success-dependent operation failed" UserInfo={NSLocalizedDescription=Success-dependent operation failed, NSUnderlyingError=0x6000020d0090 {Error Domain=com.apple.security.octagon Code=30 "state became lose_trust, was expecting <OSTPath((
))>" UserInfo={NSLocalizedDescription=state became lose_trust, was expecting <OSTPath((

which seems to be ckks state machine breakage (?), as apparently it was expecting CKKSZoneKeyState to be SecCKKSZoneKeyStateHealTLKShares


I cannot find lose_trust in Apple OSS, but there is a string in /usr/libexec/secd, so you should know :)

Emphasising again, that any help to resolve actual issue: get e2e data back if possible, is very appreciated!

Kind regards, Peter

Possibly triggered CDP bug (can't re-establish Octagon Trust, ADP, facing potential fatal data loss)