SOLUTION FOUND! (for me anyway, hope it works for you)
Ok, so I’ve never studied any of this stuff, so I’m sure there will be parts here that are amateur or sloppy but these are the steps that ultimately worked.
I had the same problem of an interrupted Big Sur upgrade, and suddenly an encrypted hard drive despite never having set up encryption, an encryption password I obviously didn’t know, a user passphrase that was being accepted at the log-on screen, but not in other places after it would hang and try to enter recover, and a machine that wouldn’t boot and I thought was toast.
Unlike Kirill, the unlocking the volume step in my Terminal did NOT accept my offered password.
The solution was a combination of things from this forum, the decryption commands from the page Kirill referenced with Terminal commands, and a key partition deletion step figured out by a guy named Kyle Kukshtel on his blog - search "MacOS APFS FileVault Permissions De-Sync During High Sierra to Big Sur Update - kyle kukshtel's game dev/design blog"
Here were my steps, including some fails if they’re helpful to avoid:
I first created a bootable usb installer for Catalina, primarily to use and be expectedly directed to recovery mode, where I then had critical access to Disk Utility and Terminal (whereas I hadn't before). [Catalina was what I could get ahold of, as I had some limitations with the other Mac I was using to make the drive]
In Terminal, I referenced decryption command instructions from the site Kirill cited. The volume with the data, disk1s1 for me, was listed as both encrypted and locked (diskutil apfs list)
Unlike Kirill, when I tried to unlock it with my user passphrase, it didn’t work. (diskutil apfs unlockVolume /dev/apfs_volume_id_goes_here) Said the passphrase was incorrect.
I tried to go ahead and decrypt it on a whim. Got the crypto user name per the instructions (diskutil apfs
listcryptousers /dev/apfs_volume_id_goes_here) and put in the decrypt command, but it gleefully told me it needed to be unlocked first as suspected.
Next I went over in Disk Utility, where I had ‘Macintosh HD’, ‘Macintosh HD data’, and ‘Update’ partitions listed, plus
the usb I had booted from.
'Macintosh HD data' was unmounted, greyed out. When trying to mount it, it asked for the user password and did not accept the correct password on several tries, similar to other places. It even gave the Hint option, the hint therein telling me I definitely had the right password.
This was the first Key step* As suggested in the blog's solution, I ERASED the Update partition in Disk Utility. It automatically rebuilt itself. I then erased Macintosh HD (NOT Macintosh HD data), and it rebuilt itself as well. (At this point, I
was ready to fail and wipe it, so it was ‘what do I have to lose’.) Suddenly, however, on trying to mount Macintosh HD data again, it accepted the password and mounted! Somehow, erasing and allowing the other partitions to rebuild allowed things to re-sync somehow?!?
I immediately created a disk image of the hard drive and put it on my external HD. I wasn’t sure whether the image data would be encrypted or not, but this was the first opportunity I had to back anything up before going further, so I
did.
I decided to next let the computer restart and see if resetting those partitions somehow would allow it to boot,
or continue the software update, recover, anything different. No dice. (would later turn out to be a mistake/barrier that had to be fixed).
Meanwhile, I attempted to load the backed up image on my older Mac. After a painfully long verification, it
informed it couldn’t load the files referencing something about the file system. The two computers aren’t on the same macOS version, and it’s much older to be sure, but I suspected it was more to do with the encryption.
I restarted again, booting back from the usb installer once more. I tried the Terminal operations again, thinking maybe
now the password would be accepted to unlock the drive. Alas it was not, couldn’t be unlocked, and again couldn’t be decrypted.
I went back into Disk Utility. Curiously, there were now two Macintosh HD’s listed, two greyed out data partitions, ‘Macintosh HD data’ and ‘Macintosh HD data data’, and two update partitions. Macintosh HD data data was the one showing the significant amount of storage, the rest had a few kb’s each. Infuriatingly, though, all of a sudden, trying to
mount it gave the same problem of not accepting the correct password typed carefully multiple times. What the heck?
I erased the HD and Update partitions again (note, it turned out I needed to erase BOTH copies of the two other partitions this time – four total before it would work), and all of a sudden, the password took again and the data partition mounted. [Side note – I was desperate to see if there was some other functionality in Disk Utility that would allow me to move data around. By attempting to open a file image, I was actually able to see ALL of my data/files/documents in the ‘data data’
partition via the ‘Open’ dialogue box. But the functionality did not exist to simply drag them over to my waiting USB, which was also shown in the box; wouldn’t that have been nice? Granted, seeing the file names didn’t likely mean they weren’t still encrypted. But it was a cruel tease seeing them laid out...]
OK, so I quit Disk Utility and got back into Terminal – THIS TIME DURING THE SAME REBOOT as having mounted the partition in Disk Utility with now re-accepted password.
The volume was now suddenly listed as Unlocked!
I ran the command to see the cryptousers list again, copied the long string for the user name, and attempted to decrypt
once more, hoping I could now get access with the passphrase for the user for the volume. SUCCESS! It accepted the user passphrase as sufficient and started decrypting!
I monitored the decryption progress by running [diskutil apfs list] every now and then. The thing took a couple of hours at least to complete. I didn’t dare interrupt.
When it finished the data volume was listed as ‘FileVault: No’ !
I went back into Disk Utility and went ahead and ran First Aid on the newly decrypted ‘data data’ to be complete. There were no problems. I made a new disk image back-up to replace the prior one I had put on my external HD.
Restarted it again bringing up the startup disk options and told it to boot from ‘data data’. It booted up to the desktop and now appears to work, with full access to data, programs, etc.
Hope that is helpful to those of you still sweating this problem! Thanks to everyone on the forum for being part of the solution! (HT to Kirill in particular)
