Thunderblade won't mount, stuck on "rebuilding, waiting for mount"

Ventura 13.4

@baaron 

I guess you installed 13.5 beta in the middle of your project?
this is the bug introduced that affects M2 2023 computers, and the same issue is in the 13.5 beta's, for other M1/M2 computers.

the problem is mounting now, I am guessing the panics damaged the directory.

You did the right thing by connecting to another computer, sometimes the auto repair can work on another system. Since it did not, then your options are limited.

You won't like my answer, but I think running R-Studio or Disk Drill to recover critical files to copy off is your only solution. On a volume this large, it may take hours to scan the volume for recovery, unfortunately.

Very bad news, I am afraid. the only other option is connect again to the original, then the second, wait ten minutes and go back and see if the MacOS repair can fix this. Sometimes it can. You can also try "First Aid".

There are no known third party utilities that can repair APFS volumes at this time.

softraid screenshot.  stuck here

@baaron 

Same answer applies.

@softraid-support 

I mistyped!  No, I've been on 13.4 the entire time.   Had the machine running about 2 weeks just fine. 

What could have caused this?   We bought 3 Thunderblades for our Mac Studios. 

Are we safer doing HFS Raid 5?   Or just go Apple Disk Utility Raid 0 and do daily backups? 

@baaron 

I got 13.5 from your early line, maybe a typo:
M1 Ultra. Venture 13.5. Softraid 7.5. APFS. Raid 5

The bug causing panics is in 13.5 or 13.4.1 for M2 machines. I do not think it can affect the M1 Studio, but I guess its possible.

Are you still getting the panics? I would like to see the panic report if so. Copy/paste the text in report to Apple/details, into text Edit, save, then "Make plain text" and save again. You can attach that file here.

Until 13.5 is out, I would use Apple RAID. It does not support TRIM, but short term, it will be fine.

Correct, 13.4 the entire time on an M1 machine.   (I was thinking 7.5 Softraid in my head, hence mistyping 13.5)

No kernel panics.  Thunderblade simply won't mount.   I've moved it back and forth between Intel and M1, same situation for both. 

@baaron 

When your system shuts down, MacOS closes the volume directory on all mounted volumes, then unmounts them. At startup, it checks to see if the volume was unmounted correctly. (called the dirty bit)

If MacOS detects there was an unclean mount/shutdown, it automatically runs a repair tool to fix the directory. if this tool fails, the volume is marked read only and mounted (if possible). Sometimes the volume cannot be remounted and MacOS can no longer mount the volume.

Sadly, there is no fix/repair available.

In HFS, there is an app called Disk Warrior that can repair volume directories, but Apple's "First Aid" is limited and cannot repair any significant issues with volume directories. So when an APFS volume is damaged and cannot mount, the only option is restore from backup, or run a Data Recovery application, the two we have tested to work on APFS are Disk Drill and R-Studio.

Ok, so:

If we ever force reboot in a messy shutdown, we're gonna lose our APFS Thunderblade? And there is no fix?  

If we HFS it will still be broken? But Disk Warrior might fix it?  (Disk warrior is going back decades ago for me....)

@baaron 

This is the case for all APFS volumes, but probably 4x more frequent on RAID volumes, yes (assuming 4x the number of drives, the probably of an issue is 4x also). Even "Apple RAID". I have an Apple RAID volume in the lab here, that was damaged after a couple kernel panics induced in testing. I have been keeping it in case an Apple engineer wants to investigate, with the purpose of making better repair tools for APFS.

At times, APFS can auto repair, on another computer, which I do not understand, or after a few restarts.

Until Disk Warrior adds APFS support, there are recovery apps, some which are pretty good, but they can only copy the data off, they cannot repair.
I know MicroMat (TechTool Pro) has been trying to add APFS repair, but I don't have any data or user experiences to reference yet.

What is interesting, is APFS appears rock solid on internal devices. Its only external that I have seen these kinds of issues. Even on non RAID, I have seen unmountable volumes. I suppose it is much less rare than HFS volumes, but HFS you can repair.

Thanks for the follow up, I've learned a lot. 

We're going to offline these drives as they're not safe in a production environment. 

@baaron 

Not sure I agree that they are not safe for production, we have hundreds of Studio's/production teams using Thunderbays.

I appreciate the feedback, however. Most users never experience directory damage on their volumes. A few have had more than one instances. For most this is a rare, one off event, however.
Its my experience that this has to do with workflow, other installed apps, shutdown techniques, and a hundred other variables. Normal operations, volumes should be fine. The main issue users complain about with HFS volumes, at least, is slowdowns, where it takes a long time for folders to open, etc. On RAID or non RAID, doesn't matter. But this is where Disk Warrior is fantastic.