[Sticky] Monterey Crash/kernel panics

@sp0913 

Please attach panic logs as attachments (Save as plain text in Text Edit has here)

this is the kernel panic on m1 with RAID 5 that Apple has fixed in Ventura beta's.

Mot users can avoid this by waiting 5 minutes before connecting the enclosure after startup, or removing one disk slightly, then pushing it in after 5 minutes.

Or you can create the volume again selecting 64k Stripe unit sizes.

Posted by: @softraid-support

@jmax313 

This is an ARM, correct?

This is a DART chip bug/crash, triggered by RAID 5 volumes.

Solutions are:

For most users with this issue, waiting 5 minutes after restart, avoids this.

Remove cover, pull out one disk, then after 5 minutes, you can push the drive in and it should not cause the panic

Or, delete volume, create it again as RAID 5 with 64 k Stripe unit size

Apple is finally fixing this in Ventura, macOS 13 in the autumn.

 

If I delete the volume won't it delete my data? I can not even connect the drive without causing a kernel panic... 

Edit: alright I had removed all the drives and put them in one by one, so it seems like removing one of the drives upon startup should *hopefully* be my workaround for now. I'll patiently await until a kernel fix on apples end happens. 

@jmax313 

Yes, removing one disk at startup lets you use the volume ,then plug in disk 4 after a few minutes.

At least Ventura seems to fix this permanently.

I'm having some consistent kernel panics that feels related to softraid/thunderbay potentially.  Mac studio with a thunderbay 8, running 8 drives split into a RAID 10.  Just setup the raid and then started copying files over and have gotten kernel panics overnight each night during copying.  See attached kernel panic details.  What's curious is that after reboot the TB8 drives don't show up.  I check in terminal and soft raid driver isn't loaded.  I reinstall and then can get the TB8 working but after next kernel panic again after the reboot TB8/drives don't show up and again soft raid driver isn't loaded when I check.  I originally followed the direction to enable the third party and allowed the developer in security prefs.  When I resinstall driver I don't get an option to allow again.  Fact driver isn't loaded after the kernel panic restart makes me suspect something is wrong with soft raid end.  But would that explain why the TB8/drives don't at least show up/mount?  Maybe it would since they are in a soft raid managed array?

edit: looks like it won't let me attach the .txt files of the kernel panic details??  In meantime here's some high-level details:

Panic #1

panic(cpu 1 caller 0xfffffe002971b554): "apciec[pcic1-bridge]::handleInterrupt: Request address is greater than 32 bits linksts=0x99000001 pcielint=0x00220020 linkcdmsts=0x00000000 (ltssm 0x11=L0)\\n" @AppleT8103PCIeCPort.cpp:1299\

Kernel Extensions in backtrace:\
com.apple.driver.AppleT6000PCIeC(1.0)[F9B3C740-E3B0-3BE4-94FA-B8F30680C6D8]@0xfffffe0029711c50->0xfffffe002972ad1f\
dependency: com.apple.driver.AppleARMPlatform(1.0.2)[911D503A-285D-36C8-992A-139A9297AA6E]@0xfffffe0028a32640->0xfffffe0028a7ccdb\
dependency: com.apple.driver.AppleEmbeddedPCIE(1)[290FB383-0DD0-3638-A640-443D72E5062D]@0xfffffe0029003e70->0xfffffe00290140e3\
dependency: com.apple.driver.ApplePIODMA(1)[3EE93096-03CF-3844-9D99-311FBFEB90C3]@0xfffffe0029412dd0->0xfffffe0029417667\
dependency: com.apple.driver.IODARTFamily(1)[63CB9EC5-CA95-3F3E-9C8C-59792CE63ED6]@0xfffffe002a035ae0->0xfffffe002a0493e3\
dependency: com.apple.iokit.IOPCIFamily(2.9)[37788442-2C8F-3CFD-8216-1553867CF69C]@0xfffffe002a3ae070->0xfffffe002a3ccc13\
dependency: com.apple.iokit.IOReportFamily(47)[D58BD9DF-0E66-3130-810B-E748087748BF]@0xfffffe002a3ccc20->0xfffffe002a3cfc43\
dependency: com.apple.iokit.IOThunderboltFamily(9.3.3)[6190C4F0-28BF-3625-B2AF-6BB010729CF1]@0xfffffe002a4ce090->0xfffffe002a5f1cfb\

Please attach the panic logs. What you missed is the step in text edit to "Make Plain text" (command shit t) in format menu.

They may look like text files but are .rtf files in fact.

SoftRAID is not in the backtrace, whatever is causing this.

This could be a lot of things, lets take a look.

Also attach a SoftRAID tech support file.

@softraid-support 

I'm positive I have these files in plain text but the page refuses to attach.  Have tried multiple applications to create plain text including a separate windows machine.  No clue what's wrong.  Here's the tech support file but it might not be super useful because it was run after a restart and the drives or raid volume don't show as mounted.

Last night I did a reformat and fresh install and installed nothing else on the computer.  Initialized and setup a new volume and started copying and again a kernel panic overnight, looks very similar to the two above.  I also used a different thunderbolt cable.  Same behavior where after the kernel panic the drives don't even show up.  Not sure if there is something wrong with the thunder bay or the computer or with soft raid.  I'm going to try setting up the raid array on a different computer tonight and re-running the test.  It's an M2 air I plan to try with tonight, although I'm trying to secure a friend's Intel MBP to test, part of me feels like that might be more robust test case.

@getalot681 

Use Text edit. In the Font menu, "make plain text". then you can attach it.

Is this RAID 1+0 volume the only SoftRAID volume you are testing on?

There are no known panics on RAID 1+0, so there is likely a hardware issue somewhere, I would indeed like to see the panic report.

Question: Your system shows "Activation lock" enabled. This is the switch that enables third party developer drivers to load.
Set when you startup in System Security mode, options, "set System Security" to reduced and enable third party developers.

Did you set this? Something is odd here, as if it is enabled (not set to reduced), your volume should not be able to mount.

@softraid-support 

I am using textedit (and notepad on windows), no clue why I can't attach, will try in a bit with chrome.  In meantime let me pass along some more info because yes I think something weird is going on.  Trying to put together a rough timeline of all the troubleshooting I've done.

a. I had the Mac Studio and Thunderbay 8 (TB8) with 5 drives running for ~5 days doing multi-pass drive certification.  Never had any issues.

b. Setup a RAID5 on those 5 drives.  Started copying over files and didn't have any issues.  I decided to bail on raid5 and switch to a raid10 to take advantage of write speeds given a thunderbolt connection.  I procured three new drives and ran certification on those new three drives without any issues.  This certification is a lot of I/O to the TB8 and drives over multiple days so in my mind this limits any broken 'hardware' since it was exercised without issues.

c. I then setup the raid10 on the 8 drives and started copying data.  A couple hrs into this is when the kernel panics happened.  Each time when the computer restarted the TB8/drives are not visible.  I would have to reinstall the driver to see even the drives, not just the raid volume.  I would restart and again the TB8/drives not visible and need reinstall driver.  I never got any popup to allow in security for the subsequent driver installs.

d. After all this I connected the TB8 to an Intel MBP last night.  Brand new fresh Monterey install (12.5.1).  I initialized all the drives and setup a fresh raid10 and started copying files over.  A few hours later kernel panic happened.  The kernel details down below because again I can't attach even though the file type is for sure correct.  Different message but guessing this is Intel vs AS.

e. One thing I noticed afterwards is that after the kernel panic restart the soft raid driver is not loaded because it has the same behavior where drives/volume in TB8 don't show up.  If I reinstall the driver they show up but I don't get the allow security pop-up.  Note that they show up before a reboot.  If I reinstall driver and then reboot the driver isn't loaded after restart.  In another thread I found the quarantine check and it does show as quarantined.

f. On both the Intel MBP and the Mac Studio after a kernel panic I have never been able to have the driver stay loaded after reboots.  The only way is to perform a fresh macOS install and setup from scratch.  I did check on the Mac Studio system that the special allow third party driver is still checked so that's not the issue.

I have a few thoughts.

-Could be a hardware problem.  I think it's unlikely the Mac Studio since the MBP had the same problem.  Could be the TB8 or the drives or cable.  I tried a different cable so likely not the cable.  In my mind the days and days of certification I/O and no errors eliminates the TB8 and the drives.  The only thing that's different with a certification and a raid is the RAID which is handled in the Softraid software which just passes I/O to the TB8/drives.  So my gut feeling is soft raid issue.  It's also curious to me why the driver gets quarantined after the kernel panic if it wasn't in the trace path?

-Only other thing I can think of is that between (b) and (b) above I did install 12.5.1.  Could it be that 12.5.1 changed some behavior or interaction with the soft raid 6.3 driver?

-Could there be some issues with RAID10 in the driver that would explain it working for a few days with RAID5 with no issues before I expanded the drives?  I'm currently installing macOS fresh to get driver working and then going to try setting up a RAID5 and let it copy and see if I can repeat the kernel panic with RAID5.  I'm doubtful it will be a different result but will try.

@getalot681 

Any time the drives do not show up in the left disks column in SoftRAID, it is by definition a hardware issue.

Reproduce this. Look at the enclosure and see if any/all of the lights are flashing, even lightly.

Try the test again with RAID 0, HFS. Can you reproduce it?

If you can, then you can "convert" the same volume to Apple RAID, (using SoftRAID) where I predict the same issue occurs.

After a crash, if the SoftRAID driver is not loading, what is the message, if any, when you run this in terminal:

sudo kmutil load -p /Library/Extensions/SoftRAID.kext

I finally managed to get 6.3 to run (no idea how, it just decided to start working yesterday without me changing anything). So I've now got the 6.3 driver installed. I wanted to see if it would allow me to use my backup volume for Time Machine yet, and it was sort of successful, but also crashed a couple of times (I've no idea if that is specific to me or if Time Machine just doesn't work with a softraid volume because it never has for me, always kernel panics - frustratingly couldn't get the crash reports either). I've done a couple of backups now, but this morning, out of nowhere, another kernel panic. I've turned time machine off again hoping that resolves the crashes. Any idea what is causing this? I don't understand these intermittent seemingly unprovoked issues I'm having. Is it still due to the apple I/o controller?

@jjdodders 

I assume your SoftRAID volume is APFS? (Time Machine requires APFS in Monterey now)

Did you select to encrypt the volume in Time Machine? that could cause panics.

Posted by: @softraid-support

@getalot681 

Any time the drives do not show up in the left disks column in SoftRAID, it is by definition a hardware issue.

Reproduce this. Look at the enclosure and see if any/all of the lights are flashing, even lightly.

Try the test again with RAID 0, HFS. Can you reproduce it?

If you can, then you can "convert" the same volume to Apple RAID, (using SoftRAID) where I predict the same issue occurs.

After a crash, if the SoftRAID driver is not loading, what is the message, if any, when you run this in terminal:

sudo kmutil load -p /Library/Extensions/SoftRAID.kext

Blue light is on consistently, green drive lights only blink when being accessed.

I'm going to try with apple RAID0 first, test in progress.

No message when running that in terminal.  I check afterwards and it shows the driver loaded.  However if I reboot the driver is no longer loaded.  It seems that is a separate issue from hardware issue, I cannot keep the driver loaded.  It was convoluted with what I did above but I also feel like I cannot see the bare drives without the driver loaded.  After these tests I will verify that with a fresh macOS install and verify I can see bare drives without any soft raid installation.

@getalot681 

Seeing the bare drives (such as in the SoftRAID disks column) is purely a hardware function, whether macOS can see the disks. the SoftRIAD driver only tells MacOS that it controls (SoftRAID formatted) disks.

If SoftRAID cannot see a disk, then if you run "diskutil list" in terminal, it will not show the disk either.

there may be some problem that correlates to this, but SoftRAID has no ability to influence whether the Mac can see a disk or not.

Let me know if a clean install helps.

I should start by saying that I have been a loyal OWC customer for years - I've owned countless portable and external drives, hard drives, memory, etc. I LOVE my 4M2 - its size, speed, everything. 

I've recommended OWC products to countless friends, colleagues, and clients.

I'm also having the same kernel panic described by many others here.

I purchased my new MacBook pro this week, and I'm upset that I had to spend an additional $108 to upgrade software that isn't working. I'm even more upset that I can't access my work files unless I use my old machine, which I'm supposed to be trading in. 

I can only imagine how frustrated the users on here who have been dealing with this issue for a full year now. My new laptop is essentially a costly paperweight because I can't use it with my working files, which I keep on my OWC RAIDs. 

It doesn't seem like the 64 kb stripe is a permanent or workable solution, nor is the wait five/remove a drive.

I have three OWC enclosures running RAID 5, and having to reformat them feels unreasonable. 

Here are some questions I have:

• Do we have any guarantees that Ventura will solve these issues? Has anyone tested the beta successfully?
• Do I make my RAID 5s into JBODS so I can keep working and hope that Ventura fixes this?
• Why is this Apple's issue to fix and not OWC?
• Do I look for a solution elsewhere? I would hate to have to move on to another company, but I need to be able to do my work. My livelihood and income depend on it.

Thanks - I hope someone here can help and provide some clarity. 

@orion_quest 

So far, it appears Ventura is fixing most all of these crashes. I hope it is 100%, but since it is a macOS change, and we do not know what the fix it. Apple engineering appears confident that this is fixed.

Any RAID level bug RAID 5 appears immune to this crash. the enclosure is just a box with drives, so yes you can set up any RAID you want, or non RAID volumes.

We figured out long ago this was a macOS issue. When we shipped a crashing system directly to Apple, we received an acknowledgement that this was a bug in MacOS/Apple Silicon.

Hope this helps explain this issue.