[Sticky] Monterey Crash/kernel panics

Posted by: @softraid-support

@orion_quest 

So far, it appears Ventura is fixing most all of these crashes. I hope it is 100%, but since it is a macOS change, and we do not know what the fix it. Apple engineering appears confident that this is fixed.

Any RAID level bug RAID 5 appears immune to this crash. the enclosure is just a box with drives, so yes you can set up any RAID you want, or non RAID volumes.

We figured out long ago this was a macOS issue. When we shipped a crashing system directly to Apple, we received an acknowledgement that this was a bug in MacOS/Apple Silicon.

Hope this helps explain this issue.

Thank you for your prompt reply!

I looked over the Ventura-beta thread on the subject, and it's not reassuring me, but I'm hoping you're aware of information beyond what's available in that thread. 

In theory, Ventura should be out in the next month or so, and I can maybe wait, but if I opt to go a different route perhaps you could offer me some guidance regarding my usage and what would be most beneficial, RAID 0, 5, or JBOD?

I'll start a separate thread, so I don't clutter this one, but maybe there are other people with similar needs to mine who will benefit from my questions. 

Thanks!

@orion_quest 

For video work, RAID 0, with highly diligent backups is probably best.

Posted by: @softraid-support

@getalot681 

Seeing the bare drives (such as in the SoftRAID disks column) is purely a hardware function, whether macOS can see the disks. the SoftRIAD driver only tells MacOS that it controls (SoftRAID formatted) disks.

If SoftRAID cannot see a disk, then if you run "diskutil list" in terminal, it will not show the disk either.

there may be some problem that correlates to this, but SoftRAID has no ability to influence whether the Mac can see a disk or not.

Let me know if a clean install helps.

I generally agree with you that the drives should be visible but something is definitely fishy here.  I haven't done the fresh macOS install to debug the drive visibility but I did complete the RAID0 testing.

I created 8 drive RAID0 using APPLE raid utility (stripe 64kb).  I was able to copy over 30 TB of data to the RAID0.  This was at the same roughly transfer rates as my prior RAID10 ~600MB/sec so definitely apples to apples.  I never made it past 8TB previously before I had a kernel panic and I had many that failed <2TB.

I then wiped it and created 8 drive RAID0 using SOFTRAID (stripe 64kb).  Kernel panic after about 600GB transferred.

Attached kernel panic details.  Again now after the kernel panic the soft raid drive is not loaded and to the point quoted above I do not see any drives in the enclosure.  This is the same behavior I've seen where I think the drives show fine but once soft raid driver is installed and I get the kernel panic then the drives won't show.  I have to reboot and leave soft raid uninstalled.  To me the test above though is pretty clear that soft raid is an issue.  I've had soft raid kernel panic every single time I've tried this transfer at least 10 times now and majority at less than 2TB so the fact that I could do the full 30TB transfer without any issues on APPLE raid I think is statistically significant.  And again the write speed is the same between all the tests so it's not like the RAID0 went faster and so less susceptible to some time based failure.

I will do the fresh macOS troubleshooting of the drive visibility today.  I'm very frustrated having sunk so much time into this.

@getalot681 

I wonder if there is another issue. the panic starts with this:

panic(cpu 1 caller 0xfffffe001d7cb5a4): "apciec[pcic1-bridge]::handleInterrupt: Request address is greater than 32 bits linksts=0x99000001 pcielint=0x00220020 linkcdmsts=0x00000000 (ltssm 0x11=L0)\n" @AppleT8103PCIeCPort.cpp:1299
Debugger message: panic

SOftRAID is not in the backtrace, but the DART controller is. the main difference in the Apple vs SoftRAID driver here is multi-threaded I/O. Both function the same (they are both essentially filter drivers, meaning macOS does the actual IO, the drivers are like traffic cops, directing traffic) But the Apple driver is single threaded (one IO at a time), while SoftRAID's driver is multi-threaded.

Can you attach a SoftRAID tech support file? Let me talk to engineering about this. Since I clearly do not see this in our labs, I wonder if this is a faulty enclosure, perhaps. (the clean install would help confirm or deny this hypothesis)

Posted by: @softraid-support

If you experience a kernel panic/crash with Monterey, please include a text file with the crash log.

When your system restarts, click "report to Apple". then click "More info" or "details". In the next dialog box that comes up, copy all the text and paste it into a text edit file. Under format, "Make plain text". Save the file, then rename the suffix to .txt and post it here. Also include a SoftRAID Technical Support file.

Note: if you are experiencing a kernel panic on an M1, connecting a RAID 5 volume, try using the current SoftRAID beta version and report back.

http://www.softraid.com/sr_beta

I forgot to attach a text file of what was sent to Apple. However I can send you the text from the Crash Report log if you like. I had my first kernel panic after getting my new MacBook Pro CTO 2021 and configuring the SoftRAID external to it via a T4/T3 connection. Wondering if you think I should use the beta version of SoftRAID with MacOS Monterey 12.5.1 release instead of version 6.3. Your release seems OK in all other aspects, but I have read here and elsewhere about kernel panics with M1. 

Suggestions? 

@henry-in-florida 

There are no changes to the beta that would assist here. Yes, send the panic log (in text edit, "make plain text" in the format menu, so you can attach it.

@SoftRAID Support

Here you go. I wonder why the Identifier in the Crash Log is called "bird"?

I dont know the bird thread. I found this from the Apple discussions back when Linc Davis was the main helper for users.

https://discussions.apple.com/thread/6815037

See if deleting the cache as suggested helps.

I've started having kernel panics again, completely out of the blue. This one is really baffling me because I haven't had any for weeks, I've had my Thunderbay 8 running smoothly for the whole time, haven't installed anything new, running SR6.3. But completely unprovoked, the system started to not respond as all 8 drives in the TB8 started grinding loudly for ages. Couldn't wake the system from screensaver. seemed to be using 100% cpu. Then after 10-15 mins it crashed. Rebooted. kernel panic. Then again and again. I've unplugged everything so I can get my Mac to just stay on. Attached is the panic log. Any ideas?

@jjdodders 

this is the dreaded DART panic. If you wait 5 minutes before connecting the disks, do they work?

(or remove one disk, then inserting it after 5 minutes)

Thankfully Ventura seems only a couple weeks away.

@softraid-support I thought it might be. But I’m just so confused as to why it would suddenly start panicking again after working fine for weeks. 

I tried disconnecting the thunderbolt cable and left it for quite a while and as soon as I plugged it in again it crashed. I think spotlight was doing an index scan which might be why there was a 100% cpu load on it but that’s never happened before. Can’t think of why that suddenly triggered. 

any other tips?

@jjdodders 

Remove one drive, then connect. After 5 minutes you should be able to reinsert the drive.

@softraid-support unfortunately that doesn’t work for me. Removed a disk, plugged it in and it crashes right away. Even more than 5 mins after boot up. 

@jjdodders 

Removing a disk almost always works. Sorry about this. Can you wait for Ventura? (two weeks probably)

@softraid-support Seems like I might have to wait. Do you think reinstalling SoftRaid/Drivers would help?