Kernel Panics in Ca...
 
Notifications
Clear all

Kernel Panics in Catalina 10.15.4/Thunderbay 4

Page 1 / 10
(@zcutlip)
Active Member Customer

I'm wondering if anyone else is seeing kernel panics in macOS 10.15.4 with their Thunderbay RAIDs?

I can consistently produce a panic when I drag a ~1TB folder from an external USB volume to my RAID 5 in Finder. It usually takes a few minutes (maybe 5-10).

Below is a partially symbolicated panic log (the forum wouldn't accept it as an attachment for some reason).

panic(cpu 1 caller 0xffffff8002416487): "AppleAHCIDiskQueueManager::setPowerState(0xffffff8031702000 : 0xffffff7f839485d2, 3 -> 2) timed out after 151499 ms"@/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/xnu/xnu-6153.101.6/iokit/Kernel/IOServicePM.cpp:5296
Backtrace (CPU 1), Frame : Return Address
0xffffff921e16bb40 : 0xffffff8001d215cd mach_kernel : _handle_debugger_trap + 0x49d
0xffffff921e16bb90 : 0xffffff8001e5a3c5 mach_kernel : _kdp_i386_trap + 0x155
0xffffff921e16bbd0 : 0xffffff8001e4bf7e mach_kernel : _kernel_trap + 0x4ee
0xffffff921e16bc20 : 0xffffff8001cc7a40 mach_kernel : _return_from_trap + 0xe0
0xffffff921e16bc40 : 0xffffff8001d20c97 mach_kernel : _DebuggerTrapWithState + 0x17
0xffffff921e16bd40 : 0xffffff8001d21087 mach_kernel : _panic_trap_to_debugger + 0x227
0xffffff921e16bd90 : 0xffffff80024c2c7c mach_kernel : _panic + 0x54
0xffffff921e16be00 : 0xffffff8002416487 mach_kernel : __ZN9IOService12ackTimerTickEv + 0x6f7
0xffffff921e16be50 : 0xffffff8002415d69 mach_kernel : __ZN9IOService21actionAckTimerExpiredEP8OSObjectPvS2_S2_S2_ + 0x9
0xffffff921e16be60 : 0xffffff800242d2fe mach_kernel : __ZN10IOWorkLoop9runActionEPFiP8OSObjectPvS2_S2_S2_ES1_S2_S2_S2_S2_ + 0x3e
0xffffff921e16bea0 : 0xffffff8002414b18 mach_kernel : __ZN9IOService17ack_timer_expiredEPvS0_ + 0x38
0xffffff921e16bec0 : 0xffffff8001d63545 mach_kernel : _thread_call_delayed_timer + 0xec5
0xffffff921e16bf40 : 0xffffff8001d63071 mach_kernel : _thread_call_delayed_timer + 0x9f1
0xffffff921e16bfa0 : 0xffffff8001cc713e mach_kernel : _call_continuation + 0x2e

BSD process name corresponding to current thread: kernel_task
Boot args: -v keepsyms=1

Mac OS version:
19E266

Kernel version:
Darwin Kernel Version 19.4.0: Wed Mar 4 2240 PST 2020; root:xnu-6153.101.6~15/RELEASE_X86_64
Kernel UUID: AB0AA7EE-3D03-3C21-91AD-5719D79D7AF6
Kernel slide: 0x0000000001a00000
Kernel text base: 0xffffff8001c00000
__HIB text base: 0xffffff8001b00000
System model name: Macmini8,1 (Mac-7BA5B2DFE22DDD8C)
System shutdown begun: NO

System uptime in nanoseconds: 2468583176358
last loaded kext at 18397150200: @filesystems.smbfs 3.4.2 (addr 0xffffff7f83849000, size 454656)
loaded kexts:
com.owc.driver.SCSI-Device 1.1.0b16
com.owc.driver.SATA-Command 8.1.0b16
com.softraid.driver.SoftRAID 5.8.3
@filesystems.smbfs 3.4.2
@fileutil 20.036.15
@filesystems.autofs 3.0
>AGPM 111.4.4
>!APlatformEnabler 2.7.0d0
>X86PlatformShim 1.0.0
>!AWWANSupport1 4.0.0
>!AUpstreamUserClient 3.6.8
>!AMCCSControl 1.11
>!AGraphicsDevicePolicy 5.1.16
@AGDCPluginDisplayMetrics 5.1.16
>!AHV 1
|IOUserEthernet 1.0.1
|IO!BSerialManager 7.0.4f6
>!A!IKBLGraphics 14.0.5
>pmtelemetry 1
@Dont_Steal_Mac_OS_X 7.0.0
>!AFIVRDriver 4.1.0
>!A!IPCHPMC 2.0.1
>BridgeAudioCommunication 6.70.3
>!AAVEBridge 6.1
>!A!ISlowAdaptiveClocking 4.0.0
>ACPI_SMC_PlatformPlugin 1.0.0
>!A!ICFLGraphicsFramebuffer 14.0.5
>!AThunderboltIP 3.1.4
>!ABridgeAudio!C 6.70.3
>!AGFXHDA 100.1.426
>Core!SFsck 551
|SCSITaskUserClient 422.101.1
>!AWWANSupport 4.0.0
>!U!SCoexistentDriver 489.101.1
>BCMWLANFirmware4364.Hashstore 1
>BCMWLANFirmware4377.Hashstore 1
>BCMWLANFirmware4355.Hashstore 1
>!AFileSystemDriver 3.0.1
>!ABCMWLANBusInterfacePCIe 1
|!ABCM5701Ethernet 10.3.5
>!AVirtIO 1.0
@filesystems.hfs.kext 522.100.5
@!AFSCompression.!AFSCompressionTypeDataless 1.0.0d1
@BootCache 40
@!AFSCompression.!AFSCompressionTypeZlib 1.0.0
@filesystems.apfs 1412.101.1
>!AAHCIPort 341.0.2
@private.KextAudit 1.0
>!AACPIButtons 6.1
>!ASMBIOS 2.1
>!AACPIEC 6.1
>!AAPIC 1.7
$!AImage4 1
@nke.applicationfirewall 303
$TMSafetyNet 8
@!ASystemPolicy 2.0.0
|EndpointSecurity 1
>!UAudio 322.2
@kext.triggers 1.0
>!ASMBus!C 1.0.18d1
>!AGraphicsControl 5.1.16
|IOAVB!F 840.3
>!ASSE 1.0
|IO!BHost!CUARTTransport 7.0.4f6
|IO!BHost!CTransport 7.0.4f6
|IO!B!F 7.0.4f6
|IO!BPacketLogger 7.0.4f6
>!ASMBusPCI 1.0.14d1
@!AGPUWrangler 5.1.16
|IOSlowAdaptiveClocking!F 1.0.0
>!A!ILpssUARTv1 3.0.60
>!A!ILpssUARTCommon 3.0.60
>!AOnboardSerial 1.0
>!AHDA!C 283.15
|IOHDA!F 283.15
>IOPlatformPluginLegacy 1.0.0
>X86PlatformPlugin 1.0.0
>IOPlatformPlugin!F 6.0.0d8
@!AGraphicsDeviceControl 5.1.16
|IOAccelerator!F2 438.4.5
>!AThunderboltEDMSink 4.2.3
|IONDRVSupport 575.1
|IOGraphics!F 575.1
@plugin.IOgPTPPlugin 840.3
>Core!S 551
>usb.IOUSBHostHIDDevice 1.2
|IOATA!F 2.5.3
|IOUSBMass!SClass 4.0.4
>usb.!UHub 1.2
>!AThunderboltDPOutAdapter 6.2.6
>!AThunderboltPCIUpAdapter 2.5.4
>usb.cdc.ecm 5.0.0
>usb.cdc.ncm 5.0.0
>usb.cdc 5.0.0
>usb.networking 5.0.0
>usb.!UHostCompositeDevice 1.2
>!ABCMWLANCore 1.0.0
>IOImageLoader 1.0.0
|IO80211!FV2 1200.12.2b1
>corecapture 1.0.4
|IOSkywalk!F 1
|IOEthernetAVB!C 1.1.0
>mDNSOffloadUserClient 1.0.1b8
|IOAudio!F 300.2
@vecLib.kext 1.2.0
|IOSerial!F 11
|IOSurface 269.11
@filesystems.hfs.encodings.kext 1
>!AThunderboltPCIDownAdapter 2.5.4
>!AThunderboltDPInAdapter 6.2.6
>!AThunderboltDPAdapter!F 6.2.6
>!AHPM 3.4.4
>!A!ILpssI2C!C 3.0.60
>!A!ILpssDmac 3.0.60
>!A!ILpssI2C 3.0.60
>!AXsanScheme 3
|IOAHCIBlock!S 316.100.5
|IOAHCI!F 290.0.1
>!AThunderboltNHI 5.8.6
|IOThunderbolt!F 7.6.0
>!AEffaceableNOR 1.0
>usb.!UVHCIBCE 1.2
>usb.!UVHCI 1.2
>usb.!UVHCICommonBCE 1.0
>usb.!UVHCICommon 1.0
|IOBufferCopy!C 1.1.0
|IOBufferCopyEngine!F 1
|IONVMe!F 2.1.0
>usb.!UXHCIPCI 1.2
>usb.!UXHCI 1.2
>usb.!UHostPacketFilter 1.0
|IOUSB!F 900.4.2
>!AEFINVRAM 2.1
>!AEFIRuntime 2.1
>!ASMCRTC 1.0
|IOSMBus!F 1.1
|IOHID!F 2.0.0
$quarantine 4
$sandbox 300.0
@kext.!AMatch 1.0.0d1
>DiskImages 493.0.0
>!AFDEKeyStore 28.30
>!AEffaceable!S 1.0
>!AKeyStore 2
>!UTDM 489.101.1
|IOSCSIBlockCommandsDevice 422.101.1
>!ACredentialManager 1.0
>KernelRelayHost 1
>!ASEPManager 1.0.1
>IOSlaveProcessor 1
|IOUSBMass!SDriver 157.101.3
|IOSCSIArchitectureModel!F 422.101.1
|IO!S!F 2.1
|IOUSBHost!F 1.2
>!UHostMergeProperties 1.2
>usb.!UCommon 1.0
>!ABusPower!C 1.0
|CoreAnalytics!F 1
>!AMobileFileIntegrity 1.0.5
@kext.CoreTrust 1
|IOTimeSync!F 840.3
|IONetworking!F 3.4
|IOReport!F 47
>!AACPIPlatform 6.1
>!ASMC 3.1.9
>watchdog 1
|IOPCI!F 2.9
|IOACPI!F 1.4
@kec.pthread 1
@kec.corecrypto 1.0
@kec.Libm 1

Quote
Topic starter Posted : 02/04/2020 12:28 pm
(@softraid-support)
Member Admin

This is a bug introduced in 10.15.4.

We are working with Apple engineers to help find a fix for 10.15.5, or find a workaround.

ReplyQuote
Posted : 02/04/2020 4:22 pm
(@crhendo)
Active Member Customer

You have got to be kidding?

ReplyQuote
Posted : 02/04/2020 7:01 pm
(@cheule)
Active Member Customer

This is a bug introduced in 10.15.4.

We are working with Apple engineers to help find a fix for 10.15.5, or find a workaround.

I started getting sporadic “drive hangs” since upgrading to 10.15.4. No panics, but the RAID will become unresponsive. Either a file copy will hang, and never complete, or no data can be read off the drive. I can’t restart the Mac, because the drive won’t release.

Hasn’t been too frequent, but enough to be annoying. Usually once a day since upgrade.

1) do you think this is the same issue? Or should I do some troubleshooting?

2) is the any reasonable way to downgrade to 10.15.3?

ReplyQuote
Posted : 03/04/2020 1:31 am
(@softraid-support)
Member Admin

Same bug.

I do not know of an easy way to go back, as the "snapshot" feature which made this easy using Time Machine apparently is also broken.

Another reason to stick with Mojave or older!

I always stay a little behind, it pays off in cases like this, to not be running Catalina in my main working computer.

ReplyQuote
Posted : 03/04/2020 12:23 pm
(@softraid-support)
Member Admin

You have got to be kidding?

Sad, and frustrating. Hopefully this will be fixed very quickly, or we can find out what Apple changed in the IO kit that is causing this, and work around it.

ReplyQuote
Posted : 03/04/2020 12:25 pm
(@zcutlip)
Active Member Customer

This is a bug introduced in 10.15.4.

We are working with Apple engineers to help find a fix for 10.15.5, or find a workaround.

Good to know. Thanks for the quick reply.

ReplyQuote
Topic starter Posted : 03/04/2020 7:39 pm
(@cheule)
Active Member Customer

Same bug.

I do not know of an easy way to go back, as the "snapshot" feature which made this easy using Time Machine apparently is also broken.

Another reason to stick with Mojave or older!

I always stay a little behind, it pays off in cases like this, to not be running Catalina in my main working computer.

I downgraded to Mojave. I'm tired of dealing with Catalina incompatibilities when this is just a file server. (I have Catalina on my MacBook Pro and love it).

For anyone else thinking of downgrading, I used this link: https://www.digitaltrends.com/computing/how-to-downgrade-from-catalina-to-mojave/

ReplyQuote
Posted : 04/04/2020 3:09 pm
 rent
(@rent)
New Member Customer

Am I understanding correctly from this thread, that there is a known (quite serious?) issue with 10.15.4, 5.8.3 (XT), and Thunderbay 4?

Should I even attempt to get things set up with the above combination? Or downgrading to 10.15.3 is the only path forward at this point?

ReplyQuote
Posted : 05/04/2020 11:15 am
(@softraid-support)
Member Admin

There is a serious issue with 10.15.4.
It shows up in different scenarios, even on Apple disks but is more likely when there are lots of IO threads. We think it is a threading issue. So while SoftRAID volumes are hit the hardest (its now hard to copy more than 30GB of data at a time), all systems are impacted by this.

In our bug report to Apple, we used a method to reproduce the problem with ONLY Apple formatted disks. Takes longer to reproduce, but that is more likely to get a faster fix to the user base.

I hope Apple will get a 10.15.5 out quickly to fix this. Except for downgrading with a previously downloaded installed, it is very hard. (Time Machine is broken where "Snapshots" were not working, so the magic "go back" is not working)

As far as I know, there is no way to download a fresh 10.15.x, except .4.

If you can go back to 10.15.3, I recommend it, but it is difficult.

Other things are broken also, such as Time Machine, ssh have introduced bugs.

ReplyQuote
Posted : 05/04/2020 1:04 pm
 rent
(@rent)
New Member Customer

Thanks for the prompt clarification. I guess I'm stuck with 10.15.4 for now. :-(

ReplyQuote
Posted : 05/04/2020 3:04 pm
(@bob_stan)
Active Member Customer

I routinely copy two approximately 500GB files from a soft raid striped ssd to a 20TB data volume of soft raid 1+0 in a thunderbay4 enclosure. I am on Catalina 10.15.4 and have had no issues.

ReplyQuote
Posted : 06/04/2020 1:43 pm
(@softraid-support)
Member Admin

Good!

The issue is less frequent on RAID 0, RAID 1+0, Apple RAID 0, but does occur.

It is much quicker to occur on RAID 4/5 volumes.

ReplyQuote
Posted : 07/04/2020 12:54 am
 sco
(@sco)
Active Member Customer

So far haven't experienced it here on RAID1 here either (touch wood). Is the problem more likely to occur after some time.. or to put it another way... would a regular reboot reduce the likelihood of it occurring? Or is it simply a race condition triggered by concurrent I/O?

Thanks,

sco

ReplyQuote
Posted : 07/04/2020 6:16 am
(@softraid-support)
Member Admin

It seems to be a race condition, yes. Rebooting is unlikely to have any effect.

It is much rarer with non RAID HFS volumes and simple RAID levels RAID 1 and 0. You may not see it ever.

ReplyQuote
Posted : 07/04/2020 11:19 am
Page 1 / 10
Share:
close
open