Notifications
Clear all

RAID5 Failed Disk Help

rexbinary
(@rexbinary)
Eminent Member Customer

I have a Thunderbay 4 with 4x4tb disks installed, and I have two RAID 5 volumes that span those disks. A few days ago I got a warning that a disk would fail in the next 2-8 months so I ordered a new drive that day. Today the drive went from 8 reallocated sectors to 7,808. The new drive won't be here until Wednesday. What should I do in the mean time? Can I remove this drive from the array and run without it until the new one arrives. Neither volume is even half full. Thanks!

rexraid

EDIT: I seldom post without an edit.

Quote
Topic starter Posted : 16/05/2022 10:29 pm
(@softraid-support)
Member Admin

I would wait. Don't pull the drive yet. the problem is you may have two disks out of sync. then you will have to recreate the volume and restore from backup, as SoftRAID cannot rebuild when two disks are out of sync.

Clear the iO error counters on the other drive.

You should see if you can "validate" the volume, which will update all parity blocks. its possible the system may crash, depending on how fast the other drive is failing. It is in bad shape, but may last through a validate.

Then you when get the new drive, remove the failing one, initialize the new one with SoftRAID, and "Add disk" to rebuild it into the volume.

ReplyQuote
Posted : 17/05/2022 12:18 am
rexbinary liked
rexbinary
(@rexbinary)
Eminent Member Customer

So I performed the steps you suggested with the Volume ThunderBay 4 and it said it rebuilt the Volume successfully. Now my Volume Mac mini Backup is fine, but now the disk is missing from Volume Tunderbay 4, and I cannot re-add it. Please check the error in the screenshot. Not sure what to do now.

Screen Shot 2022 05 17 at 5.28.27 PM

 

EDIT: I seldom post without an edit.

ReplyQuote
Topic starter Posted : 17/05/2022 5:32 pm
rexbinary
(@rexbinary)
Eminent Member Customer
Screen Shot 2022 05 17 at 5.46.12 PM
Screen Shot 2022 05 17 at 5.46.18 PM
Screen Shot 2022 05 17 at 5.46.24 PM
Screen Shot 2022 05 17 at 5.46.33 PM

Here are the drives. Looks like there is no partition for the ThunderBay 4 Volume on disk3 which is the disk that was replaced.

EDIT: I seldom post without an edit.

ReplyQuote
Topic starter Posted : 17/05/2022 5:49 pm
(@softraid-support)
Member Admin

@rexbinary 

Don't use Disk utility for SoftRAID disks, it is has a terrible user interface.

Attach a SoftRAID tech support file, so I can see what is up.

ReplyQuote
Posted : 17/05/2022 6:37 pm
rexbinary
(@rexbinary)
Eminent Member Customer

Please see attached. Thank you for your help.

 

EDIT: I seldom post without an edit.

ReplyQuote
Topic starter Posted : 17/05/2022 6:46 pm
(@softraid-support)
Member Admin

I assume both drives are intentionally unmounted?

Did you try to "Add disk" now that the other volume is not mounted? What is the error if it cannot be added?

ReplyQuote
Posted : 18/05/2022 9:47 am
rexbinary
(@rexbinary)
Eminent Member Customer

Yes they are unmounted as I kept getting the disk is in use.

The error when trying to add the disk is above. https://forums.softraid.com/postid/15827/

EDIT: I seldom post without an edit.

ReplyQuote
Topic starter Posted : 18/05/2022 10:03 am
(@softraid-support)
Member Admin

@rexbinary 

I think I know the cause of this. The Mini Backup volume is a Time Machine volume. As such, macOS never really unmounts it. Time Machine "holds on" to it somehow.

 

Maybe you need to "remove" it from Time Machine, then after adding the disk to the other volume, add it back.

ReplyQuote
Posted : 18/05/2022 11:38 am
rexbinary
(@rexbinary)
Eminent Member Customer

@softraid-support Same issue. I removed the Mac Mini Backup volume from Time Machine and tried to add the disk again, same a error. I rebooted, removed from Time Machine, same error on add to disk. I even tired killing backupd and backupd-helper. No go. I also tried to remove the disk itself from the Mac mini Backup volume, same error, the disk is in use by another application.

EDIT: I seldom post without an edit.

ReplyQuote
Topic starter Posted : 18/05/2022 11:53 am
(@softraid-support)
Member Admin

@rexbinary 

Restart. Unmount both volumes.

Run these two commands in terminal: (copy paste each line)

sudo lsof /Volumes/Mac\ mini\ Backup

sudo lsof /Volumes/Thunderbay\ 4

 

Does this give any clues to what process is open on the volumes?

ReplyQuote
Posted : 18/05/2022 5:38 pm
rexbinary
(@rexbinary)
Eminent Member Customer

@softraid-support No, unfortunately not. Since the volumes are not mounted it lsof cannot find them.

Screen Shot 2022 05 18 at 6.06.36 PM

If I mount them I get this:

❯ sudo lsof /Volumes/Mac\ mini\ Backup
COMMAND   PID USER   FD   TYPE DEVICE   SIZE/OFF     NODE NAME
mds       118 root   22r   DIR   1,42        288        2 /Volumes/Mac mini Backup
mds       118 root   24r   DIR   1,42       3264       19 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43
mds       118 root   28r   DIR   1,42        288        2 /Volumes/Mac mini Backup
mds       118 root   29u   REG   1,42          0       84 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/journalExclusion
mds       118 root   30r   DIR   1,42        288        2 /Volumes/Mac mini Backup
mds_store 330 root  txt    REG   1,42         56       52 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-1.map.header
mds_store 330 root  txt    REG   1,42         56       56 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-2.map.header
mds_store 330 root  txt    REG   1,42         56       60 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-3.map.header
mds_store 330 root  txt    REG   1,42         56       64 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-4.map.header
mds_store 330 root  txt    REG   1,42         56       68 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-5.map.header
mds_store 330 root  txt    REG   1,42          8       27 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/0.indexIds
mds_store 330 root  txt    REG   1,42          4       28 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/0.indexBigDates
mds_store 330 root  txt    REG   1,42       3277       29 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/0.indexGroups
mds_store 330 root  txt    REG   1,42          8      105 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/0.indexCompactDirectory
mds_store 330 root  txt    REG   1,42       2056      104 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/0.indexDirectory
mds_store 330 root  txt    REG   1,42         64      106 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/0.indexArrays
mds_store 330 root  txt    REG   1,42      65536 11802158 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.2.indexCompactDirectory
mds_store 330 root  txt    REG   1,42      90428 11802187 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.1.indexGroups
mds_store 330 root  txt    REG   1,42     263168 11802157 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.2.indexDirectory
mds_store 330 root  txt    REG   1,42      12288       53 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-1.map.data
mds_store 330 root  txt    REG   1,42       4096       54 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-1.map.offsets
mds_store 330 root  txt    REG   1,42       4096       55 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-1.map.buckets
mds_store 330 root  txt    REG   1,42     729088       57 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-2.map.data
mds_store 330 root  txt    REG   1,42     419431 11802152 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.2.indexGroups
mds_store 330 root  txt    REG   1,42      94208       58 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-2.map.offsets
mds_store 330 root  txt    REG   1,42     172048       59 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-2.map.buckets
mds_store 330 root  txt    REG   1,42     452136 11802186 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.1.indexBigDates
mds_store 330 root  txt    REG   1,42       4096       61 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-3.map.data
mds_store 330 root  txt    REG   1,42       4096       62 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-3.map.offsets
mds_store 330 root  txt    REG   1,42       4096       63 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-3.map.buckets
mds_store 330 root  txt    REG   1,42      28672       65 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-4.map.data
mds_store 330 root  txt    REG   1,42       8192       66 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-4.map.offsets
mds_store 330 root  txt    REG   1,42      12928       67 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-4.map.buckets
mds_store 330 root  txt    REG   1,42     151552       69 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-5.map.data
mds_store 330 root  txt    REG   1,42       8192       70 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-5.map.offsets
mds_store 330 root  txt    REG   1,42       9696       71 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/dbStr-5.map.buckets
mds_store 330 root  txt    REG   1,42    1315850 10203796 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.0.indexGroups
mds_store 330 root  txt    REG   1,42     904272 11802185 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.1.indexIds
mds_store 330 root  txt    REG   1,42    2097152 11802151 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.2.indexBigDates
mds_store 330 root  txt    REG   1,42    4194304 11802150 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.2.indexIds
mds_store 330 root  txt    REG   1,42   13158496 10203794 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.0.indexIds
mds_store 330 root  txt    REG   1,42    6579248 10203795 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.0.indexBigDates
mds_store 330 root  txt    REG   1,42    4483108 11802192 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.1.indexDirectory
mds_store 330 root  txt    REG   1,42   12951296 11802193 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.1.indexCompactDirectory
mds_store 330 root  txt    REG   1,42  119907976 10203801 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.0.indexDirectory
mds_store 330 root  txt    REG   1,42  129762028 10203802 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.0.indexCompactDirectory
mds_store 330 root  txt    REG   1,42 1108777088 10203803 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.0.indexArrays
mds_store 330 root  txt    REG   1,42   78635776 11802194 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.1.indexArrays
mds_store 330 root  txt    REG   1,42   67108864 11802154 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.2.indexTermIds
mds_store 330 root  txt    REG   1,42   67108864 11802156 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.2.indexPositionTable
mds_store 330 root  txt    REG   1,42  134217728 11802159 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.2.indexArrays
mds_store 330 root    6u   REG   1,42       4096 11802149 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.2.indexHead
mds_store 330 root   25r   DIR   1,42       3264       19 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43
mds_store 330 root   41r   REG   1,42   27934196 11802188 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.1.indexPostings
mds_store 330 root   43r   DIR   1,42       3264       19 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43
mds_store 330 root   49u   REG   1,42         28       25 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/indexState
mds_store 330 root   50u   REG   1,42  134217728 11802153 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.2.indexPostings
mds_store 330 root   51u   REG   1,42  362991616       51 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/.store.db
mds_store 330 root   53r   REG   1,42  573227019 10203797 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/live.0.indexPostings
mds_store 330 root   55r   REG   1,42          4      100 /Volumes/Mac mini Backup/.Spotlight-V100/Store-V2/67DA2A63-7173-4F4E-8904-4AB8562AFF43/0.indexPostings
❯ sudo lsof /Volumes/Thunderbay\ 4
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
mds       118 root   27r   DIR   1,39      612    2 /Volumes/Thunderbay 4
mds       118 root   32r   DIR   1,39      612    2 /Volumes/Thunderbay 4
mds       118 root   34r   DIR   1,39      612    2 /Volumes/Thunderbay 4
revisiond 153 root   10r   DIR   1,39      612    2 /Volumes/Thunderbay 4

EDIT: I seldom post without an edit.

ReplyQuote
Topic starter Posted : 18/05/2022 6:14 pm
rexbinary
(@rexbinary)
Eminent Member Customer

Could the issue be internal to SoftRAID? That physical disk is in use by the Mac mini SoftRAID Volume regardless if it's mounted or not. I can't remove it from Mac mini Backup Volume, nor can I add it to ThunderBay 4 Volume. Maybe if I just delete the Mac mini Backup Volume it will release the disk and I can save the other ThunderBay 4 Volume?

This post was modified 1 month ago by rexbinary

EDIT: I seldom post without an edit.

ReplyQuote
Topic starter Posted : 18/05/2022 6:17 pm
rexbinary
(@rexbinary)
Eminent Member Customer

I gave up, wiped the whole thing, and started over. At least I was able to save what was on the main Volume even if I did have to lose all my Time Machine backups. 😑

EDIT: I seldom post without an edit.

ReplyQuote
Topic starter Posted : 18/05/2022 10:25 pm
(@softraid-support)
Member Admin

@rexbinary 

It looks like it was a spotlight problem, but you cannot disable spotlight on a Time Machine volume. Hope your system is working fine going forward.

ReplyQuote
Posted : 19/05/2022 8:37 am
Share:
close
open