First RAID 5 volume in SoftRAID - speeds seem low

@softraid-support

Thanks for continuing to work on this.  I must admit I seem to have run out of ideas of things to try and different combinations to experiment with, so I'm totally baffled.  I'd have expected with everything we've tried something would've pointed to the issue but there hasn't been a single combination that's worked.  It's a shame OWC don't have a presence in the UK - now we're outside the EU we're a little outside the scope of the direct OWC EU store as the customs charges would be unpredictable, although I was at least able to get the ThunderBays via the OWC EU Amazon store so that was something.  They only had 5 units in (of which I grabbed 2) and they don't have them in stock anymore, otherwise I would have just grabbed some fresh ThunderBays to try.

@guygamer

I know we are working on UK sales channels, global distribution is complex and the UK/EU change made your part of the woods more complicated.

@softraid-support

I know when I was planning this project I originally looked at the ThunderBay 8. Then ended up going with the 2x ThunderBay 4 because they were all I could find in stock and from OWC and also because it said two enclosures attached to different TB3 buses would give higher performance. Well now I can't get replacements of the ThunderBay 4s that I've had zero luck with but I have found a solitary ThunderBay 8 in stock with a reseller of OWC EU for next day delivery.  I notice that one has different chips for the the SATA and TB3. Based on your experience with these and my intended use of a 7 drive RAID 5 eventually converting to 8 drive RAID 6 when you add it to SoftRAID, would I be losing out on much performance going from 2x ThunderBay 4 to 1x ThunderBay 8?  Would it be a noticeable difference? Just looking for your honest opinion on this as I want to get the best performance I can out of everything considering the expense, but if it's a negligible difference I might be able to live with it.

@guygamer

With HDD's there is no difference, they are not fast enough, even 8 drives, to have an effect on performance.

SSD's matter. Remember Thunderbolt 3 is at 2800MB/s. Even Thunderbolt 2 is nearly 1,500MB/s and should still be full bore in one enclosure, as compared to two enclosures on two buses. With RAID 0 you may be hitting the peak, but barely, as there is file system overhead.

@softraid-support

Thanks again. I think in the first place it was product availability and the "exponential performance gains" on the listing that got in my head without realising that was more with SSDS in mind. 

From what you've said I won't be disadvantaged with a Thunderbay 8 on RAID 5 or 6 and it may be simpler and use less energy with one fan and psu. Plus it's $141 USD cheaper as well which I can offset the pricier drives. As it's the last unit in stock I've ordered it to arrive Wednesday. Fingers crossed this just works right away with none of the previous issue. 

I've already triggered a return of the Ultrastar drives with the intention of replacing with Exos when they come back in stock, but they won't have gone before the new unit arrives so I can at least test them in it. 

@guygamer

Thanks, please let me know.

Your timing is bad, as there is a temporary shortage of larger hard drives, so prices have really risen. Larger drives are as much as 50% higher just in the past few months.

@softraid-support

So the ThunderBay 8 arrived today and it's not good news unfortunately.  Re-ran the tests and getting similar if not worse results than before.  Here's the table:

ThunderBay 8 test 16 June 2021

Really wish there was an explanation for this.  I generated a support file in SoftRAID after the tests to see if that shows anything. Here that is:

Also I know they're probably not so useful but for the sake of being thorough I took a screenshot of each test result - I still don't understand the fluctuations in the graphs when testing in the ThunderBay devices. Attached so that you have them:

As you say the hard drive availability and prices are not at all good right now.  I have one site where I got the Ultrastars from who aren't price gouging but their stock is seriously depleted.  Other places are just charging silly prices.  I emailed a company about availability of drives they had listed yesterday and within 3 minutes of me hitting send I could see them editing the listings on the site and they doubled the price for the drive I had asked them about!!!!

I have absolutely no clue where to go from here.  There's clearly an issue somewhere in the pipeline but where is a mystery.  I really hoped this new enclosure was going to resolve things.

@guygamer

This almost has to be a disk issue. I will perform some tests with other drives for you.

TW: Did you try this on a clean install of your system?

@softraid-support

I haven't tried it on a clean install - honestly I have so may services and things critical to my workflow that I'm not confident of being able to get up and running smoothly without significant time and disruption.  I have, however, connected the ThunderBay 8 to my MacBook Pro tonight and recreated the various volumes to run all the tests again.  As with the ThunderBay 4 units I'm getting pretty much the same performance as on the Mac Mini.  Here's the results table:

MBP ThunderBay 8 test 16 June 2021

@guygamer

I have one more test for you. Start a verify disk on all 4 drives, for 15 minutes or so, then cancel. Now run AJA on the drives immediately after you cancel. Is the performance different? (Better?)

I have been running some tests and have results I will post tomorrow.

I ran a lot of tests with multiple drives, on USB and Thunderbolt. I am able to see your exact problem with some 10TB Hitachi drives. I am not seeing it with any Other drives, including some older 4TB drives. So this is 100% a drive issue. I am also seeing the better performance with the Hitachi over USB 3.1 gen 2 (our Mercury Elite Pro enclosures.

I can get you some numbers, but need to consolidate them first. Something about these Hitachi drives do not work well with Thunderbolt (at least on a mac), it appears.

@softraid-support

I tried the verify then cancel and ran tests for both a 4 drive stripe and 4 drive RAID5:

The results were actually a little worse than I've had from a lot of the tests over the past few weeks.  Should I have expected them to be better?  Over the course of the tests I've generally noticed slightly faster results on the occasions the tests were performed immediately after a system reboot. Don't know if that's related.

@softraid-support

Thanks for having stuck with this and continued to investigate.  To be honest it's a relief that you've been able to replicate this, if only for my sanity and knowing it isn't some outlier / quirk specific to my setup.  It'll be interesting to compare the numbers you got with mine (I've kept all the tests I've done in a document).  I wonder what it is about the drives that doesn't play well with Thunderbolt. I suppose as enterprise drives maybe they're not used in that configuration too often but I'm still surprised I haven't come across other similar accounts online.

In terms of trying to get replacement drives (when they eventually turn up in stock) my best option is looking like 12TB Seagate Exos X16 drives (ST12000NM001G). Have you used / encountered these at all or are there any known compatibility issues?

I'l have to decide which enclosures to keep or return now. From our discussion the other day I'll probably stick with the ThunderBay 8 for the RAID for simplicity.  I'm tempted to also keep one of the ThunderBay 4 units to replace both the old USB 3.1 gen1 4 drive enclosure and the single 3.1 gen2 TimeMachine drive (that thing generates so much heat!). That way I could use an old 8TB drive to add a mirror for my TimeMachine disk and then use my existing 10TB and 12TB as local backup drive for my most important / difficult to replace files - with the amount of data I have it'll never be viable to keep a complete local backup.

@guygamer

I think Seagate and Toshiba are currently the least problem drive brands, but this changes. For a long time, Seagate drives had high failure rates, but they addressed that issue and they are great again. I used both Exos, Iron Wolfs in my testing.

@softraid-support

Thanks, I went ahead and back ordered the 12TB EXOS.  They're the only drives that were sensibly priced (at the same store as the Ultrastars as they seem to be the only place no taking advantage of the shortage to massively increase prices!) and if there weren't any issues with them in your testing then I'm hoping I won't run into any with them either. Got them for what would have been a good price even before the supply issues so that's not too bad, the downside is no concrete ETA. Just within the past couple of days the estimate on their site has jumped from 24 June to 2 July, so if it keeps slipping it could be a while before I get them. Not ideal but not really any alternative.  Obviously I'm just impatient to get them and hopefully have a speedy, working RAID up and running ASAP!