Notifications
Clear all
Topic starter
I have a couple of trial copies of SoftRAID software and am considering purchasing it for my data needs. I set up a RAID 5 volume with it and found that read/write performance was much worse than I would have expected.
To illustrate, I have attached a screenshot with benchmarks on several volumes that I completed using XBench. The upper left is a RAID 5 volume with 5 spinning disks (two WDs and three Seagates), 4x4TB + 1x5TB. The upper right is a RAID 5 volume I created just using the four 4TB disks, in case that was the issue.
The lower left is a single WD 4TB drive (not used in any of the RAID arrays) and the lower right is a single Seagate 5TB drive (also not used in any of the RAID arrays). The benchmarks also give basic info on the system I am running this on. The processor has 6 cores at 3.5 GHz each, so I doubt processing power is an issue.
As these show, the performance of the RAID volumes as compared to the single disks is at best modestly better and at times even worse. I've also tried a variety of other ways to test speed. For instance, copying files with pv (perhaps not so reliable due to caching?), reading and writing files line by line in python, using BlackMagic speed test, etc. For instance, reading a 3GB txt file line by line in Python took 16.5 seconds for the RAID5 volume and 22.3 seconds for the single disk. Writing the same file took 142.8 seconds on the RAID 5 volume and 141.6 seconds on the single volume. If there is some preferred performance metric that you have, I'm happy to use it, but I doubt it would fundamentally change the results here.
I saw here https://srforums.wpengine.com/forum/showthread.php?tid=67&page=2 that softRAID commented that sometimes benchmark programs are not so reliable for volumes with a lot of files on them - my volumes do indeed have a lot of files on them. But, this applies for both the RAID volumes and the single drives. Moreso, I found these performance differences even when not using explicit benchmark programs (e.g. just using pv or python), and perhaps above all, I'm ultimately only interested in the read write speeds for volumes with data on them. A solution that delivers great performance but only on empty disks is not so helpful to me. None of the volumes are particularly full here (e.g. the RAID 5 volume has max capacity of 13.5 TB and only about 7 TB of data on it, the two single disks that I tested had similar proportions).
These various disks are connected via USB 3.0 cables. None of the read-write speeds get particularly close to the bandwidth limit on such cables, so I don't think that is the issue.
Do you know what could be accounting for this poor performance and if these is anything that could be done about it? The softRAID software seems like it has nice potential and I'd like for it to be able to work for my needs, rather than me needing to purchase a hardware RAID setup. But, trying to get these performance issues resolved would be very helpful in feeling that softRAID is a more viable option to move forward with.
Any assistance that you could provide on this matter would be most appreciated.
Posted : 11/12/2016 5:14 pm
The reason to have standardized benchmarks and use cleanly initialized volumes is to have consistency in results. If we do a benchmark and get 500MB/s (using a specific benchmark test), and a user complains they are getting 400MB/s, there has to be a common set of parameters to compare, or the results are not valid. The more files on a volume, the less reliable benchmarks will be, and the slower the results.
Everything contributes to overall performance. If you have a i7 based iMac or Mac Pro, connected via thunderbolt, for instance, you can expect >500MB/s on a RAID 5 with standard disks. An i5, for instance, will reduce RAID 5 write performance by 10% or more. Dual core will be slower than quad cores.
The SoftRAID driver has response times in microseconds, so is not the "cause" of poor performance numbers. All the bottlenecks need to be examined to see where your system is slowing you down.
It seems like you have several potential bottlenecks:
USB 3. (also, use SoftRAID to be sure you are running each device at USB 3, not USB 2. Look in the disk tiles)
Perhaps a USB hub?
Computer?
All these can combine to reduce performance.
Here is what we need to help understand your issue.
What is the CPU/Mac Model?
You are using a USB multi-disk enclosure? (not so good for RAID performance)
Built in, or connecting via a USB hub? (USB Hubs reduce performance by 30% or more)
Is your data fragmented?
Lets simplify your benchmark to something we can relate to here. So use Black Magic's basic benchmark tool.
It will give an approximation to your performance. We don't use X-Bench here, we have our own internal benchmarking tools.
SoftRAID will help your disks operate as fast as the disks can go in your system.
RAID 5 has some CPU requirements to maximize performance, so if you don't have a 4 core i7, you will show significantly slower results than on our web pages (but still much faster than a single disk.)
Strongly consider migrating to Thunderbolt, USB is not meant for such data loads. In addition, there are several USB issues unique to OS X, so we don't really recommend USB for RAID implementations.
Posted : 12/12/2016 4:58 pm
Topic starter
Ok, thank you, this is helpful. In response to the questions, I am running this on a mac pro 6,1 (late 2013). It has an intel Xeon E5 process with 6 cores x 3.5GZ. Thus, it isn't specifically an i7. but it's a pretty powerful processor. Would this genuinely be slowing me down here?
I noticed that the volume was being indexed when I initially ran the tests. When I shut this off I went up to about 220 MB / sec for both the read / write using BlackMagic. Thus, this is now a decent bit better than what I get from a single disk, but it would still be nice if I could identify ways to further improve performance.
All of the disks are indeed USB 3.0. I have one of the five plugged directly into one of the computer's four USB ports. For the other four, I have two each on two USB 3.0 hubs. I would think though that this shouldn't slow me down, at least not to the 220 MBps speeds that I am seeing, since USB 3.0 transfers at 640 MBps. Thus, e.g. for a read, operation where I am just pulling data off each of the five disks, even if I were drawing on each of the five drives at 100 GBps, I would be well below the bandwidth limit on the USB cables and hubs yet well above the performance that I am seeing. Am I missing something here though in this reasoning? I'm sure that there is some additional overhead from the RAID software, i.e. that it takes more than 100 GB of data transfer over the cable to get you 100 GB of data you actually want off the disk, but would the discrepancy be so high?
I considered getting a thunderbolt hub to plug some of the USBs into. My thought had been that this wouldn't improve performance, based on my above comments on USB 3.0 transfer speeds. But, do you have experience with such hubs and any specific comments on (a) how much, if at all, they could improve performance and (b) which you would recommend? Obviously, a purely thunderbolt setup would deliver better performance, but I'm trying to see what I can accomplish with the hardware that I already have on hand.
Lastly, in terms of the data fragmentation, could you tell me a bit more of what you mean there? The tests I was doing were all just reading and writing from a single file. If I just go off of the BlackMagic speed tests, which I am fine to do for the purposes of this diagnostic exercise, is that something that is relevant?
Thanks!
Posted : 12/12/2016 5:57 pm
Lots here:
The Mac Pro 6,1 is a great machine and will get you maximum performance. In a Thunderbolt enclosure, you will get 500MBs or faster, consistently.
This should give you confirmation that USB is the issue. Something you may not be aware of with USB, is the CPU must manage every point of contact for each device. With a hub involved, this means the hub has to traffic cop all I/O. Yes, this makes a huge difference. Take the same disks you are using with USB, put them into a Thunderbolt enclosure, and you will immediately see a major performance boost.
As an example, we researched and purchased the fastest USB 3.0 hub we could find. Compared performance on the Mac Pro, with devices plugged directly into the Mac Pro. There was a 35% drop in performance going through the Hub, even with slower USB devices. The management cost of USB is very high when multiple disks are in use.
You can use your own benchmark tests, but it gives you nothing to compare to. The tests on the front page of our web site use AJA performance test 2.x, with the parameters specified on the page. You want to be able to compare Apples to Apples. Your tests are fine, but you do not have a point of reference against another system.
If you can dredge up a budget for a 4 bay Thunderbolt enclosure, do so. With the Xmas sales, OWC for example offers an empty enclosure for $400. ($490 with the Thunderbay version of SoftRAID) 525MBs is a reasonable read/write expectation. Compare that to what you are seeing with USB.
Posted : 12/12/2016 8:04 pm
Topic starter
Ok, got it, this is helpful to know, thank you. I'll post again if I have further followup questions.
Posted : 12/12/2016 10:26 pm
