Software RAID

[The_Unbeliever]

Hey

Just want to share my experience with Software RAID under Linux

Got a server back from overseas, damagement decided to scrap it as it was giving them too much hassles with Windows7 and the hard drives.

The add-on Intel RAID card packed up, so the installers reinstalled Windows7 on one hard drive, and spanned four other hard drives.

Of course this is problematic, given that any one HDD failure on the spanned set will lead to loss of the entire span. And this happened regularly.

So I got my grubby little fingers on it and tried Zentyal (eBox) first.

Zentyal.

But I wasn't happy-chappy with the Zentyal installation routine. It didn't detect that there was more than one HDD available, and thus installed to a single HDD. So I scrapped it off my list as I need something that a n00b can install onsite if need be.

Next up was SME Server.

SME Server.

Installed it, and it picked up the 5 HDD's straightaway. During installation, the one HDD developed a problem, and I had to redo the installation again. (I used RAID5).

Synchronization was slow, and I had a shufty at the motherboard SATA controller settings, and found that these was set to 'compatibility'. Switched them over to 'native' and 'AHCI' - and this improved the synchronization transfer rate greatly.

Of course I messed around with the HDD's just to make sure that SATA0 = SDA etc, and in the process messed up everything.

After a reinstall everything is up and running sweetly, sync rate is between 200K/s and up to 10000K/sec and things are looking good. This time I opted for RAID6 instead of RAID5.

Every 1.0s: cat /proc/mdstat                            Tue Feb  1 15:40:04 2011

Personalities : [raid1] [raid6]
md2 : active raid6 sda2[0] sde2[4] sdd2[3] sdc2[2] sdb2[1]
      2929966080 blocks level 6, 256k chunk, algorithm 2 [5/5] [UUUUU]
      [>....................]  resync =  4.1% (40834304/976655360) finish=3025.0
min speed=5153K/sec
md1 : active raid1 sda1[0] sde1[4] sdd1[3] sdc1[2] sdb1[1]
      104320 blocks [5/5] [UUUUU]

unused devices: <none>

The only difference between software RAID and a proper RAID controller is that software RAID requires a bit more CPU overhead as the proper RAID controller's onboard CPU will handle the RAID array and other overheads.

Proper RAID controllers do cost a bit of money though.

Personally I would prefer to use a proper RAID controller and a hot spare should a server go to site - purely for the reason that a n00b admin can simply power the server down, remove the faulty HDD, plug in the new one, and be up and away without any issues.

In most cases, a simple mirrored setup between two hard drives will be more than enough though.

So who's done software RAID5 or 6 so far? And is it holding up?

Regards

Ook

 

[zamicro]

I have been running software raid 0+1 for a few years and then went to software raid 5. This was all done on a vanilla Fedora installation. But after re-evaluating my needs, I recently moved away from software raid and started using Greyhole. It is an interesting concept, but if you need performance, I suggest you stay with raid. I suddenly have a lot more storage since using Greyhole, as not all data needs to be redundant (movies). This was all done with Amahi. Not for the SME market, but perfect for the home server market. I would recommend it anytime.

 

[The_Unbeliever]

I have been running software raid 0+1 for a few years and then went to software raid 5. This was all done on a vanilla Fedora installation. But after re-evaluating my needs, I recently moved away from software raid and started using Greyhole. It is an interesting concept, but if you need performance, I suggest you stay with raid. I suddenly have a lot more storage since using Greyhole, as not all data needs to be redundant (movies). This was all done with Amahi. Not for the SME market, but perfect for the home server market. I would recommend it anytime.

Interesting concept.

 

[spiderz]

Still using Ubuntu + ZFS (zraid). Works like a charm!
Busy moving hdd's to anotherserver so i can upgrade the hdd's in the current server. (have space for 8 sata hdd's)

 

[<?php?>]

Some MB’s come with “soft” RAID, which you can enable from the BIOS settings. I found the soft RAID approach to be slow at times and somewhat unstable, and now run software RAID5 on an NAS / SAN server without any problems thus far.

 

[w1z4rd]

Software raid > Hardware raid imho.

Hardware raid = propriety vendor lockins most of time.

You really should move over to ClearOS Libs, its changed since it was ClarkConnect. Those distro`s you use are so dead.

 

[Tinuva]

software raid thats built into the motherboards are always slow, Linux's mirror disk (aka software raid) is far superior over that. The linux software raid1 for example balance reads over the 2x raid1 disks, ie. reading get the same speed performance increase that raid0 has. However it will be normal write speed. Software raid from a motherboard on the other hand does all reads from the same disk, so no performance increase there.

Hardware however has its place, for example I have seen excellent performance from 3Ware's raid cards. They also handle disk failures better than most of these motherboard solutions ect.

Sadly, hardware raid will always have their proprietary vendor lockin, it is just something you have to live with.

 

[koffiejunkie]

I have been using mdraid for many years. Most recently, I had a bunch of 2.5" 500GB drives in my home server, primarily to keep heat and noise down, but I've found them unreliable in a 24/7 application. Went through about 8 drives in two years. So recently I switched to two WD Green 2TB drives in a RAID-1.

One thing to note about mdraid is that it's pretty sensitive. I haven't found a way to see what sort of errors hardware RAID cards typically consider as a drive failure, but mdraid is certainly more aggressive. I've had several drives being failed in an mdraid array, only to find that they pass every kind of test I can throw at them, and continue to work reliably for years after that.

One thing to consider when building a RAID system (software or hardware) is how long you can afford to be without redundancy. On my servers I use at least three drives in a RAID-1, particularly for machines with very large drives. The time it takes to rebuild a 500GB+ drive in a RAID set on a busy server (if I can't afford the downtime to do an off-line rebuild) can be many hours, sometimes more than a day.

Software raid > Hardware raid imho.

Hardware raid = propriety vendor lockins most of time.

That's a slightly simplictic view. With every brand of hardware RAID controller I've worked with, even crappy ones, if your card fails, you can shove in another card, assemble a fresh array from the same discs, and you have access to your data. As long as you remember not to initialize it

Hardware RAID gives you other benefits. RAID battery, bigger cache, and with bigger sets you don't end up saturating the PCI bus (although that's less of an issue with PCIe).

I agree that linux RAID comfortably outperforms desktop class and low-end server cards, but if you're spending on a high performance solution, a good RAID controller still makes the most sense.

software raid thats built into the motherboards are always slow, Linux's mirror disk (aka software raid) is far superior over that. The linux software raid1 for example balance reads over the 2x raid1 disks, ie. reading get the same speed performance increase that raid0 has.

That's true (or at least, should be) for any RAID-1 setup.

 

[Tinuva]

Should be yes, sadly not

Try a few benchmarks on a motherboard onboard raid from these normal desktop pcs and tell me if its any faster than running a single disk straight.

 

[koffiejunkie]

Try a few benchmarks on a motherboard onboard raid from these normal desktop pcs and tell me if its any faster than running a single disk straight.

The onboard RAID on desktop class motherboards are really software RAID, most of the time. You can set up your RAID-0 or RAID-1 set in the bios, but if you boot a linux CD, it still shows you the two hard drives. It lets Windows boot off one disc, and only once the motherboard manufacturer's drivers are loaded, does it actually start seeing your discs as one device. So it's not really RAID at all, and in a RAID one the two discs are not identical (at least not in the setups I've seen).

Needless to say, as is the case most of the time, motherboard manufacturers excel at making crappy quality drivers, so I'm not surprised the performance isn't on par. We once had this on a client's server - Intel board nogal. All was working well with Windows 2000 Server, until Windows update pulled down a driver update for the motherboard RAID that changed the stripe format. So as soon as the drivers load, windows stops seeing the set and bluescreens. Bad for business...

You're far better off using mdraid if you're using Linux. One nifty advantage is that the metadata on each drive defines it's role. So it doesn't matter if you plug the drives in in a different order, or even switch from hd* to sd* - mdraid will still assemble the RAID regardless.

 

[w1z4rd]

I have been using mdraid for many years. Most recently, I had a bunch of 2.5" 500GB drives in my home server, primarily to keep heat and noise down, but I've found them unreliable in a 24/7 application. Went through about 8 drives in two years. So recently I switched to two WD Green 2TB drives in a RAID-1.

One thing to note about mdraid is that it's pretty sensitive. I haven't found a way to see what sort of errors hardware RAID cards typically consider as a drive failure, but mdraid is certainly more aggressive. I've had several drives being failed in an mdraid array, only to find that they pass every kind of test I can throw at them, and continue to work reliably for years after that.

One thing to consider when building a RAID system (software or hardware) is how long you can afford to be without redundancy. On my servers I use at least three drives in a RAID-1, particularly for machines with very large drives. The time it takes to rebuild a 500GB+ drive in a RAID set on a busy server (if I can't afford the downtime to do an off-line rebuild) can be many hours, sometimes more than a day.



That's a slightly simplictic view. With every brand of hardware RAID controller I've worked with, even crappy ones, if your card fails, you can shove in another card, assemble a fresh array from the same discs, and you have access to your data. As long as you remember not to initialize it

Hardware RAID gives you other benefits. RAID battery, bigger cache, and with bigger sets you don't end up saturating the PCI bus (although that's less of an issue with PCIe).

I agree that linux RAID comfortably outperforms desktop class and low-end server cards, but if you're spending on a high performance solution, a good RAID controller still makes the most sense.



That's true (or at least, should be) for any RAID-1 setup.

Here is a less simplistic version of what I was saying from someone who had the time to write out more detail.

Why prefer Linux software RAID?


Potential for increased hardware and software biodiversity
Kernel engineers have much greater ability to diagnose and fix problems, as opposed to a closed source firmware. This has often been a problem in the past, with hardware RAID.
Disk format is public
...thus, no vendor lock-in: Your data is not stored in a vendor-proprietary format.
A controller-independent, vendor-neutral layout means disks can be easily moved between controllers. Sometimes a complete backup+restore is required even when moving between hardware RAID models from the same vendor.
Eliminates single-points-of-failure (SPOF) compared to similar configurations of hardware RAID.
RAID5 XOR runs on host CPU, which practically guarantees that it is far faster than most hardware RAID microcontrollers.
RAID5 XOR speed increases as host CPU speeds increase.
RAID speed increases as host CPU count (multi-thread, multi-core) increases, following current market trends.
Cost. A CPU and memory upgrade is often cheaper and more effective than buying an expensive RAID card.
Level of abstraction. Linux software RAID can distribute data across ATA, SCSI, iSCSI, SAN, network or any other block device. It is block device agnostic. Hardware RAID most likely cannot even span a single card.
Hardware RAID has a field history of bad firmwares corrupting data, locking up, and otherwise behaving poorly under load. (certainly this is highly dependent on card model and firmware version)
Hardware RAID firmwares have a very limited support lifetime. You cannot get firmware updates for older hardware. Sometimes the vendor even ceases to exist.
Each hardware RAID has a different management interface, and level of feature support.
Your hardware RAID featureset is largely locked in stone, at purchase time. With software RAID, the featureset grows with time, as new features are added to Linux... no hardware upgrade required.
Additional RAID mode support. Most hardware controllers don't support RAID-6 as Linux software RAID does, and Linux will soon be adding RAID-5E and RAID-6E support.
Error handling and logging varies from vendor to vendor (and card to card), with hardware RAID.
Many ATA-based hardware RAID solutions either (a) fail to manage disk lifetimes via SMART, or (b) manage SMART diagnostics in a non-standard way.

Why prefer Linux hardware RAID?

All the efficiencies that may be derived from reducing the number of copies of each WRITE (system -> controller, when compared to the software RAID case).
Software RAID may saturate PCI bus bandwidth long before a hardware RAID card does (this presumes multiple devices on a single PCI bus).
RAID5 XOR calculations are not performed on the host CPU, freeing the host CPU from other tasks and preventing host CPU saturation under load or DoS.
It is easy to parallelize RAID operations, simply by purchasing more HW RAID cards, without affecting host CPU usage.
Battery backup on high end cards allows faster journalled rebuilds.
Battery-backed write-back cache may improve write throughput.

http://backdrift.org/hardware-vs-software-raid-in-the-real-world-2

So yeah, while hardware raid does have its niche, the advantages of software raid clearly outweigh that of hardware raid. Only time I would lock myself into a hardware raid vendor is if I have a server that can not go down. The only real advantage I see is hot swappable drives.

 

[koffiejunkie]

So yeah, while hardware raid does have its niche, the advantages of software raid clearly outweigh that of hardware raid.

For you. The advantages listed in that quote is anecdotal, at best. Do you have any proof that a general purpose CPU has better XOR performance than the made-for-purpose chips found on hardware raid controllers? XOR is of course not the only thing RAID controllers do...

The argument about vendor lock-in, to me, is as good as saying a tin-foil hat will protect you from the alien invasion. How exactly do is your perceived lack of vendor lock-in of any use to you?

Only time I would lock myself into a hardware raid vendor is if I have a server that can not go down. The only real advantage I see is hot swappable drives.

You can hot swap SATA/SAS drives. You can build a server with a SATA/SAS backplane and if you use a decent enough SATA/SAS controller, you can yank drives out and shove other ones in. This is why eSATA works at all. You'll have to tell mdadm that the old drive is gone in the new one is in its place, but you have to do that with hardware RAID too.

 

[w1z4rd]

The argument about vendor lock-in, to me, is as good as saying a tin-foil hat will protect you from the alien invasion. How exactly do is your perceived lack of vendor lock-in of any use to you?

Im not locked into a vendor, if they stop producing the card I dont have a problem, I dont have to worry about firmware updates (or a firmware update corruption), I dont have to worry about scaling, I know with every linux update I get more features and a cleaner system, easier monitoring (remember, I dont manage one or two servers Ive got close to 50 out there), easier management. Since I do remotely manage and monitor all my servers and their raid arrays I find this useful. If Im using LVM I can more data storage to a server with two simple commands. I dont need to use stupid propriety tools to manage my raid array (not saying this is true for all raid cards out there).

My personal opinion is I would rather take the money wasted on raid cards and put it into more memory. Software raid works for what I need it for, Im sure hardware raid is good for what you need to do.

 

[koffiejunkie]

Im not locked into a vendor, if they stop producing the card I dont have a problem

You have no guarantee that the kernel developers won't, at some point, change the raid subsystem and drop the current format entirely. And your card doesn't stop working when the manufacturer stops producing them. And if you're serious about redundancy you'll have spare cards. And as you rightly say, with VLM you can easily migrate your data to another storage device without incurring downtime.

I dont have to worry about firmware updates (or a firmware update corruption), I dont have to worry about scaling, I know with every linux update I get more features and a cleaner system, easier monitoring (remember, I dont manage one or two servers Ive got close to 50 out there), easier management.

Scaling? 50 servers "out there"? I work for a company with over 100,000 servers in 9 data centers on four continents. We monitor and automate the living daylights out of everything we do. HW RAID are just as monitorable and remotely manageable (if not more so) as mdraid.

If Im using LVM I can more data storage to a server with two simple commands.I dont need to use stupid propriety tools to manage my raid array (not saying this is true for all raid cards out there).

LVM has nothing to do with RAID. You can put LVM on any type of storage.

You sound like a typical opensource zealot defending your cheapskate infrastructure choice (and I don't know you to be this biased). The stupid propitiatory tools we use is as easy to use as mdadm, does everything that mdadm does and a whole lot more. Like migrating RAID levels. Try that with mdadm... The tools haven't changed in many years and the tools that came before them is still available from the vendors.

I'm not trying to persuade you that mdadm isnt' any good - I use it myself. But please stop trying to make hardware RAID out as somehow inferior because of some idealistic BS you read on the internet. Your arguments simply don't wash.

 

[Elvis007]

No systems Admin would ever implement software raid in a production environment, if you do you have no idea what you are doing, simple as that.
For home use it's fine.

 

[w1z4rd]

You sound like a typical opensource zealot defending your cheapskate infrastructure choice (and I don't know you to be this biased).

.... some idealistic BS you read on the internet..

Name calling and character attacks. Such quality conversion. This is when I leave.

For the rest of you, I really dont care enough about the issue to get as emotional as others. Use whats best for you. I was simply giving my opinion on what I prefer to libs initial OP. A simple Google search will tell you whats best for your setup: http://www.google.co.za/search?hl=en&q=software+raid+vs+hardware+raid&aq=f&aqi=&aql=&oq=

 

[koffiejunkie]

Name calling and character attacks. Such quality conversion. This is when I leave.

I was actually saying that your comments were out of character.

Never mind.

 

[Gnome]

I thought I'd chip in. Software RAID is superior to hardware RAID, from everything I've read.

Anandtech did a article about it a short while ago: http://www.anandtech.com/Show/Index...=1&slug=zfs-building-testing-and-benchmarking

They compare a enterprise hardware based system (with a cost close to R100 000) to a ZFS based software system. Notice however that the ZFS system is still very expensive (almost as expensive as the hardware based system. Still it's performance is about 2-4 times that of the hardware based solution depending on the test. I've also recently read that FreeNAS developers with the imminent release of FreeNAS v8 will have fairly enterprise level performance, with them claiming that they saturate 10 Gigabit LAN (that is a serious feat. for storage on a single box).

Fact is software has several advantages over hardware. I won't go into them now but hardware RAID will eventually die out, not for a while but it is inevitable. Sun Microsystems ZFS developers wrote an article about why hardware RAID will not keep scaling in the future, essentially it comes down to the fact that you cannot separate the implementation of redundancy from the file system and keep scaling. Which is why RAIDz and ZFS file system is rolled into one.

I don't argue these types of things with SA "admins" because the reality is SA is usually a few years behind the US. But there are already large data centers running software based NAS, and many companies are selling NAS systems based on software rather than hardware.

In SA it is true that no company would run software but it is because the people in positions of power make choices based on past experience rather than research and testing. Those same companies in SA however also still use AS400 and buy into the IBM Mainframes are superior nonsense when companies much larger (google) run completely on standard x86. I've dealt with these people before, they get defense and angry if you even suggest another package. Just suggesting it makes them angry. Hence, no change.

 

[koffiejunkie]

Gnome, we were talking about mdraid, not ZFS. ZFS is definitely a beast of a different kind. The benchmark you link to compares ZFS to an entire storage solution that happens to contain RAID, rather than a RAID controller. That's not exactly a for or relevant comparison. There are differences between iSCSI and internal SCSI.