NAS storage

[Gnome]

Gnome, thanks for the info.

np

Knowing that a bad drive can be replaced with any similar or larger drive is one point for FreeNAS. I was working on the worst case scenario, that a drive had to be replaced with an identical drive - same model, same size, same firmware, same speed, etc. What about replacing a 7200 drive with a 5900, or a vari speed?

ZFS doesn't care about the drive apart from the fact that the size must be equal or more to replace a current disk. Things like speed, type (PATA vs SATA), model, etc. isn't even known to ZFS.

The sector size does play a role, but FreeNAS has the "force 4KB sector size option" when you create the array. I suggest you check that option even if you don't know if your drives are 4KB sector size drives or what it is.

Btw. the 4KB sector size is applicable to ANY NAS, it has to do with the disks and how they are accessed. There is now way around it, no hackity hack to make it go away. Western Digital refer to this as Advanced Format. Seagate I think has another name. Either way, all new drives are now 4KB sector size.

I actually wonder what Drobo does about 4KB sector size because if you don't format correctly, it places a lot of extra wear on your disk and the performance of the disk is pathetic (4mb/s vs 150mb/s).

The upgrading question is (for me) easily summed up - you can't upgrade a FreeNAS box. Buy a new one, copy everything across, and get rid of the old one. At least I know now.

Well you can upgrade your FreeNAS box by adding an extra RAID card (eg. IBM M1015 flashed to LSI 9112 IT mode gives you extra 8 SATA ports, costs ~R1k on eBay). Then you can add extra disks. But you cannot "upgrade" by increasing the space of an existing RAID-Z or ZFS Mirror.

It's now just a case of whether I'm happy to live with the restrictions, costs and benefits of the different options. Right now a mirrored 3TB drive for my desktop will cost about R 3300. A 6TB (12TB raw) FreeNAS upgrade for my MicroServer is about R 7800, and I get a static, but fast and reliable system (do I need a caching SSD?). A Drobo is about R 11700 for 6TB (12TB raw), or R 13400 for 9TB (15TB raw).

You don't need a caching SSD. With my NAS (Intel 2100T CPU, Intel 1GB NIC) I get 100-110megabytes/s copy speed (100% of the time). I have 6x2TB Seagate 5400RPM + 3x2TB Western Digital GP drives all in a single RAID-z1. I've had 2 disk failures thus far, both when I bought new drives. Both times recovery was as easy as falling out of a tree.

In fact the second time a hard-drive failed, I also had bad sectors on another drive. ZFS reported to me that there were 3 files which are probably damaged because there were multiple failures. Those files still play fine tho.

You really only need a caching SSD(s) if you need to fill up 10GB Ethernet.

Note that it is unlikely you'll get more than 30megabytes/s with the HP Proliant with the standard NIC card. The standard one isn't well supported in any OS but Windows. If you replace it with an Intel NIC (there is a R400 one that is PCI-Express x1 that will fit in the HP) you can probably get close to maximum speed for Gigabit Ethernet depending on the type of hard-drives you fit and the ZFS options you go for.

The Drobo's a bit too costly and the MicroServer can't expand when I fill it within 24 months. That leaves another internal drive with an external mirror - it's the cheapest of the lot, the most "repair friendly", but not the most reliable, and I'll still have 1 SATA port left for a 4TB in two years time. After that my PC will be due for replacement anyway. Pity. I was hoping to set up a reliable, resilient NAS for the whole network. I'll just keep streaming media from my PC.

Why don't you just get 5 drives and go for RAID-z1?

 

[Gnome]

I forgot to say:

Very important! ZFS MUST have at least 4GB of RAM and I HIGHLY recommend 8GB if you can!


Also don't bother with 32bit versions.

Gnome, you seem to be the one to ask here so, if you can indulge me: as I said earlier, I have a situation where I am stuck with a server presenting an NTFS share. Would this be feasible by having a FreeNAS box as a backend presenting an iSCSI target to, say a 'doze server (that, one way or another I tend to get stuck with, at least in my home office) so that said server offers up an NTFS volume and all the software is happy with life? Please excuse terminology mangling, I'm new in these waters.

I really haven't a clue how iSCSI works bud. Wish I could help, but I would recommend you create a FreeNAS setup on VirtualBox (give the VM at least 4.1GB of RAM) and test with that. Just to see if what you want is possible.

Performance wise ZFS is incredible (the file system itself) but it depends on your hardware and again I haven't tested the performance of iSCSI. With SMB (Windows Shares) I always max out Gigabit but that might not be the case for iSCSI depending on the implementation of it.

What NAS you buy, is your decision at the end of the day. We can only give some suggestions and recommendations, but you need to live with the consequences of your decision - either good or bad. <snip>

Nice post.

 

[ntfscoza]

Another good option is OpenFiler. You would need to be comfortable with partitioning etc, but its quite a sturdy NAS system and there's a lot of guides available.

Regards
@Ntfscoza

 

[bdt]

Another good option is OpenFiler. You would need to be comfortable with partitioning etc, but its quite a sturdy NAS system and there's a lot of guides available.

Once, in my (more) foolish youth I tried taking OpenFiler on ..and then got to meet with (HA) LIGHTweight directory access protocol and retreated in horror, gibbering to myself. Yea it's powerful and all, but also qualifies as swatting a fly for a Buick for small network environments.

 

[Gnome]

Did some testing a very long time ago with OpenFiler. I had bad network performance with OpenFiler. Got 30-40mb/s on SMB where I got 60mb/s on FreeNAS 7 (long time ago). Also it isn't updated much any more and driver support is terrible (because they use a seriously out of date Linux Kernel).

Still I stuck with OpenFiler for a few months, then went over to FreeNAS 7. That was about 3 years ago.

 

[w1z4rd]

Hey gnome. Does the amount of RAM you have affect the network copy speed? Reason I am asking is I was messing around with two FreeNAS servers. Both were ml110 servers. Both had GB NICs in them. I put both servers on their own GB switch. Nothing else on the network. I tried two different types of switches. A netgear and a D-link. Both switches showed the NICs connecting at GB speeds.

Problem is, when I ran rsync (via SSH) between the two servers. The speed was up and down. It would sit at around 20MBs for an hour, then shoot up to 200MBs for about 20-30 minutes, then drop down to 20MBs. I would say around 70% of the copy happened at 20MBs. When you are moving TB`s of information between the servers this takes a pretty long time.

These were just test machines and its probably my fault as I am only running them on 4GB of RAM each. Another possible reason for the bottleneck was I was using normal desktop hard drives instead of constellation drives. Im trying to figure out why the rsync was so slow It was probably the RAM right? The servers each had 8TB of storage. I know my RAM is on the low side, I just didnt expect it to affect network speeds that much. Where do you think I am going wrong?

 

[Gnome]

What kind of NIC cards?

Could be RAM related (4GB is the bare minimum) but that sounds like NIC or possible SAS/SATA controller issue. You are using ZFS right?

The golden rule is to stick to Intel NICs. I've had terrible performance from Broadcom and RealTek NIC adapters.

Onboard Intel or nVidia SATA controllers have worked well for me.

But if you need an offboard SATA controller there is only 1 controller you should consider. IBM M1015/Intel RS2WC/LSI 9211i/Supermicro (some code). Those controllers are all LSI SAS 9211 controllers. It has 2 Mini-SAS connectors that break out to 8 total SAS/SATA 6GB ports. It is a PCI-Express x8 card. If you buy the IBM M1015 version from eBay you pay around R800-R1000 for the card and R200-300 for the Mini-SAS to SATA cables. Then you flash it to LSI 9211i IT mode which gives you 8 SATA/SAS ports you can use just like that in any OS (driver support is awesome). The IT mode disables RAID and gives you a vanilla SATA/SAS controller that gives the OS direct access to the drives (SMART, hot-swap, etc. all work).

I'm personally using an IBM M1015 and Intel NIC on my NAS.

 

[w1z4rd]

Well come to think of it.. it could have been the NICs. The one was the default on board card that comes with a HP ML110. The other was a Realtek card. The reason I think it could be the NIC is because the rsync died like 6 times during the copy (this is initially why I swapped out the switch). Though it is strange for it to "kinda work".

As per usual, thanks for the excellent advice You are the "go-to" person on mass storage You are to blame for me moving off raid+extX to ZFS

 

[Gnome]

Well come to think of it.. it could have been the NICs. The one was the default on board card that comes with a HP ML110. The other was a Realtek card. The reason I think it could be the NIC is because the rsync died like 6 times during the copy (this is initially why I swapped out the switch). Though it is strange for it to "kinda work".

Sounds like the exact same thing I had with Realtek 8111e. When I actually Google'd it I found that the driver crashes, then recovers causing that issue. It was on all OSs except Windows. And it only happens if you place it under load. Not sure if that is sorted now.

Anyway I just realized after doing all the research that Intel is the only company producing decent NICs. Wouldn't ever recommend anything else again.

I'm not the only person tho, Asus recently started putting Intel NICs in their motherboards again (after having used RealTek for ages) and their workstation boards use exclusively Intel NIC adapters. According to Anandtech, it was due to customers explicitly stating it as a requirement (no surprises).

All these Intel NICs from the most expensive Intel server NICs to the cheapest desktop NICs are identified as the same thing in Linux/FreeBSD/Solaris "Intel PRO/1000". They all use the same stable driver.

As per usual, thanks for the excellent advice You are the "go-to" person on mass storage You are to blame for me moving off raid+extX to ZFS

Haha, nice thanks dude

 

[stroebs]

Hi peeps, directing this specifically at Gnome but opening it to comment of course.

I'm currently setting up a NAS for home/small business usage and need an opinion or two on my choice of hardware.

I've already decided on using FreeNAS as I'm running FreeNAS on an OC file server with 3 x 1Gbps cards in LACP and can get a sustained ~200MBps which I am VERY happy about.

Here's the hardware I'm looking at getting:
- ASUS P8B75-M LX LGA1155
- Intel SandryBridge Celeron G465 1.9GHz Single Core + HT (Should I be getting something faster?)
- 2 x Corsair ValueSelect 2GB DDR3-1333
- Corsair Voyager 8GB USB 2.0 Flash Drive - 19mb/s
- CFi a4000 Chassis (Reason for this is the 6 x hotswap 3.5" bays)
- CoolerMaster eXtreme Power 2 475Watt PSU

Anything else I should be considering?

Bearing in mind I'll probably pop an Intel PRO/1000 PT Server NIC into it as well.