Data Storage in the Home : Best Practises

Y

yeti

Well-Known Member
Joined
Oct 13, 2009
Messages
434
Reaction score
0
Location
Cape Town
I'm looking to find best practises on data storage, whether it be mirroring disks within a server using HW/SW RAID, an external NAS device or a cluster of servers running a DFS, all for a home environment.

The goal is to amass a set of guidelines and best practises to home storage that improve fault tolerance / cost while attaining reasonable performance.

Any input into any functional aspects of a storage subsystem, whether it be the software used for RAID, the FS or a NAS OS, hardware configurations and recommendations, or a combination of all of the above - any suggestions are welcome and appreciated.

Your guiding principles should be (not in order of precedence):
  • Cost - the minimalisation of costs w.r.t housing and exposing storage to the home network
  • Fault tolerance - data should be resilient, always be available and have no single point of failure
  • Performance - data should be accessible by the home network at reasonable performance (in the order of streaming)
  • Capacity - data should be stored in an efficient manner while respecting the other guiding principles.
  • Security - access to the data should be restricted by ACL

There is no wrong answer. Any suggestion can highlight one or more of the guiding principles, or any of your own, should mine not suffice.
 
This thread touches on that. There are (many) others.

Basically, start with a budget and work from there. If you've got R 3 000 and need 6TB, zero-redundancy-internals are your only option. Up your budget to R 9 000 and you'll get 6TB with 2 disk redundancy in a microserver. Need more space: lose some redundancy and keep the microserver, or up your budget to close to R 20 000 using desktop components or a basic server, or R 30 000+ using server components with all the bells and whistles.

Backup is another story. Tape is still considered your best bet, but at those sort of sizes it becomes prohibitively expensive. That's why I suggest 2 disk redundancy. Your porn should be safe, but for more important data, keep 2 copies off site using off-the-shelf externals.

I have no clue on OSs, FreeNAS is popular, but I have my reservations (see the above thread). Someone mentioned UnRAID which seems good too. Hardware based is generally frowned upon, but I figure, how far wrong can you go with a high end (R 10 000) Adaptec card? Cheaper stuff - I have no clue.

Then you need to consider how much performance you need, and how to achieve that. Fancy NICs, more RAM (much, much more), caching SSDs (MLC or SLC), etc, etc.

You'll run out of budget long before you get anywhere near that. I ran out when I realised that my microserver wouldn't be sufficient, so I'm using internals mirrored onto externals.

Good luck.
 
Yeah, I'm going to completely disagree with Drunkard #1.

I've experimented with OpenFiler, EON, Windows Server (software RAID5), Hardware RAID 5, FreeNAS

My experiments included testing on different types of hardware (Intel, AMD). I also did tweaks and tests beyond count to test performance. My goal was to see how I could get the best performance at a budget with enterprise reliability.

Instead of telling you why it is the best (which always results in users that do not understand what I'm talking about or misinterpret the info). I'll just recommend something and if you build it exactly as I say, you'll have an incredible NAS with high performance.

Hardware:
- HP Proliant Microserver N40L
- Intel Pro/1000 CT
- Icy Dock OR 3.5" to 5.25" Drive bay converter
(if you use the converter you also need this)
- SATA Cable (generic)
- 8GB Kingston ECC Value-RAM
- 4GB Kingston Flash-drive

NAS device cost: R3281 (if you use the expensive Icy Dock)

Yes that is R3.2k, but at that cost, please point me to another NAS that will allow you to install 5 drives?

- 5 Hard-drives of your choice.
- - eg. 5x2TB hard-drives, must be same size (but brand, rpm, etc. can differ). Desktop drives are fine (you do not need RAID drives). Avoid Western Digital Green Power drives because they park the read/write head every 5 seconds unless disabled (and some cannot be disabled). The disk won't last more than months unless you disable the head parking.

Software:
- Latest version of HP Proliant BIOS (NB!)
- Latest version of FreeNAS

Setup:
- Install the Icy Dock in the top or use the converter
- Install 8GB RAM
- Install Intel Pro/1000 CT in the x1 Pci-Express slot
- Install the SATA cable from the motherboard to top
- Next to where the SATA cable plugs in is an internal USB. Plug flash-drive in there.
- Disable internal HP Proliant Network Interface Card (NIC) -- NB!
- Change the HP Proliant boot setting to USB (boot from USB Flash-drive)

- Setup FreeNAS from a myriad of guides available online.
- - You can either install FreeNAS onto a flash-drive from VirtualBox or from a temporary CD-Rom on the HP Server.

FreeNAS:
- NB! - Use ZFS RAIDz1 over all 5 drives (single disk failure redundancy)
- NB! Set the following on EVERY hard-drives: HDD Standby: Always on, Advanced Power Management: Maximum Performance, Maximum power usage, Acoustic level: Maximum, SMART: Enable
- Enable the SMART Service from the FreeNAS GUI.
- Do not let the hard-drives power down constantly, hard-drives aren't worn out by reading and writing or running 24/7. They are worn out by constantly parking the head, being turned on/off and constant temperature changes. The best way to avoid that is by not powering them down AT all.

Take this from someone who is running a 9x2TB RAID-z, some of the hard-drives have been in 24/7 usage in a NAS for over 3 years and they are still going strong. These are normal desktop drives.

Once done you'll have a 5 disk drive NAS with single disk failure redundancy and the space of 4 drives. If you set it up properly it will email you if a disk fails or a disk has a SMART failure.

Some of my disks have had SMART failures since their first month but years later they are still running. A SMART failure does not guarantee a disk will fail. But it is good to know what kind of SMART failures you have so that you can monitor the situation (eg. constantly increasing bad sector count is a serious problem).

This will give you a seriously powerful, high performance and reliable NAS.

This should get you on your way. But if you are the point and click easy ride type of person, this isn't for you. Get a pre-built unit (which will not match the performance of the FreeNAS setup but whatever).
 
Last edited:
Gnome said:
Yeah, I'm going to completely disagree with Drunkard #1.

...

This should get you on your way. But if you are the point and click easy ride type of person, this isn't for you. Get a pre-built unit (which will not match the performance of the FreeNAS setup but whatever).
Click to expand...

Thank you for the extremely informative post. You're currently running a very similar configuration to me ~ albeit mine runs atop VMWare ESXI and provides some trivial ancillary services to the network with slightly less disk space.

I was thinking about moving away from the HPMS-as-a-NAS to a DFS across the entire network; each device allocates a certain amount of disk space and that provides reliable and robust storage to the networ. This broad solution has its own caveats as it needs to deal with a dynamic network topography where nodes 'come and go'. The principle idea here is to maximise the use of all equipment and thus lower the overall cost of buying hardware for this purpose alone.
 
Can some one please explain to me what a NAS is? Is it basically a file server/media hub?
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X