Large capacity 40Tb+ storage for video archiving, RAID? Any advice?

K

kbilly

Active Member
Joined
Mar 9, 2007
Messages
54
Reaction score
1
Hi everyone, we have a need for a large capacity storage solution 40Tb+ for long term storage of video material. We will mostly be writing. I am looking at something like a 6 bay Raid tower with 6x8Tb seagate archiving HDDs, probably USB3.

Anyone done this or have any advice regarding methos/brands, I seem to have options around R40k for Raid tower + 6 disks.

Looking at Raid 3 since integrity is most important and we will not be perfoming regular reads.
Thank you
 
Just remember to always double your archive. One copy isn't enough. Unless you don't mind losing the archive. All machines fail eventually - it's the way the universe is. Don't let it cause your business to fail.
 
Last edited:
you will need more than 6 8TB disks to get 40TB + worth of storage.. you need to account for filesystem overhead which reserves a chunk of space.. next, personally i have not seen off the shelve raid towers which support raid 3 neither have i seen adaptec hardware raid cards which support raid 3.. raid 5 is the same thing essentially.. again, personally, off the shelve raid towers i have found to not have the flexibility that i prefer.. in my case i would build a storage server from scratch on linux to gain the flexibility i would like..

when you say 40k for raid tower + 6 disks, is that 40k for the raid tower alone or for the raid tower and the disks as the total price..? going on takealots price for a single 8tb seagate archive hard drive, you are at 39k for the disks alone..

i have built 4 backup servers with 16 6TB hard drives, wd red hdd, with adaptec 6805 raid cards on raid 10 to total around 37TB of storage.. os was centos 7.. never had any issues other than a hard drive or 2 failing which was easy to replace and rebuild the volume..
 
Definitely wouldn't be thinking 6 8TB drives for long term archival storage.

You will need at least twice the storage capacity to be on the safe side...

I would honestly be thinking, given the fact that you are looking at an archival type scenario, that your biggest concern should be equipment integrity and for that I would look at some rack mounted storage unit probably if your budget allows for it.
 
Just a thought and excuse if I'm totally missing the pot here, I don't really have a clue how data centres work, but isn't renting hard drive space in a commercial data centre a option, someone like Hetzner?
They are experts in storing data and have much more specialized and more expensive equipment than what the average persons money can buy. Sounds like this is not for personal use but for a business and that this video material isn't something which you would want to lose, so let the experts handle it, no matter how you look at it, you'll anyway require a off site backup solution as well, otherwise you're not really backed up.
 
bigboy529 said:
Just a thought and excuse if I'm totally missing the pot here, I don't really have a clue how data centres work, but isn't renting hard drive space in a commercial data centre a option, someone like Hetzner?
They are experts in storing data and have much more specialized and more expensive equipment than what the average persons money can buy. Sounds like this is not for personal use but for a business and that this video material isn't something which you would want to lose, so let the experts handle it, no matter how you look at it, you'll anyway require a off site backup solution as well, otherwise you're not really backed up.
Click to expand...

The problem with that solution is getting the data from your location to the DC itself... that starts to get costly when you are talking 40TB of data and its also a schlep to retrieve it.

THOUGH in saying that, if data links are not an issue, then the OP should investigate something such a Amazon Glacier storage, or Google Nearline storage, or Azure Blob Cool level storage.
 
ToxicBunny said:
The problem with that solution is getting the data from your location to the DC itself... that starts to get costly when you are talking 40TB of data and its also a schlep to retrieve it.

THOUGH in saying that, if data links are not an issue, then the OP should investigate something such a Amazon Glacier storage, or Google Nearline storage, or Azure Blob Cool level storage.
Click to expand...



If going local like Hetzner that's not a problem, you drive to their data centre in Gauteng or Capetown, I'm sure you can even courier them your drives for upload. Retrieving is a issue with internet speeds in SA yes, but by the OP it sounds like they mostly want to write to the hard drives and forget about the data, but once again if going local retrieving will anyway also not be a big issue.

BTW I sit with this situation though on a much smaller scale, I use Crashplan as a cloud backup solution, as I type here I have around 400 GB still to upload and it says 48 days remaining. If I whent local I could have couriered my hard drives.
 
bigboy529 said:
If going local like Hetzner that's not a problem, you drive to their data centre in Gauteng or Capetown, I'm sure you can even courier them your drives for upload. Retrieving is a issue with internet speeds in SA yes, but by the OP it sounds like they mostly want to write to the hard drives and forget about the data, but once again if going local retrieving will anyway also not be a big issue.

BTW I sit with this situation though on a much smaller scale, I use Crashplan as a cloud backup solution, as I type here I have around 400 GB still to upload and it says 48 days remaining. If I whent local I could have couriered my hard drives.
Click to expand...

You can courier drives to crashplan as well since they use amazon as a backend...

And using the cloud providers gives you dr on the data included in the price pretty much. With a local provider it would be extra
 
Didnt read through everything but if you are storing that amount of data then just do it to tape, its cheap as chips and large volume is its purpose.
Amazon Glacier is cheap for this reason, its a storage facility with slow retrieval times but large volume storage.
 
bigboy529 said:
Just a thought and excuse if I'm totally missing the pot here, I don't really have a clue how data centres work, but isn't renting hard drive space in a commercial data centre a option, someone like Hetzner?
They are experts in storing data and have much more specialized and more expensive equipment than what the average persons money can buy. Sounds like this is not for personal use but for a business and that this video material isn't something which you would want to lose, so let the experts handle it, no matter how you look at it, you'll anyway require a off site backup solution as well, otherwise you're not really backed up.
Click to expand...
Upload speed is the problem. Renting 40TB is the easy part.
 
Hi could you elaborate why I would need double the storage for the amount of data I have?
 
Haha love your signature... Old cobol prog... Please let me know if I can DIY tape solutions, thanks!
 
Well that is the question, should I get a 400Mbps uncapped account for upload or get stoarge media myself, racks, or tape etc? Thanks!
 
kbilly said:
Hi could you elaborate why I would need double the storage for the amount of data I have?
Click to expand...

Duplication so you have redundancy pretty much...

Might not be double, might be a bit more for things like overhead/error checking bits etc.
 
kbilly said:
Well that is the question, should I get a 400Mbps uncapped account for upload or get stoarge media myself, racks, or tape etc? Thanks!
Click to expand...

You will need to give us more info really...


How often do you access the data, what is the rate of your data growth, what are your retention requirements in terms of age.
 
Yeah Seagate Archive HDD- 3 Year Warranty Seagate Archive HDD 8TB - Serial ATA III (SATA3) 6GB/s with 128MB Cache @ 5900rpm
is R3800 ex vat = <R24 for 48TB
 
kbilly said:
Hi could you elaborate why I would need double the storage for the amount of data I have?
Click to expand...
Backups. Seen RAID arrays fail out of the blue before, with no backups for months. I'd get a 10Gb link running between two servers, and then replicate between the two. Sure will be expensive, but it's better than seeing 40TB of data go bye bye.
 
If the possibility and budget for a 400 MBps internet connection is there, then I'd say go for that. Upload everything to the cloud as a first step and then assess the options to add local storage later for quick access to the data etc.

You want 40 TB, but from what I understand and what the oaks in this thred says, you'll actually need 80 TB in hard drive space for redundancy, so going for the cloud solution as a first step should be a lot cheaper.
 
kbilly said:
Hi everyone, we have a need for a large capacity storage solution 40Tb+ for long term storage of video material. We will mostly be writing. I am looking at something like a 6 bay Raid tower with 6x8Tb seagate archiving HDDs, probably USB3.

Anyone done this or have any advice regarding methos/brands, I seem to have options around R40k for Raid tower + 6 disks.

Looking at Raid 3 since integrity is most important and we will not be perfoming regular reads.
Thank you
Click to expand...

CHASSIS WITH DUAL POWER + HBA CARD + ZFS + LOTS OF DISKS = AWESOME STORAGE.

Why?:

* ZFS has snapshots.
* ZFS has compression and does not suffer from bit rot like traditional RAID.
* Ability to load hot spares that automatically insert themselves in to the RAID if any of the drives fail.
* Can mirror the RAID to another storage box via ZFS SEND / RECV so you have the exact same copy of data in another location.
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X