Need to upload 16TB to cloud storage for PhD. How?

Fireball

Fireball

Well-Known Member
Joined
Feb 20, 2011
Messages
190
Reaction score
53
Location
Stellenbosch
Hi all

A friend of mine is doing her PhD at University of Stellenbosch. She has about 12TB of wildillife acoustic recordings used for analysis and study, which has taken many years to collect and save. I'm trying to help her come up with a solution to back up her data, because having irreplaceable data on stored on single drives is not a good backup strategy.

I know Google Drive offers unlimited storage for 10USD a month on their business package. That should be fine and reasonably priced.

The trouble comes in with the bandwidth to actually upload all that data. Neither of us has fiber (LTE only, ~8Mpbs, 50g cap per month).

I'm thinking about asking the IT guys at Stellenbosch varsity if they may be willing to allow her some storage on one of their file servers (12TB is a lot though), and then from there it can be uploaded to Google Drive using the University's gbps fibre line. Does that seem like a reasonable option? Uploading it on a normal residential line would take weeks or months.

Alternatively I guess there must be some data center that could perhaps be able to assist? But from what I've seen they're quite expensive. My friend's income is only the stipend she gets for her research, so affordability is a big problem.

I know I'm asking for champagne with a lemonade budget, but I really want to help her. I unfortunately have no ties or contacts at Stellenbosch varsity.

Do you fine folks maybe have any ideas? I could really use some input (or a favor I could perhaps repay somehow?)

Thank you for your time.
 
I say talk to the University as she is doing her PhD at Stellies, so they might be willing to assist. I also say get her professor involved and to motivate to IT services so at least it doesn't seem like a student is just trying to kill the WAN. I dont know how fast Stellies internet is but when I was there it was damn quick so I would really push for that.
 
Contact Vox. They used to allow clients to bring in their drives for cloud backup.
 
Can these recordings not be compressed some what? I mean how many hours of audio are we talking? Perhaps the method of recording created unnecessarily large files that can be made tiny.
 
Ares1000101 said:
Can these recordings not be compressed some what? I mean how many hours of audio are we talking? Perhaps the method of recording created unnecessarily large files that can be made tiny.
Click to expand...
Transcoding that entire footage would also probably take months plus require a lot of machine power...
 
Johnatan56 said:
Transcoding that entire footage would also probably take months plus require a lot of machine power...
Click to expand...
If it is wav audio compressing to aac / mp3 shouldn't be undoable

But back to the original question - I'm sure the IT guys at stellies will be able to make a plan.
 
Storage in MS Azure or AWS would probably be cheaper than a Dropbox or Google Drive type service. Depending on how frequently the data needs to be accessed, she might be able to make use of cool or archive tier storage. Amazon and MS both have procedures for sending large amounts of data on a hard drive to ingest into Azure or AWS Cloud Storage.
 
Speedster said:
If it is wav audio compressing to aac / mp3 shouldn't be undoable

But back to the original question - I'm sure the IT guys at stellies will be able to make a plan.
Click to expand...
What's the conversion rate, about 3-4 seconds per minute? And then audio quality loss when transcoding. Plus, are you sure it's wav to begin with?
I would take advantage of @websquadza, it's a nice offer of them.

@InvisibleJim Google's unlimited storage is still probably cheaper if you abuse the business tier at $10/month, but is limited to 1TB per user if less than 5 and has a max upload of 750GB (Linus Tech tips abused it, so they could have changed it).
G Suite for education (if the uni has it) basically counts as business G Suite btw, so needs more than 5 for unlimited.

So minimum 16 days, if 24 hours, need to average about 8.7MB/s ~70Mbps. On my 100/50 Mbps line I only get like 3 MB/s upload to Google Services. :(
Think it will be different on the ISP one, probably goes through Teraco instead of dumb IPC.
 
Last edited:
Johnatan56 said:
What's the conversion rate, about 3-4 seconds per minute? And then audio quality loss when transcoding. Plus, are you sure it's wav to begin with?
I would take advantage of @websquadza, it's a nice offer of them.

@InvisibleJim Google's unlimited storage is still probably cheaper if you abuse the business tier at $10/month, but is limited to 1TB per user if less than 5 and has a max upload of 750GB (Linus Tech tips abused it, so they could have changed it).
G Suite for education (if the uni has it) basically counts as business G Suite btw, so needs more than 5 for unlimited.

So minimum 16 days, if 24 hours, need to average about 8.7MB/s ~70Mbps. On my 100/50 Mbps line I only get like 3 MB/s upload to Google Services. :(
Think it will be different on the ISP one, probably goes through Teraco instead of dumb IPC.
Click to expand...
Google doesnt enforce the limit.
Ive got a single user GSuite account with 12TB sitting on it.....
 
also if uploading to google drive i would highly recommend rclone as the method of transfer. It is far more robust than any of the traditional methoda
 
Fireball said:
I'm trying to help her come up with a solution to back up her data, because having irreplaceable data on stored on single drives is not a good backup strategy.
Click to expand...

Before you hand drives over to anybody I suggest you backup to spare drives in case of accidental data loss...
 
Doesn’t she have office space at Stellies with access to fast broadband? If not, her supervisor certainly does.
 
@ActivateD what format is the audio? If it is PCM or WAV you can encode to flac (i.e. lossless) at a level setting of 0. In my experience that is like a 50% storage saving (on WAV and PCM), and with a half decent processor like a Ryzen 2700 for me the encodes are possible at about 700x realtime. Maybe encode for a day or two? 6TB is a little more managable than 12TB :)
 
agentrfr said:
@ActivateD what format is the audio? If it is PCM or WAV you can encode to flac (i.e. lossless) at a level setting of 0. In my experience that is like a 50% storage saving (on WAV and PCM), and with a half decent processor like a Ryzen 2700 for me the encodes are possible at about 700x realtime. Maybe encode for a day or two? 6TB is a little more managable than 12TB :)
Click to expand...


Hi you have me confused with @Fireball who started the conversation. @Fireball please see the above post.
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X