Need to upload 16TB to cloud storage for PhD. How?

Fireball

Well-Known Member
Joined
Feb 20, 2011
Messages
160
Hi all

A friend of mine is doing her PhD at University of Stellenbosch. She has about 12TB of wildillife acoustic recordings used for analysis and study, which has taken many years to collect and save. I'm trying to help her come up with a solution to back up her data, because having irreplaceable data on stored on single drives is not a good backup strategy.

I know Google Drive offers unlimited storage for 10USD a month on their business package. That should be fine and reasonably priced.

The trouble comes in with the bandwidth to actually upload all that data. Neither of us has fiber (LTE only, ~8Mpbs, 50g cap per month).

I'm thinking about asking the IT guys at Stellenbosch varsity if they may be willing to allow her some storage on one of their file servers (12TB is a lot though), and then from there it can be uploaded to Google Drive using the University's gbps fibre line. Does that seem like a reasonable option? Uploading it on a normal residential line would take weeks or months.

Alternatively I guess there must be some data center that could perhaps be able to assist? But from what I've seen they're quite expensive. My friend's income is only the stipend she gets for her research, so affordability is a big problem.

I know I'm asking for champagne with a lemonade budget, but I really want to help her. I unfortunately have no ties or contacts at Stellenbosch varsity.

Do you fine folks maybe have any ideas? I could really use some input (or a favor I could perhaps repay somehow?)

Thank you for your time.
 

ActivateD

Expert Member
Joined
Jun 7, 2004
Messages
1,335
I say talk to the University as she is doing her PhD at Stellies, so they might be willing to assist. I also say get her professor involved and to motivate to IT services so at least it doesn't seem like a student is just trying to kill the WAN. I dont know how fast Stellies internet is but when I was there it was damn quick so I would really push for that.
 

Fox1

Expert Member
Joined
Mar 3, 2009
Messages
4,641
Contact Vox. They used to allow clients to bring in their drives for cloud backup.
 

Ares1000101

Well-Known Member
Joined
Jun 6, 2011
Messages
444
Can these recordings not be compressed some what? I mean how many hours of audio are we talking? Perhaps the method of recording created unnecessarily large files that can be made tiny.
 

Johnatan56

Honorary Master
Joined
Aug 23, 2013
Messages
24,959
Can these recordings not be compressed some what? I mean how many hours of audio are we talking? Perhaps the method of recording created unnecessarily large files that can be made tiny.
Transcoding that entire footage would also probably take months plus require a lot of machine power...
 

Speedster

Executive Member
Joined
May 2, 2006
Messages
7,662
Transcoding that entire footage would also probably take months plus require a lot of machine power...
If it is wav audio compressing to aac / mp3 shouldn't be undoable

But back to the original question - I'm sure the IT guys at stellies will be able to make a plan.
 

InvisibleJim

Senior Member
Joined
Mar 9, 2011
Messages
999
Storage in MS Azure or AWS would probably be cheaper than a Dropbox or Google Drive type service. Depending on how frequently the data needs to be accessed, she might be able to make use of cool or archive tier storage. Amazon and MS both have procedures for sending large amounts of data on a hard drive to ingest into Azure or AWS Cloud Storage.
 

Johnatan56

Honorary Master
Joined
Aug 23, 2013
Messages
24,959
If it is wav audio compressing to aac / mp3 shouldn't be undoable

But back to the original question - I'm sure the IT guys at stellies will be able to make a plan.
What's the conversion rate, about 3-4 seconds per minute? And then audio quality loss when transcoding. Plus, are you sure it's wav to begin with?
I would take advantage of @websquadza, it's a nice offer of them.

@InvisibleJim Google's unlimited storage is still probably cheaper if you abuse the business tier at $10/month, but is limited to 1TB per user if less than 5 and has a max upload of 750GB (Linus Tech tips abused it, so they could have changed it).
G Suite for education (if the uni has it) basically counts as business G Suite btw, so needs more than 5 for unlimited.

So minimum 16 days, if 24 hours, need to average about 8.7MB/s ~70Mbps. On my 100/50 Mbps line I only get like 3 MB/s upload to Google Services. :(
Think it will be different on the ISP one, probably goes through Teraco instead of dumb IPC.
 
Last edited:

Concentric

Senior Member
Joined
Feb 16, 2017
Messages
767
What's the conversion rate, about 3-4 seconds per minute? And then audio quality loss when transcoding. Plus, are you sure it's wav to begin with?
I would take advantage of @websquadza, it's a nice offer of them.

@InvisibleJim Google's unlimited storage is still probably cheaper if you abuse the business tier at $10/month, but is limited to 1TB per user if less than 5 and has a max upload of 750GB (Linus Tech tips abused it, so they could have changed it).
G Suite for education (if the uni has it) basically counts as business G Suite btw, so needs more than 5 for unlimited.

So minimum 16 days, if 24 hours, need to average about 8.7MB/s ~70Mbps. On my 100/50 Mbps line I only get like 3 MB/s upload to Google Services. :(
Think it will be different on the ISP one, probably goes through Teraco instead of dumb IPC.
Google doesnt enforce the limit.
Ive got a single user GSuite account with 12TB sitting on it.....
 

Concentric

Senior Member
Joined
Feb 16, 2017
Messages
767
also if uploading to google drive i would highly recommend rclone as the method of transfer. It is far more robust than any of the traditional methoda
 

ponder

Honorary Master
Joined
Jan 22, 2005
Messages
76,501
I'm trying to help her come up with a solution to back up her data, because having irreplaceable data on stored on single drives is not a good backup strategy.
Before you hand drives over to anybody I suggest you backup to spare drives in case of accidental data loss...
 

3WA

Executive Member
Joined
Sep 25, 2012
Messages
6,019
Doesn’t she have office space at Stellies with access to fast broadband? If not, her supervisor certainly does.
 

agentrfr

Expert Member
Joined
Jul 8, 2008
Messages
4,681
@ActivateD what format is the audio? If it is PCM or WAV you can encode to flac (i.e. lossless) at a level setting of 0. In my experience that is like a 50% storage saving (on WAV and PCM), and with a half decent processor like a Ryzen 2700 for me the encodes are possible at about 700x realtime. Maybe encode for a day or two? 6TB is a little more managable than 12TB :)
 

ActivateD

Expert Member
Joined
Jun 7, 2004
Messages
1,335
@ActivateD what format is the audio? If it is PCM or WAV you can encode to flac (i.e. lossless) at a level setting of 0. In my experience that is like a 50% storage saving (on WAV and PCM), and with a half decent processor like a Ryzen 2700 for me the encodes are possible at about 700x realtime. Maybe encode for a day or two? 6TB is a little more managable than 12TB :)

Hi you have me confused with @Fireball who started the conversation. @Fireball please see the above post.
 
Top