Blob vs File system storage

[Gnome]

What gkm said, Amazon S3 or similar.

Not really sure why you would use a database or filesystem over that.
The costs would almost certainly be higher (taking into cost maintenance, storage, etc.)

 

[Thor]

What gkm said, Amazon S3 or similar.

Not really sure why you would use a database or filesystem over that.
The costs would almost certainly be higher (taking into cost maintenance, storage, etc.)

Because I'm not looking at storing 100Gb's of user profile images.

 

[Gnome]

Because I'm not looking at storing 100Gb's of user profile images.

Yeah, but don't you want the cheapest and simplest solution?
I mean filesystem is bad for many reasons:
1) Migration will be a nightmare
2) You need to worry about data corruption
3) You need to monitor file size
4) You need to actually write the software or at least install software to manage the storing of the files, path it for url access, etc.

Or I guess if you typically just put software down and never monitor it nor care about the quality then you can forget about points 1-3.
Still point 4 is a mission.

Over that S3 or equivalent it is a breeze (Step 1 create bucket, step 2 add files, step 3 enable http sharing, step 4 profit, no more than 5 minutes of work)
Database can work assuming it is a small amount of data.

 

[[)roi(]]

Even Amazon S3 isn't a hammer for every nail: Amazon's own services are still heavily dependant on Akamai CDN.

The last points against file system are opinions; opinions != fact.

 

[Willie Trombone]

Why store them at all?

Let uses link them from independent hosts that aren't your problem.

Extra name lookups, though that can be mitigated by a good CDN. Depends on the size of the images though.

On blob vs file, database storage is often more expensive than file.
How I'd approach it is... does being able to use or reference the image in the DB make your app easier and more logical - e.g. linking metadata to an image... if so, yay, otherwise nay. File system reference can be done faster and asynch too.

 

[Swa]

Files should always go in the file system. That's what it's optimised to do and the web server is better equipped to serve files than data.

Serving from the database would require running a program such as php, doing a lookup, transferring data to memory, and then outputting to the web server. All bottlenecks in the system. Then using unspecified rather than fixed length data structures turns your whole database field into an unspecified structure so field lookups takes longer.

The web server has better tools including DMA so files can be transferred to memory without even involving the CPU.

 

[Murmaider]

It's better to store them on a file system and then use something like varnish which will cache them in memory, this is far faster then retrieving files from a database.

Another rule of thumb is that "flat files don't crash, Databases do". From a risk and recovery perspective, if you can store it to disk, rather do it because it is safer.

 

[gkm]

What gkm said, Amazon S3 or similar.

Not really sure why you would use a database or filesystem over that.
The costs would almost certainly be higher (taking into cost maintenance, storage, etc.)

My wife got this going to upload profile pictures for users to S3 and display directly from S3 in less than an hour using Rails (migrated from using file system), in her first attempt to use S3. She never looked back and have not had a moment's hassle in this area. Also, her webserver file system in now completely expendable, so no need to even bother to back it up. (Can restore website directly from Git and/or DB backup.) Saved her money and admin effort and reduced load on the webserver, since it only serves a few very small images, so keeps the site very snappy. Was so happy with the outcome that she moved most of the other more static images on the site to another bucket in S3. That website handles big load spikes without even blinking, since all the heavy lifting has been delegated to S3.

Getting started:
https://aws.amazon.com/articles/4261

 

[Thor]

My wife got this going to upload profile pictures for users to S3 and display directly from S3 in less than an hour using Rails (migrated from using file system), in her first attempt to use S3. She never looked back and have not had a moment's hassle in this area. Also, her webserver file system in now completely expendable, so no need to even bother to back it up. (Can restore website directly from Git and/or DB backup.) Saved her money and admin effort and reduced load on the webserver, since it only serves a few very small images, so keeps the site very snappy. Was so happy with the outcome that she moved most of the other more static images on the site to another bucket in S3. That website handles big load spikes without even blinking, since all the heavy lifting has been delegated to S3.

Getting started:
https://aws.amazon.com/articles/4261

What does it cost?

 

[Gnome]

Even Amazon S3 isn't a hammer for every nail:

True, but in this case it matches 100% what S3 was made for.

Amazon's own services are still heavily dependant on Akamai CDN.

What? Truly I can't see how and where S3 makes use of Akamai.
If that were true however, not really sure why it is relevant.

But mostly point 1. S3 is a storage service. They have absolutely nothing to do with Akamai.

The last points against file system are opinions; opinions != fact.

Your counter argument to those points are that I'm using opinions? Not a single point challenged on its merits, simply that it is opinion.
Hmm ok.
You must be fun in design meetings.

What does it cost?

http://bfy.tw/8kXE

 

[gkm]

Even Amazon S3 isn't a hammer for every nail: Amazon's own services are still heavily dependant on Akamai CDN.

The last points against file system are opinions; opinions != fact.

Emmm: https://www.google.com/search?q=amazon+cdn&ie=utf-8&oe=utf-8

 

[Mr Scratch]

Amazon used to use Akamai, not sure if they still do. Even so, why does it matter?

 

[[)roi(]]

True, but in this case it matches 100% what S3 was made for.

Yet that was the OP's question: "Blob vs File system storage", or did I miss the part about S3 vs Blob vs File system storage. Going S3 has no bearing on that question, because both choices are supportive by S3 or any other service.

What? Truly I can't see how and where S3 makes use of Akamai.
If that were true however, not really sure why it is relevant.

But mostly point 1. S3 is a storage service. They have absolutely nothing to do with Akamai.

It's not relevant re OP's question, but it is relevant if we're comparing service options; hence the hammer adage.

Your counter argument to those points are that I'm using opinions? Not a single point challenged on its merits, simply that it is opinion.
Hmm ok.

Simple, I didn't agree.

You must be fun in design meetings.

How's that relevant. Unless you're either working for the government or SOEs; selection is usually a process (e.g. tender), based as example on an agreed weighted criteria.

Emmm: https://www.google.com/search?q=amazon+cdn&ie=utf-8&oe=utf-8

That doesn't confirm anything. They (and may others) still use Akamai, even though they have their own CDN. simply because Akamai network is superior i.t.o sheer number of servers or more specifically: the locality of Akamai servers >70K.

Amazon used to use Akamai, not sure if they still do. Even so, why does it matter?

Exactly, in the case "Blob vs file system" it has no bearing. However in the "Amazon S3 or similar" bit it does, re them eating their own dog food.

 

[gkm]

Note, I did not suggest using any CDN. I suggested serving directly from S3. So agree with others, CDN discussion not relevant here. But I guess CloudFront is an option if eventually needed.

In terms of suggesting S3, just wanted to point out there is a better option than both DB and file system, which as Gnome pointed out, is designed for exactly this kind of thing. Nice fire and forget exact match solutions are usually the best, rather than using something that will work, but needs constant attention.

 

[[)roi(]]

Note, I did not suggest using any CDN. I suggested serving directly from S3. So agree with others, CDN discussion not relevant here. But I guess CloudFront is an option if eventually needed.

In terms of suggesting S3, just wanted to point out there is a better option than both DB and file system, which as Gnome pointed out, is designed for exactly this kind of thing. Nice fire and forget exact match solutions are usually the best, rather than using something that will work, but needs constant attention.

If I understand correct.y then it's less about S3 and more about "Amazon Simple Storage Service?" vs Blob vs. file system vs... (e.g. CDN)

If that's the case then my biggest counter argument would be code / SDK lock-in.

 

[Willie Trombone]

If I understand correct.y then it's less about S3 and more about "Amazon Simple Storage Service?" vs Blob vs. file system vs... (e.g. CDN)

If that's the case then my biggest counter argument would be code / SDK lock-in.

Good point - why I'm not a big fan of App Engine and similar platforms.

 

[semaphore]

I use S3.

 

[f0Iygu7gXHVM]

Why store them at all?

Let uses link them from independent hosts that aren't your problem.

Tbh, never let the user insert their own remote content on your site.

 

[SauRoNZA]

Tbh, never let the user insert their own remote content on your site.

But it's not on your site.

It's only linked through by your browser really.

Also you could just use a known standard like Gravatar if you want to enforce it somewhat.

 

[f0Iygu7gXHVM]

But it's not on your site.

It's only linked through by your browser really.

Also you could just use a known standard like Gravatar if you want to enforce it somewhat.

It's not on your box but it is on your site.

https://www.owasp.org/index.php/Cross-site_Scripting_(XSS)