Amazon EC2 explosion ... might be why some uk/europe-hosted sites are down

Interesting news. The writer of the article doesn't seem to be very familiar w/ the Amazon cloud though:

Are we to believe that Amazon hadn’t planned for this kind of eventuality?

Click to expand...

They planned for it, but true redundancy is expensive so only the top level packages have it. You can select for example to use the cloud datacentres in Europe *and* N America...then naturally you're fine if EU gets knocked out.

ah interesting. i think it's a fair point though that any data centre should be able to withstand a storm on its own, you know? why should we have to pay for true redundancy when amazon should be building centres with 101 failovers to begin with?

jct said:

ah interesting. i think it's a fair point though that any data centre should be able to withstand a storm on its own, you know? why should we have to pay for true redundancy when amazon should be building centres with 101 failovers to begin with?

Click to expand...

Lighting storms are tricky. Mostly easy to survive, until it hits a part of the power grid with a direct strike. Then it doesn't really matter what you did to protect it...its no use. Thats what happened here.

jct said:

why should we have to pay for true redundancy when amazon should be building centres with 101 failovers to begin with?

Click to expand...

The customer is footing the bill either way.

True redundancy is tricky. Take Google...they duplicate each piece of data at least 3 times...some 6 times. Now if you combine that with the data volumes used (think 7gb gmail inbox) and how many people use it that becomes an unmanageable amount of data real fast. Even if you can do it, some small startup will take a risk on redundancy, undercut your price by half & then suddenly you're in real trouble on the business side.

Fail-over 101 is also easier said than done. Requests are usually routed to the nearest data center that has the relevant data for speed & efficiency. Hypothetical scenario: Lets say there are 3 of these, all manage day to day traffic easily, never going above 60% usage. So now one of the 3 falls over (for whatever reason, electric storm if you like). That leaves 2 centers with just enough capacity (90%) to cope. Excellent....until one of the 2 remaining centers is even a tiny bit closer to the bulk of the traffic so it gets 20% extra traffic...which pushes it over the limit. Naturally the 3rd one can't cope on its own so falls too. So you went from 3 centres with barely above half load to all 3 100% dead in a couple of seconds. Notice also that we started off with 1.2 centers worth of capacity standing totally idle which kills profitability...and it wasn't even enough.

Even Google is battling with this stuff. The above scenario is actually roughly based on a google incident. They pulled a part of their cloud for maintenance assuming automatic failover will cover for it. And it did....except the automatic routing to closest center nailed one particular center.

More:
http://www.datacenterknowledge.com/...knocks-amazon-microsoft-data-centers-offline/

EC2 is quite expensive. I've managed to build up a bill of $700 within 4 months with only 2 small servers idling.

I hope for the EC2 users' part that it is only the network equipment that got damaged due to the lightening storm and not the storage devices too.

from what i've heard from people i know who use EC2, there is a LOT more wrong and this is far from fully resolved.