MTN Business Data Centre outage

/facepalm

There is seriously flawed logic exhibited by some people here, please don't ever take up a career in designing critical power systems.
 
ponder said:
/facepalm

There is seriously flawed logic exhibited by some people here, please don't ever take up a career in designing critical power systems.
Click to expand...

Really? care to point out the flaws in my logic? I'm sure there are many, hence why I can't put a finger on it
 
So is the MTN Gallo Manor DC a Tier 4? What about SLA agreements, are there any with MTN Business?
 
AcidRaZor said:
Really? care to point out the flaws in my logic? I'm sure there are many, hence why I can't put a finger on it
Click to expand...

It's a regulatory requirement to shut off generated power in a fire event apparently.
 
I might buy that explanation in isolation but didn't MTN have another major outage due to generator failure about two months ago?

Seriously, I know it costs money but get them serviced once in a while.
 
Most generators are self-contained units - so a fire in one of them won't threaten the rest of the building.

The one which I was working with many moons ago, was such an unit - separate from the main building itself.
 
The_Librarian said:
Most generators are self-contained units - so a fire in one of them won't threaten the rest of the building.

The one which I was working with many moons ago, was such an unit - separate from the main building itself.
Click to expand...

That's my question... why were both gennies in the same room/building?
 
But a proper DC will have redundant systems separately (separate buildings, separate from the main DC itself), or am I missing something here? :confused:
 
The_Librarian said:
But a proper DC will have redundant systems separately (separate buildings, separate from the main DC itself), or am I missing something here? :confused:
Click to expand...

One would think so....

FWIW, this is actually a Verizon datacenter acquired by MTN in the merger, so I think they are fixing all that stuff slowly.
 
cda said:
It's a regulatory requirement to shut off generated power in a fire event apparently.
Click to expand...

Yes, I understand that. But to shut off the backup generator AS WELL as the rest of the power?

Or do you mean _all_ generated power even if it's not affecting each other?
 
AcidRaZor said:
Really? care to point out the flaws in my logic? I'm sure there are many, hence why I can't put a finger on it
Click to expand...

Was not referring to you but some other individuals ;)

For example, why house all the gennies in the same building, same goes for ups etc. Everything should be contained separately in all stages of the power feed all the way from fuel tanks (contamination etc) to DB's with different cable routes etc. Split everything and keep it all separate with the option to cross connect as well..
 
ponder said:
For example, why house all the gennies in the same building, same goes for ups etc. Everything should be contained separately in all stages of the power feed all the way from fuel tanks (contamination etc) to DB's with different cable routes etc. Split everything and keep it all separate with the option to cross connect as well..
Click to expand...

+1

Now that would be a true redundant system.
 
Crowley said:
I don't work at Gallo Manor so I don't know the finer details but this is what happened yesterday.

Firstly...
MTN Business <--- (Huge gap) ---> MTN SP
They may share a name but they are two different companies.

The Eskom power went down at the Gallo Manor hosting facility and when the first generator kicked in a oil line running to the generator broke. The line started spewing oil over the exhaust which created a crapload of smoke and set off the fire alarms. That also caused the first generator to "explode". The fire alarms are set that if they detect a fire it shuts down everything including the automatic failover and that prevented the backup generator from kicking in.

Both the backup generators have weekly, monthly and quarterly maintenance schedules that are done and checked. There simply was no way that anybody could have forseen this happening.

*disclaimer*

This is a unofficial recount of what happened. A official explanation should follow some time today.
Click to expand...
Thanks for that :).

Any news on why or what caused that oil line to break, and was it metal tubing or a rubber hose?
 
some thoughts on reading this thread...

- MTNB HAS redundant systems, this was a MECHANICAL failure that could not be predicted. All generators are serviced like they should be, I don't think MTNB will make such a stupid mistake... and it's pessimistic to think they will...

- Cost cutting was mentioned, like they don't service generators to save money... I can assure you that this downtime cost MTNB quite a penny, especially with their uptime SLA to certain clients... there is no way they will not care if a DC goes down...

- the amount of technicians can never be 1 to 1, or even 1 to 100... If you see how many servers are located in the Gallo Manor DCs, they will need so many techies that they will sit and do nothing for the 99.9% of the time that the DC is up... You plan on uptime, not downtime. I assure you MTNB was crapping themselves when this happened.

The point here is ***** happens, this was one of those cases... You can do your best to achieve great uptime, but you'll never Murphy-proof any DC....
 
TBlaar said:
some thoughts on reading this thread...

- MTNB HAS redundant systems, this was a MECHANICAL failure that could not be predicted.
Click to expand...

The failure happened to one device. The entire system died

that does not equals redundant.
 
the failure happened to one system, which rendered it unable to fail over to the secondary unit.... so redundancy is in place. It just didn't work this time... :)
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X