MTN Business Data Centre outage

...and I thought by farting in the TK I caused this downtime :o

But seriously I was stationed at <site> which had a generator AND UPS.

Part of my duties was to test the genset on a monthly basis - and we never had any problems.

Power failures happened - and when it did, we never had any downtime due to a faulty UPS/genset or no diesel.

One month an Eskom pole close to the site burnt down due to a veld fire, and we were without electrical power for 5 days - and also when an Eskom distribution box literally blew up.

The other site had endless problems with their generator - the diesel either ran out, or they forgot to close the tap after filling up the tank with diesel so teh diesel would be spilled over the whole place... :rolleyes:

But my genset was the cleanest and neatest of the lot :D

So - in light of my experience, an excuse like "the genset blew up" does not make sense, because when you purchase a genset, you have to spec it correctly in order to ensure that you have enough power now, and also to cater for future expansion.

Also, somebody will have to be kept responsible for things like checking the diesel level, perform routine inspection (oil level, water level etc) and the such. But people will be people, and be lazy, and say that they did do routine inspection when all they did was nothing.

So... yes... quite interesting to see somebody who's supposedly to be clued up and on the ball to fumble like this.

EDIT : When the site was built, the electical usage of the site was estimated, and erred on the high side. So the generator is a bit overspecced, but it is best to be a bit overspec than to be too low with the specs and have endless hassles.
 
Last edited:
There are still techies who take pride in their work. Old school - everything must shine!
 
cavedog said:
What is the chances that 2 major DC's will fail just minutes apart and leave SA web in utter darkness!
Click to expand...

One in a million chance, but, according to the Discworld mathematicians, this will be nine times out of ten :D

So the possibility is there.
 
How does a main generator blow itself up?? The better ones can last for 15 - 20 years if maintained properly.
 
moggie said:
How does a main generator blow itself up?? The better ones can last for 15 - 20 years if maintained properly.
Click to expand...

It was either

1 - Not inspected regularly, and ran out of either oil or water
2 - Not serviced regularly, and seized (yes, diesel engines can seize when the oil is too old)
3 - Was underspecced and overloaded
4 - Got stolen and they're reluctant to admit it :p

On one of our sites we got an old genset which've started to give problems - it is in for overhauling at the moment.
 
keru said:
Totally uncalled for. Mistakes are made by people of all colour.
Click to expand...

Everyone makes mistakes, but I'm referring to a systemic problem that I've noticed with engineers in this country; They chase out all the old white guys (the only guys with real experience - whether you like it or not), and replace them with inexperienced varsity grads who happen to be mostly BEE appointments. For the record, I'd rather have an experienced black engineer designing a project than a recently graduated white guy. The sort of design features which prevent these "unexpected problems" from downing an entire DC, are not taught from textbooks. Without the experienced okes guiding them, the new grads can only learn by trial and error. Errors which caused the chaos we saw today.
 
10 years ago I installed a banking system in Namibia. We had a Generator and UPS and plenty of power failures. The generator UPS combination performed flawlessly each time. Now if we could get it right 10 years ago? German precision in planning implementing and testing. :)
 
UPDATE: At approximately 18h00 MTN Business again lost power to JHB DC2. It was restored at 18h15. Servers are being booted up in sequence.
Click to expand...

Someone needs a new generator ;)
 
dabbler said:
Someone needs a new generator ;)
Click to expand...

Something like this?

http://mybroadband.co.za/news/Telecoms/10606.html

For power, there are dual 2400kW Uninterruptable Power Supplies (UPS) for the data room floor, each capable of powering the entire data room. There are dual 2300kW supplies for the cooling system.

Eskom provides power from two separate substations (Stikland and Oakdale) but as a failsafe, there are diesel generators off site. Four 16 cylinder 65.37 litre Mitsubishi engines, and two 20 cylinder 95.4 litre MTU Friedrichschafen engines can combine to guzzle the 78,000 litres of diesel stored on site, for 72 hours of standby full load power.
Click to expand...
 
Chaos at MTN Business

I was busy doing some work at a large MTNBusiness client nearby when the DC outage happened. This client lost ~80Mbits of uplink, but could luckily switch to IS as backup. Needless to say MTNBusiness is rapidly loosing traction within this company.

Let me just add that MTNs call center number was either offline or busy the whole time and neither of the two account managers we have answered their mobile phones during this time.

I popped into MTN's Gallo Manor DC (where the outage happened). It was total and absolute chaos. Besides the fact that there was almost no parking left due to all the clients coming in to fix machines (yes, many power supplies were burned out), the DC was completely understaffed.

We all stood in a long queue at the sign-in desk and the poor guards couldn't keep up with the influx of customers - this shows even more bad planning on MTN's side when dealing with outages of this nature. Customers were literally fighting for trolleys (screen+keyboard on wheels) and guards were running around trying to unlock cabinets for the customers, fumbling due to the onslaught of customers.

Everywhere groups of customer techies were standing around, discussing which alternative datacenters exist. TeraCo was mentioned frequently during these discussions, Cybernest also came up. There was almost a sense of camaraderie as techies shared screwdrivers, helped with unracking of servers and offering advice here and there.

One techie I spoke to said “it’s a joke”, another said “I have over 3000 domains hosted here, but I’m moving now, no matter how much pain it’s gonna cause”.

At that point I left the chaos behind and exited the DC into an almost-cloudburst.

--deckert
 
The_Librarian said:
...and I thought by farting in the TK I caused this downtime :o

But seriously I was stationed at <site> which had a generator AND UPS.

Part of my duties was to test the genset on a monthly basis - and we never had any problems.

Power failures happened - and when it did, we never had any downtime due to a faulty UPS/genset or no diesel.

One month an Eskom pole close to the site burnt down due to a veld fire, and we were without electrical power for 5 days - and also when an Eskom distribution box literally blew up.

The other site had endless problems with their generator - the diesel either ran out, or they forgot to close the tap after filling up the tank with diesel so teh diesel would be spilled over the whole place... :rolleyes:

But my genset was the cleanest and neatest of the lot :D

So - in light of my experience, an excuse like "the genset blew up" does not make sense, because when you purchase a genset, you have to spec it correctly in order to ensure that you have enough power now, and also to cater for future expansion.

Also, somebody will have to be kept responsible for things like checking the diesel level, perform routine inspection (oil level, water level etc) and the such. But people will be people, and be lazy, and say that they did do routine inspection when all they did was nothing.

So... yes... quite interesting to see somebody who's supposedly to be clued up and on the ball to fumble like this.

EDIT : When the site was built, the electical usage of the site was estimated, and erred on the high side. So the generator is a bit overspecced, but it is best to be a bit overspec than to be too low with the specs and have endless hassles.
Click to expand...

That, and where were their secondary diesel gen? Did it also magically blow up?

all this happened before at MTN
 
AcidRaZor said:
That, and where were their secondary diesel gen? Did it also magically blow up?

all this happened before at MTN
Click to expand...

And they claimed a fully redundant system...

I would also expect backup generators for the backup generators.
 
1st line of protection : UPS
2nd line of protection : generator
3rd line of protection : backup generator

A lot of money, but if you're in a mission-critical business, then you cannot afford to [-]diddle-daddle[/-][-]shilly-shally[/-]fiddle and diddle around.

Maybe the exodus of clients will teach them a lesson - but that will be like bolting the stable door after the horse [-]have bolted[/-]got stolen.

And maintenance on the UPS and gensets is not cheap either... but it is neccessary.
 
The_Librarian said:
1st line of protection : UPS
2nd line of protection : generator
3rd line of protection : backup generator

A lot of money, but if you're in a mission-critical business, then you cannot afford to [-]diddle-daddle[/-][-]shilly-shally[/-]fiddle and diddle around.

Maybe the exodus of clients will teach them a lesson - but that will be like bolting the stable door after the horse [-]have bolted[/-]got stolen.

And maintenance on the UPS and gensets is not cheap either... but it is neccessary.
Click to expand...

Actually it's more like this in normal data centers:

Primary UPS bank -> Diesel Generator -> Backup Diesel Generator
Secondary UPS bank -> Diesel Generator -> Backup Diesel Generator

Those people with redundant power supplies have immediate benefit of the secondary UPS bank's

With above system, they can do maintenance on the primary diesel engine even if there is a power outage, the backup gen should kick in. Same with doing maintenance on the backup gen. Primary is available and ready to rock.

So if they'd like to do maintenance on the primary UPS bank? Anytime, switch over to the secondary UPS bank, do your maintenance... done. If there's a power outage it's taken care of. Same with doing maintenance on the secondary.

It really isn't rocket science...
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X