MTN Business JHB DC 2 - Down again today..

[JasonH]

So the MTN DC2 in JHB went down again today @ around 12.50PM and affected our services there and MyBB. What ever happened to generators and redundant failovers? It seems every time there's a power issue at MTN Business the datacentre goes offline :/

 

[Deckert]

They went down a little earlier than that. Our monitoring shows they went down at 12:46 and came up again at 13:06.

After the previous fiasco on 2 December 2009, where the MTN Business DC was effectively down from 14:30 to 18:20, our account manager ensured us that they put additonal measures in place to prevent future occurances. Looks like their promises were all for nothing.

After the last incident they said they will be constructing a secondary feed in the facility to create a 2x n+1 system. Completion of this project was scheduled for end Q1 2010. End Q1 was 31st of March, not sure if they made their deadline or not.

--deckert

 

[rpm]

MTN Business: Power Failure

Status: Resolved

Type: MTN Business: Power Failure

Start Date: 2010-04-20 12:53:00 SAST

Resolved Date: 2010-04-20 13:06:00 SAST

Point of impact: Johannesburg Data Centre 2

Symptoms experienced: Customers will be unable to access hosting services in the Johannesburg Data Centre 2.

Cause of problem: Power Failure

Estimated time of repair: 13 minutes

Additional information: A circuit breaker failure on the output of the UPS at Gallo Manor Data Centre caused a power outage in JNB2 DC.

Power was restored within 15 minutes and servers would have booted up shortly after.

Hetzner are waiting for further detail from MTN Business as to the exact nature of the failure.

Attending: Hetzner Engineers

 

[ksweb]

What's really annoying is the larger companies such as Hetzner (who host a fair amount of servers at Gallo manor) don't seem to be putting much pressure on MTN Business to sort all of these problems out once and for all.

 

[privet]

CAUSE OF PROBLEM: Emergency Maintenance
ADDITIONAL: Please note that we will be doing emergency maintenance
tomorrow morning 21 April 2010 from 01:00 to 02:00 at Gallo Manor
Datacentre Facility. We need to replace the faulty electronic module on
the Main Circuit Breaker which caused the power failure on the Eastern
Section of the Datacentre (Colocate 3, 4 & 5) today. The main circuit
breaker needs to be switched off for the module to be replaced. The
estimated downtime during the maintenance is 1 hour. MTN Business would
like to recommend where possible that customers have resources on site at
the Gallo Manor Datacentre to safely power down/up your equipment before
and after the maintenance. Alternatively please open a help desk ticket
with our Customer Support Centre to request an on-site MTN Business
engineer to power down/up your equipment accordingly. MTN Business would
like to sincerely apologies for the inconvenience caused.

 

[threegee]

I truly hope that the power maintenance last night (5 hours outage) actually fixes the power problems this time. There have been several abrupt outages in the last year and each time the MTN engineers promise that they are addressing the problem and then they schedule a maintenance window and promise faithfully that they have fixed the problem, then lo and behold it just happens again a few months later. I cant figure out how many times they need to upgrade circuit breakers and line feeds to be able to keep all systems running 24/7 ?? it's not rocket science and other data centres seem to have no problem keeping their system up 24/7. I host with Hetzner who is located in this data centre and it seems Hetzner like everyone else sharing this data centre are all at the peril of MTN. Hetzner is beyond world class but MTN hmmmm .... not so much.