We apologize for the downtime

[rpm]

Hi folks

We experienced an outage of a few hours today and would like to apologize for this inconvenience. The power went out in the Verizon Data Centre in Johannesburg at around 16:00 where our servers are hosted at Hetzner, and this resulted in the downtime which lasted a few hours.

According to unconfirmed feedback the building was hit by lightning which in turn caused a problem with the communication between the UPS system and the backup generators. When the power came back there were problems with the firewalls/switches on Hetzner’s side. When we hear more we will post the feedback here.

We again apologize for the website unavailability during this period. At least a few guys were having beers and a braai in Cape Town, Johannesburg and Pretoria at the get-togethers.

Regards,

RPM

 

[CrzWaco]

Wow was thinking I was going crazy when I did not see an announcement for downtime...

 

[The_Unbeliever]

We again apologize for the website unavailability during this period. At least a few guys were having beers and a braai in Cape Town, Johannesburg and Pretoria at the get-togethers.

Regards,

RPM

Hi RPM

Thanks for the notification. Luckily the get-togethers was in full swing and very few people was affected by this outage. Most, if not all, was still having a good time, and the few (such as ourselves) headed back home for bed.

Hope more information is forthcoming aboout this issue.

Regards

Libs

 

[Peder]

Thanx For letting us know, at least most of us were at the meets

 

[|tera|]

Thanks uncle rpm

WebAfrica was down as well today during the same time.

As the alerts on their site specify:

Our upstream connectivity provider is having an issue in the Johannesburg region which is affecting their entire network.

It is currently affecting all international access to our local datacentre networks and may be affecting access to the datacentre from specific regions of the country as well.

Verizon Engineers are attending to the problem.

Though the larger issue is still prevalent on the main network, most traffic has been re-routed.

It may still be affecting access to the datacentre from specific regions of the country. Verizon Engineers are attending to the problem and we will update here as we receive more information.

 

[RSkeens]

Thanks for the clarification regarding the outage, I missed this place

 

[Nicci]

Thanks for letting us know RPM! I was getting withdrawal symptoms lol!

 

[shani786]

Thanks for the info.

 

[semiautomatix]

It was crazy storms in Joburg today. Not surprised that there was a problem!

 

[Milano]

Is it my imagination or have they had a few problems recently?

 

[|tera|]

Is it my imagination or have they had a few problems recently?

They have had problems for the past year

 

[AirWolf]

It was crazy storms in Joburg today. Not surprised that there was a problem!

Now you know what we've been going through down here the last couple of months.

They have had problems for the past year

Yep. I only tried to log in today after 4. Then when I couldn't log in after 9 I was thinking like WTF, this place off-line for so many hours, must be something wrong with my internet connection.

 

[koffiejunkie]

According to unconfirmed feedback the building was hit by lightning

Given that they went and built their data center on a hill in an area that is prone to lightning, I'm surprised this didn't happen earlier.

 

[ToxicBunny]

koffiejunkie : I actually don't think this is the first time this has happened to Verizon....

But thanks for letting us know what happened rpm.. I was going nuts testing all sorts of issues from my side trying to get things to work again...

 

[gregmcc]

Thanks for the info. More downtime for Verizon - it ain't looking god for them

 

[The_Unbeliever]

koffiejunkie : I actually don't think this is the first time this has happened to Verizon....

But thanks for letting us know what happened rpm.. I was going nuts testing all sorts of issues from my side trying to get things to work again...

Will you visit them in person and k*k them out big-time?

 

[WildPikachu]

Given that they went and built their data center on a hill in an area that is prone to lightning, I'm surprised this didn't happen earlier.

Someone didn't do their research when they looked for where to setup shop.

In any case haven't they heard of proper lightning rods? there is equipment to prevent lightning damage to equipment. Everytime there is a lightning storm you dont see Telkoms microwave dishes, antennas ... etc being hit way way way up there in the sky ... why? because they have invested money in the proper equipment.

 

[koffiejunkie]

koffiejunkie : I actually don't think this is the first time this has happened to Verizon....

Fortunately, during the time when I had a box there, it never did.

In any case haven't they heard of proper lightning rods? there is equipment to prevent lightning damage to equipment.

It doesn't prevent. It reduces the probability. Lightning doesn't always hit the highest point or metal. I've seen lightning hit the grass right in-between a house with a straw roof and the lightning rod.

Everytime there is a lightning storm you dont see Telkoms microwave dishes, antennas ... etc being hit way way way up there in the sky ... why? because they have invested money in the proper equipment.

Actually, the do get hit. But like you say, they have the right equipment for the conditions.

 

[rebel998]

I have seen Telkom cable that was hit by lightning where the actuall copper inside totally disintegrated!

And true, you cannot prevent it just try and minimise the effect it has on your equipment.

 

[.Froot.]

Hi folks

We experienced an outage of a few hours today and would like to apologize for this inconvenience. The power went out in the Verizon Data Centre in Johannesburg at around 16:00 where our servers are hosted at Hetzner, and this resulted in the downtime which lasted a few hours.

According to unconfirmed feedback the building was hit by lightning which in turn caused a problem with the communication between the UPS system and the backup generators. When the power came back there were problems with the firewalls/switches on Hetzner’s side. When we hear more we will post the feedback here.

We again apologize for the website unavailability during this period. At least a few guys were having beers and a braai in Cape Town, Johannesburg and Pretoria at the get-togethers.

Regards,

RPM

Now we know where it is hosted....
/prepares for a full-scale nuclear attack