MWeb Johannesburg DC Down?

I've rented a separate hard drive with them to store backups, everything gone of 130 clients and 14 years except the static stuff. Disaster.
 
sonicai said:
Unfortunately not. All data have been lost and attempts at data recovery failed (according to their support late yesterday afternoon). They said they will inform us if they were able to recover anything but I would recommend you assume the worst.
Click to expand...

This must be a nightmare for system admins....

Lawsuits are gonna fly everywhere soon....

I remember when this happened at mtn DC,
Facilities company got canned, and lawsuits ran into the millions.....
 
Sounds to me like they had a URE crash on their SAN. It happens to RAID 5 and 6 Arrays. Never use RAID 5 or 6 for data you cant afford to lose. It mostly happens during array rebuilds, but has been know to occur when you have 1 dodgy drive in an array.

It would explain the lengh of the outage and the resulting data loss. They spent dozens of hours trying to get data back, after which they gave up and setup the array from scratch.

So many VM "redundant" infrastructures rely on a single SAN device for storage back-end. Contrary to providing "redundancy" they actually amplify their risk of a failure and when it does go, it goes big-time.

This is what happens when you design hosted infrastructures according to the vendor's brochure and not with real world experience.
 
Last edited:
I am totally horrified by mwebs service.


Both of our vms have been down since Thursday night 24:00. One of them came back online this morning and is totally clean.

All Ip addresses have changed. Thank goodness I used dns addresses is clients software.

Not once during any of this was I contacted in anyway, I had even updated all technical contacts 2 weeks prior. I have been in the dark the whole time. Never did we get an eta to decide if we should start making other arrangements.

I think loosing everyone data is appalling. And their email they broadcast out, is of the attitude. Oh we sorry. Lets hope you have made backups. When I log into the my account page there are no snapshots there. I created one on each vm when I initially set them up.

Its understandable that things go wrong etc. But the way mweb just kept everyone in the dark is BS. As paying customers for a reliable service. Everyone should getting hourly updates of whats happening. So we can go an inform our customers.

They seem very happy to say that services should be resumed by the end of Sunday. But how is an empty VM services resumed? And again we haven't been informed. So now I have an empty VM. Can i begin restoration on the server or are they going to try and do a backup later and reset everything I have done?

I must say I have lost allot of faith in mweb the last 3 days...

Sorry for the ranting, But im sure im not the only **** of one.
 
Sapphiron said:
Sounds to me like they had a URE error crash on their SAN. It happens to RAID 5 and 6 Arrays. Never use RAID 5 or 6 for data you cant afford to lose. It mostly happens during array rebuilds, but has been know to occur when you have 1 dodgy drive in an array.

This is what happens when you design hosted infrastructures according to the vendor's brochure and not with real world experience.
Click to expand...
Careful of falling in the trap of thinking you have experience. When you don't know what you don't know you should plan for the worst-case scenario always. Multi-SAN setup is not going to save you from a careless "rm -rf /" or a bit-flip which results in data corruption. Humans and machines fail in unpredictable and spectacular ways and redundancy != backups ;)
 
ukj said:
Careful of falling in the trap of thinking you have experience. When you don't know what you don't know you should plan for the worst-case scenario always. Multi-SAN setup is not going to save you from a careless "rm -rf /" or a bit-flip which results in data corruption. Humans and machines fail in unpredictable and spectacular ways and redundancy != backups ;)
Click to expand...

Amen! if you think you know everything, you are John Snow.

Compartmentalize and never share the same authentication details across multiple devices in your infrastructure back-end.
 
Sapphiron said:
Amen! if you think you know everything, you are John Snow.

Compartmentalize and never share the same authentication details across multiple devices in your infrastructure back-end.
Click to expand...

And if that Y2K or leap second bug results in simultaneous data corruption on both devices? Backup to different medium/platform/city/country/continent/planet if you can. Remove single points of failure as much as cost permits ;)
 
Last edited:
My pain is the change in IP's, have backups with me, but try get firewall rules changed on a weekend, :twisted:...

Thx Murmaider will check your services

This outage does now make it easier to explain to a non technical client why they need redundancy, before they said "whats the chances of that happening". I never knew of any previous ones I could cite
 
eg2505 said:
What's a bit flip?

Does it only happen on raid 5-6 arrays?
Click to expand...
When a 1 becomes a 0 or vice versa either when the data is at rest or in transit. In a physical world components fail and electrical signals over wires are subject to interference.

It can happen in any system and it can be mitigated in several ways, but this is a very general discussion with very problem-specific nuances. ECC memory, parity/checksums in transit, shielded network mediums etc.

See https://en.wikipedia.org/wiki/Byzantine_fault_tolerance
 
Last edited:
eg2505 said:
What's a bit flip?

Does it only happen on raid 5-6 arrays?
Click to expand...

For what it is worth, the reason why it is called a bit "flip" is because circuits does not have 1's and 0's. But they "interpret" high and low voltages as 1's and 0's. So the voltage are never a constant "low" voltage, it can be anything between nothing and a threshold. So interference can cause this voltage to fluctuate. But as long as it stays low enough it is always seen as a 0. So when the interference is high enough, it would cause the voltage to go over the threshold, effectively causing the interpreter to assume a 1. So it basically builds up and eventually flip over to the other side. If you look at a voltage graph the line would increase and cross over to the other side where it is interpreted as a 1.
 
Waste said:
For what it is worth, the reason why it is called a bit "flip" is because circuits does not have 1's and 0's. But they "interpret" high and low voltages as 1's and 0's. So the voltage are never a constant "low" voltage, it can be anything between nothing and a threshold. So interference can cause this voltage to fluctuate. But as long as it stays low enough it is always seen as a 0. So when the interference is high enough, it would cause the voltage to go over the threshold, effectively causing the interpreter to assume a 1. So it basically builds up and eventually flip over to the other side. If you look at a voltage graph the line would increase and cross over to the other side where it is interpreted as a 1.
Click to expand...

Is such a state unrecoverable?
Is this something that happens regularly, or once in a blue moon.....

Do designers develop in mind for such cases?
 
So does anyone have any info regarding backups? Must we sit around waiting for a miracle. Or do we start rebuilding our systems?
 
Last edited:
I'm waiting for a miracle while I try to rebuild from thin air (at another provider). Since they're not answering my calls, I stopped my payment, so I'll wait for them to call ME.
 
Zapo said:
So does anyone have any info regarding backups? Must we sit around waiting for a miracle. Or do we start rebuilding our systems?
Click to expand...

You can start rebuilding...

"Re-provisioning" does not include your backups and the message we got from support sounds as if there will be no restoration of data.
 
Sapphiron said:
Sounds to me like they had a URE crash on their SAN. It happens to RAID 5 and 6 Arrays. Never use RAID 5 or 6 for data you cant afford to lose. It mostly happens during array rebuilds, but has been know to occur when you have 1 dodgy drive in an array.

It would explain the lengh of the outage and the resulting data loss. They spent dozens of hours trying to get data back, after which they gave up and setup the array from scratch.

So many VM "redundant" infrastructures rely on a single SAN device for storage back-end. Contrary to providing "redundancy" they actually amplify their risk of a failure and when it does go, it goes big-time.

This is what happens when you design hosted infrastructures according to the vendor's brochure and not with real world experience.
Click to expand...

I'm not sure why a URE would require re-IP of virtual machines or complete decom of the SAN. I think more than this happened here.
 
Vengeful employee maybe? I saw an advert that someone posted recently, where IS were looking for a new Engineer - maybe the old one got screwed so screwed IS back?
 
eg2505 said:
Is such a state unrecoverable?
Is this something that happens regularly, or once in a blue moon.....

Do designers develop in mind for such cases?
Click to expand...

I think a more suitable response would be "Is this something you can eat?" :P

But seriously I'm talking electronics, your questions are way to high level. The other guy may have answers... but thanks for the response.
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X