Internet Solutions virtual machine platform outage for former Mweb Business clients

QuintonB

BusinessTech
Moderator
Joined
Sep 26, 2011
Messages
1,786
Internet Solutions virtual machine platform hosting outage

Internet Solutions’ Johannesburg data center appears to be suffering extended downtime, according to former Mweb Business clients.

Update: Mweb Business has confirmed that its virtual machine platform is suffering an outage, and that the IS data centre is not experiencing any downtime.
 
Last edited by a moderator:

anthor

Active Member
Joined
Jul 6, 2010
Messages
72
I also used to host with them. I moved about 2 years ago and I'm so glad I don't have to deal with this situation and all that comes with it.

It always felt like they weren't too bugged by a few minutes / hours downtime every few days.

I sincerely hope for their sake and for the sake of their clients that they manage to get everything sorted out without data loss. It's never a fun position to be in for either party.
 

Jan

Drifting in the black
Joined
May 24, 2010
Messages
4,645
Article updated to reflect that Mweb Business clients were acquired by IS.
 

deadlikeme

Active Member
Joined
Mar 18, 2009
Messages
39
They lost all the data. They haven't been able to restore the VMs and everyones IP addresses changed. What a nightmare. I suspect they will lose customers over this one.
 

deweyzeph

Executive Member
Joined
Apr 17, 2009
Messages
7,088
I wouldn't trust IS for anything mission critical. I've got a few VM's with ISIgnite because they're cheap and cheerful, but for anything mission critical I have proper dedicated servers with Hetzner.
 

Sapphiron

Centadel
Company Rep
Joined
Jan 29, 2004
Messages
2,011
Sounds to me like they had a URE error crash on their SAN. It happens to RAID 5 and 6 Arrays. Never use RAID 5 or 6 for data you cant afford to lose. It mostly happens during array rebuilds, but has been know to occur when you have 1 dodgy drive in an array.

It would explain the lengh of the outage and the resulting data loss. They spent dozens of hours trying to get data back, after which they gave up and setup the array from scratch.

So many VM "redundant" infrastructures rely on a single SAN device for storage back-end. Contrary to providing "redundancy" they actually amplify their risk of a failure and when it does go, it goes big-time.

This is what happens when you design hosted infrastructures according to the vendor's brochure and not with real world experience.
 

ignitions

Well-Known Member
Joined
Mar 2, 2015
Messages
154
Sounds to me like they had a URE error crash on their SAN. It happens to RAID 5 and 6 Arrays. Never use RAID 5 or 6 for data you cant afford to lose. It mostly happens during array rebuilds, but has been know to occur when you have 1 dodgy drive in an array.

It would explain the lengh of the outage and the resulting data loss. They spent dozens of hours trying to get data back, after which they gave up and setup the array from scratch.

So many VM "redundant" infrastructures rely on a single SAN device for storage back-end. Contrary to providing "redundancy" they actually amplify their risk of a failure and when it does go, it goes big-time.

This is what happens when you design hosted infrastructures according to the vendor's brochure and not with real world experience.
So these people operate without a DMZ or some kind of clustered storage setup?
 

Taqyon

Well-Known Member
Joined
Dec 10, 2003
Messages
199
So these people operate without a DMZ or some kind of clustered storage setup?
Well it's a cluster *something* I can tell you.. many hundreds of companies' services down as a result. We had our backups on the same infrastructure.

@Sapphiron are you certain "they gave up" ?
 

Ipwn 4

Expert Member
Joined
Nov 6, 2010
Messages
1,680
lol, I love the fck you attitude.
Just restore your data from backups... No backups? eish....
 

kianm

Honorary Master
Joined
Jan 13, 2014
Messages
10,433
This sounds really really bad. Such a fail from such big companies. Maybe i should move the little that i have with mweb elsewhere
 
Top