Internet Solutions virtual machine platform outage for former Mweb Business clients

QuintonB

BusinessTech
Moderator
Joined
Sep 26, 2011
Messages
1,786
#1
Internet Solutions virtual machine platform hosting outage

Internet Solutions’ Johannesburg data center appears to be suffering extended downtime, according to former Mweb Business clients.

Update: Mweb Business has confirmed that its virtual machine platform is suffering an outage, and that the IS data centre is not experiencing any downtime.
 
Last edited by a moderator:

anthor

Active Member
Joined
Jul 6, 2010
Messages
66
#3
I also used to host with them. I moved about 2 years ago and I'm so glad I don't have to deal with this situation and all that comes with it.

It always felt like they weren't too bugged by a few minutes / hours downtime every few days.

I sincerely hope for their sake and for the sake of their clients that they manage to get everything sorted out without data loss. It's never a fun position to be in for either party.
 

Jan

Drifting in the black
Joined
May 24, 2010
Messages
4,604
#5
Article updated to reflect that Mweb Business clients were acquired by IS.
 

deadlikeme

Active Member
Joined
Mar 18, 2009
Messages
39
#6
They lost all the data. They haven't been able to restore the VMs and everyones IP addresses changed. What a nightmare. I suspect they will lose customers over this one.
 

deweyzeph

Executive Member
Joined
Apr 17, 2009
Messages
6,827
#8
I wouldn't trust IS for anything mission critical. I've got a few VM's with ISIgnite because they're cheap and cheerful, but for anything mission critical I have proper dedicated servers with Hetzner.
 

Sapphiron

Centadel
Company Rep
Joined
Jan 29, 2004
Messages
1,999
#9
Sounds to me like they had a URE error crash on their SAN. It happens to RAID 5 and 6 Arrays. Never use RAID 5 or 6 for data you cant afford to lose. It mostly happens during array rebuilds, but has been know to occur when you have 1 dodgy drive in an array.

It would explain the lengh of the outage and the resulting data loss. They spent dozens of hours trying to get data back, after which they gave up and setup the array from scratch.

So many VM "redundant" infrastructures rely on a single SAN device for storage back-end. Contrary to providing "redundancy" they actually amplify their risk of a failure and when it does go, it goes big-time.

This is what happens when you design hosted infrastructures according to the vendor's brochure and not with real world experience.
 

ignitions

Well-Known Member
Joined
Mar 2, 2015
Messages
154
#10
Sounds to me like they had a URE error crash on their SAN. It happens to RAID 5 and 6 Arrays. Never use RAID 5 or 6 for data you cant afford to lose. It mostly happens during array rebuilds, but has been know to occur when you have 1 dodgy drive in an array.

It would explain the lengh of the outage and the resulting data loss. They spent dozens of hours trying to get data back, after which they gave up and setup the array from scratch.

So many VM "redundant" infrastructures rely on a single SAN device for storage back-end. Contrary to providing "redundancy" they actually amplify their risk of a failure and when it does go, it goes big-time.

This is what happens when you design hosted infrastructures according to the vendor's brochure and not with real world experience.
So these people operate without a DMZ or some kind of clustered storage setup?
 

Taqyon

Well-Known Member
Joined
Dec 10, 2003
Messages
199
#11
So these people operate without a DMZ or some kind of clustered storage setup?
Well it's a cluster *something* I can tell you.. many hundreds of companies' services down as a result. We had our backups on the same infrastructure.

@Sapphiron are you certain "they gave up" ?
 

kianm

Honorary Master
Joined
Jan 13, 2014
Messages
10,395
#13
This sounds really really bad. Such a fail from such big companies. Maybe i should move the little that i have with mweb elsewhere
 
Top