MWEB Business clients' massive data loss

[herbertk]

Sound like a provisioning script for their VM's that end dated all the VM's.

http://www.itweb.co.za/index.php?op...cle&id=144811&A=DRE&S=Business&O=E&E=3-165383

 

[km2]

I know this is going to ruffle feathers but I can't feel sorry for anyone who lost data. If you data is so important to you why would you entrust Mweb of all people to keep it safe?

There's a lot of effort being spent selling the benefits of the "cloud future", and providers are misleading people into thinking their data is safe with them, but then burying in the contract disclaimers of all responsibility. It doesn't matter of you choose MWEB, Microsoft or Google, they all contractually can shrug their shoulders of liability if anything happens.

Our host charges us around 50% more then Mweb for virtual machines but this includes daily backups which go to tapes which are then stored offsite and also replication to another DC (Teraco) in 15min intervals.

Have you read the contract from your fancy host? They probably charge you 50% more but still have it clearly spelt out that you can go get stuffed if there is data loss.

That's how you learn though, you cannot ever place all your trust in a single host or datacenter because things like this will happen

But aren't you describing a single host setup yourself? They provide the service and the backups. You're trusting them with both aspects.

 

[Necuno]

Massive MWEB Business data loss

Internet Solutions, which operates MWEB Business, experienced a major problem recently which resulted in data loss for clients.

This can't be. Mweb always works and are never down

 

[quovadis]

The issue, described as a "system provisioning error", is "100% resolved," says Debbie Pretorius, general manager, MWeb Business. "The issue was resolved in stages, part on Friday morning and fully by Sunday afternoon."

Clients are assumed to be fully responsible for their own backups, she notes. "There are no SLA clauses that would activate from this system error; however, as a gesture of goodwill we will not be invoicing clients who have our virtual machine product for July."

http://www.itweb.co.za/index.php?option=com_content&view=article&id=144803

 

[Fulcrum29]

http://www.itweb.co.za/index.php?option=com_content&view=article&id=144803

Which party in the contract is responsible to do system provisioning? I don't see the customer to have their hands in the network. All the large cloud network operators on the globe has an SLA clause in place to minimise the risk in provisioning. It is hard to determine how Mweb is dismissing the responsibility…

to quote,

The bad news for customers is that data is confirmed to be permanently lost. "All clients received new IPaddresses as this was the quickest way to ensure the restoration of the service," Pretorius says. "Customers who do not back up their sites would be affected."

Untrue, customers which do have backups was also exposed to and experienced the risk.

Cop out.

 

[The_Unbeliever]

As for backups - if you were running mission critical systems in that DC and didn't have backups then it is your own fault if you now have big chunks of data missing.

^ that.

I have triplicate of everything in backup... better be safe than sorry! (cryptolocker gave me a wake-up call).

Wonder what will be the way forward for some companies... most of them will have to shut shop... and most will be harassed by SARS and they will be unable to prove anything due to missing data.... >.<

 

[Waste]

While I am in no way saying that what Mweb did is alright, customers should ensure that they make backups of there data regardless whether the service provider does it or not. To not ensure your backups are done, and just trust that nothing will go wrong is pretty foolish.

Give me one good reason to backup your statement???..... oh wait....

 

[Waste]

How to sink a business in four days....

+1

 

[km2]

Which party in the contract is responsible to do system provisioning? I don't see the customer to have their hands in the network. All the large cloud network operators on the globe has an SLA clause in place to minimise the risk in provisioning. It is hard to determine how Mweb is dismissing the responsibility…

Those SLA clauses for Google services or Microsoft Azure pretty much just translate into money back (maybe for up to 12 months of paid service). Nothing to compensate for any business problems as a result of data loss.

 

[Waste]

There's a lot of effort being spent selling the benefits of the "cloud future", and providers are misleading people into thinking their data is safe with them, but then burying in the contract disclaimers of all responsibility. It doesn't matter of you choose MWEB, Microsoft or Google, they all contractually can shrug their shoulders of liability if anything happens.



Have you read the contract from your fancy host? They probably charge you 50% more but still have it clearly spelt out that you can go get stuffed if there is data loss.



But aren't you describing a single host setup yourself? They provide the service and the backups. You're trusting them with both aspects.

+1 Agreed!

The statement "You can never be to safe!" is true. But we all have to decide how far we want to go.

I'm not expecting them to restore your database when you accidently mass delete records, or restore your files if you accidently override them. But I do not think it is unreasonable to expect them to at least be able to sustain their service. If they could restore a copy of the VPS that is a month old, a lot of clients would have jumped for joy....

If you have the time and budget to build backups on top of their backups (or not), why don't you just get a hosted dedicated server in 2 or 3 datacentres and roll your own backup strategy from the hardware up? This way you can physically respawn your service in minutes from mirrors or replication sites. And and added Bonus you get 3Tb of storage and not just 50Gb.

Because of this disaster even you with all the backups would have had no service for 2 to 3 days.

There is nothing wrong to build on top of what they provide (even if it is bare). But you have so many ways to play this game. Pros and Cons are all over the place.

 

[Waste]

Pictorial Interpretation of The Industry Cloud vs. Mweb Business Cloud

The Industry Cloud:


Image from http://betanews.com/

Mweb Business Cloud:


Image from http://www.jtechdata.net/

 

[mpdjhb]

Pointing at clients and saying "You should do backups" might be true on the surface. But even hourly backups would mean up to 59 minutes loss.
Even on-line replication will mean a record or two is lost.

And even with a cluster, you might still lose business due to down time.

 

[The_Unbeliever]

Pointing at clients and saying "You should do backups" might be true on the surface. But even hourly backups would mean up to 59 minutes loss.
Even on-line replication will mean a record or two is lost.

And even with a cluster, you might still lose business due to down time.

But still an hour/day/week's loss of data may be preferred above losing ALL data?

No backup solution will back up all your data 100% guaranteed - and if you do want that, expect to pay big $$$ - which most companies cannot afford... so they'll have to stick with daily/weekly backups...

 

[durbandave]

Pointing at clients and saying "You should do backups" might be true on the surface. But even hourly backups would mean up to 59 minutes loss.
Even on-line replication will mean a record or two is lost.

And even with a cluster, you might still lose business due to down time.

I am sure saying to your boss that you lost an hour of data is better than saying you lost all there data.

Depending on your DR strategy they would have an acceptable RPO and MTO.

Losing all your data because you are to stupid to back it up at all is inexcusable. Saying you expected MWEB to do is even more inexcusable.

If you are a serious business you should be planning for the worst scenario. Good governance practices actually mandate proper business continuity planning.

The saying don't put all your eggs in one basket is very apt here.

Backing up data to the same volume is only good when there is normal data loss e.g someone deleted something by mistake. For DR you should have backups stored away from the production environment.

 

[ponder]

This can't be. Mweb always works and are never down

You poking fun at PostmanPot?

 

[MrGray]

Even if you had backups they were down for 3+ days without much clarity as to what was going on, and over a weekend as well. That's just not good enough. Most smaller businesses will not have hugely sophisticated DR plans in place apart from (hopefully) their daily backups and source, so this means a lot of time restoring everything (preferably to another DC!!).

 

[deweyzeph]

I use Cloudberry (http://www.cloudberry.com) and backup to my Amazon S3 storage account. Works like a charm. Backing up to the same DC as your server is just asking for trouble!

 

[Wasabee!]

How to sink a business in four days....

Capn biznes iz sunked. Shud wi sel de external hdds fr luntch moneh?

 

[FatBoySlim]

SO i run a mobile app business for the financial services industry, MWEB has dropped my IP address. So its all good to say backup the data, but when they have changed the IP address apps cannot find a frikken server.

I hope the monkey (s) are fired whoever did this!! AFRIKA

 

[deweyzeph]

SO i run a mobile app business for the financial services industry, MWEB has dropped my IP address. So its all good to say backup the data, but when they have changed the IP address apps cannot find a frikken server.

I hope the monkey (s) are fired whoever did this!! AFRIKA

Why are you using an IP address instead of a domain name to reach your server? That's just stupid. If you were using a domain name you could just change the A record in the nameserver to the new IP address and you'd have no problem.