MWEB Datacentre - !@#$ lemme prevent that

[Polemus]

how many peeps out there just signed up for off-site backup solutions, and is frantically busy making VPS snapshots

i am not a MWEB client, but i sure as hell almost had heart failure just thinking what would have happened if i was.

..... transferring backup 16% .....

 

[Ipwn 4]

/Checks Nagios, backups all green.
/checks latest DR simulation report, cutover completed in 20min.
/lols at everyone else

If the same were to happen to me I would probably just never come back to work...

 

[crudler]

I am hosting with IS. I am happy with them, but they are not infallible.
I think as a lesson learn't I should make sure that my backups are stored at a completely different data centre.
Any recommendation that is non IS ? I just need storage and ftp access

edit: preferably some crowd that I can sign up online and put in credit card info. nothing more frustrating than having to wait days for forms, and meet with an architect etc etc (*cough*IS*cough*)

 

[Progenix Oj101]

Does anybody actually know what happened at Mweb?

 

[Necropolis]

Does anybody actually know what happened at Mweb?

Things broke - people lost data

But seriously I haven't seen a straight forward answer as to what happened - methinks one of the engineers stuffed something up, royally - causing much groaning and gnashing of teeth.

 

[Drifter]

Here you go:

MWeb Business outage cause: 'Someone pushed the wrong button'

Details are emerging of the provisioning error which caused MWeb Business to erase its customers' virtual machines (VMs). Customers who have interacted with technical staff at MWeb report that a provisioning tool was allegedly updated, with an error in a script causing it to end-date all running VMs, leading to their instant erasure.

http://www.itweb.co.za/index.php?op...cle&id=144811&A=DRE&S=Business&O=E&E=3-165383

 

[Ipwn 4]

Here you go:

MWeb Business outage cause: 'Someone pushed the wrong button'

Details are emerging of the provisioning error which caused MWeb Business to erase its customers' virtual machines (VMs). Customers who have interacted with technical staff at MWeb report that a provisioning tool was allegedly updated, with an error in a script causing it to end-date all running VMs, leading to their instant erasure.

http://www.itweb.co.za/index.php?op...cle&id=144811&A=DRE&S=Business&O=E&E=3-165383

Haha testing changes in production, no one have a dev environment anymore?

 

[herbertk]

Here you go:

MWeb Business outage cause: 'Someone pushed the wrong button'

Details are emerging of the provisioning error which caused MWeb Business to erase its customers' virtual machines (VMs). Customers who have interacted with technical staff at MWeb report that a provisioning tool was allegedly updated, with an error in a script causing it to end-date all running VMs, leading to their instant erasure.

http://www.itweb.co.za/index.php?op...cle&id=144811&A=DRE&S=Business&O=E&E=3-165383

Wow how the heck do you "instantly delete" a VM that is end of life. That is just dumb a 7 day delay would have saved them a lot of issues.

EDIT : Meaning shut the VM down keep it down for a day 7 days a month and then delete it. Just deleting it is just brain dead, perhaps for a free service but not for a paid "Business" service.

 

[The_Unbeliever]

Haha testing changes in production, no one have a dev environment anymore?

Cutting costs... dev environments are too expensive, just push it onto the production server and hope for the best...

 

[Necropolis]

Here you go:

MWeb Business outage cause: 'Someone pushed the wrong button'

Details are emerging of the provisioning error which caused MWeb Business to erase its customers' virtual machines (VMs). Customers who have interacted with technical staff at MWeb report that a provisioning tool was allegedly updated, with an error in a script causing it to end-date all running VMs, leading to their instant erasure.

http://www.itweb.co.za/index.php?op...cle&id=144811&A=DRE&S=Business&O=E&E=3-165383

Oh dear.

The person that did that must be is a world of ****...

 

[Doxol]

This is the correct story, remember IS now owns Mweb Business so to say you are glad you are with IS makes no sense since they will use that same Solid tool to provision VM's, why it didn't affect other VM's is something only hey can answer.

http://www.itweb.co.za/index.php?option=com_content&view=article&id=144838

 

[nivek]

snapshots on the array?

 

[Necuno]

Cutting costs... dev environments are too expensive, just push it onto the production server and hope for the best...

 

[Ipwn 4]

Cutting costs... dev environments are too expensive, just push it onto the production server and hope for the best...

At their size with their big brother IS backing them this is not an excuse, incompetence and negligence is though. This is what happens though when access isn't controlled and no proper change management exists.