Sage hit by major outage in South Africa

quovadis

Executive Member
Joined
Sep 10, 2004
Messages
5,496
Do you by any chance maintain in-house infra for a company?
I have customers who leverage many options. In saying that I also ensure that my customers are aware of the benefits and risks associated and I ensure that they are mitigated or advantaged as best as possible by their decisions.
 

ToxicBunny

Oi! Leave me out of this...
Joined
Apr 8, 2006
Messages
94,262
Do you by any chance maintain in-house infra for a company?

Many companies do still manage to host their own stacks.

BUT, it is becoming increasingly difficult to do, with the offerings from the providers all moving towards SaaS and PaaS type solutions.

In all honesty a 6 day outage on your headline product is totally unacceptable really, and it will probably mean a portion of their user base will move back to an onprem hosted Sage, or migrate to an entirely different product.
 

Johand

Expert Member
Joined
Jan 21, 2005
Messages
1,613
Wowser. 6 days are insane. When I heard outage I thought a couple of hours.

What did they do to manage 6 days worth it outage? Did they just autoinstall on all servers simultaneously with no rollback plan? 6 days are crazy even for consumers but talking about a core, paid for business application? It is insanity.

I would drop sage like a hit potato. Because 1 day is a technical problem- 6 days is a management problem. And fixing the technical problem would give you no idea whether the management problem was solved.
 

ToxicBunny

Oi! Leave me out of this...
Joined
Apr 8, 2006
Messages
94,262
Wowser. 6 days are insane. When I heard outage I thought a couple of hours.

What did they do to manage 6 days worth it outage? Did they just autoinstall on all servers simultaneously with no rollback plan? 6 days are crazy even for consumers but talking about a core, paid for business application? It is insanity.

I would drop sage like a hit potato. Because 1 day is a technical problem- 6 days is a management problem. And fixing the technical problem would give you no idea whether the management problem was solved.

To me 6 days sounds like a fundamental data corruption problem that they are scrambling to try and rectify without having to notify their client base.
 

Mystic Twilight

Expert Member
Joined
Dec 23, 2010
Messages
2,749
To me 6 days sounds like a fundamental data corruption problem that they are scrambling to try and rectify without having to notify their client base.

I'm guessing it's not just data corruption, they probably also did a one way update/upgrade to the architecture with either no rollback contingency or the rollback itself doesn't work so they are completely stuck and have to rework the various systems by hand.

And it seems they outsourced their underlying infrastructure, so there is probably red tape and permissions issues between sage, any outsourced/contracted software devs, and their server providers.
 

quovadis

Executive Member
Joined
Sep 10, 2004
Messages
5,496
I'm guessing it's not just data corruption, they probably also did a one way update/upgrade to the architecture with either no rollback contingency or the rollback itself doesn't work so they are completely stuck and have to rework the various systems by hand.

And it seems they outsourced their underlying infrastructure, so there is probably red tape and permissions issues between sage, any outsourced/contracted software devs, and their server providers.
Would be interesting to know if Sage One allows you to export your data for safekeeping/backup purposes?
 

Herr der Verboten

Executive Member
Joined
Feb 14, 2012
Messages
8,474
I hope they sort it out ASAP. My company uses them.
I wonder if this is going to influence any payment instructions tied to your salary. Like medical... Salary payment instructions is going to be fun on recon if they don't have this sorted before 1st feb/end of jan.
 

ToxicBunny

Oi! Leave me out of this...
Joined
Apr 8, 2006
Messages
94,262
I'm guessing it's not just data corruption, they probably also did a one way update/upgrade to the architecture with either no rollback contingency or the rollback itself doesn't work so they are completely stuck and have to rework the various systems by hand.

And it seems they outsourced their underlying infrastructure, so there is probably red tape and permissions issues between sage, any outsourced/contracted software devs, and their server providers.

Well yeah, probably a perfect storm really.

Have experienced a few of those in my life... though one way updates/upgrades to the architecture always involved building out a new setup of the infrastructure for the new architecture and migrating over rather than upgrading the existing (then at least you have a break glass rollback plan hopefully)
 
Top