MTN says data network fault developed after upgrade

[noxibox]

Would be interesting to have more information. Why did the upgrade fail? Why was an untested upgrade pushed into the network? If it was tested why did the testing not pick up the problem?

Because testing isn't perfect. Google screw up all the time. So do Microsoft and Apple. So this could be like the scenario where a faulty product gets by quality control in spite of testing, resulting in a naïve customer insinuating that it means the factory does no checking. Perhaps they don't or perhaps it is just the rare case that slips by.

 

[r00igev@@r]

Because testing isn't perfect. Google screw up all the time. So do Microsoft and Apple. So this could be like the scenario where a faulty product gets by quality control in spite of testing, resulting in a naïve customer insinuating that it means the factory does no checking. Perhaps they don't or perhaps it is just the rare case that slips by.

There statement is half baked and not good enough. Its generic and leaves open ended assumptions in play like what you have made.

Their statement does not mention any testing being completed before the upgrade so it cannot be assumed. In actual fact it MUST be concluded that there was none, unless stated. Don't give a hoot if its MTN. Vodacom, Linus Tovalds, Google, apple or myself.

Same as the banks. Blame everything on a "glitch". These guys all try and pull the wool over naive customers that they in fact have done their due diligence.

Having worked in the environment I can make some assumption as well based on experience. What often happens is that some larney said, I don't care about testing or excuses. I will fire your ass if you don't do it now. The guy at the bottom is intimated to press the button after which everything is then professionally swept under the carpet when it goes TITSUP.

PS: I'll take everything back if MTN can provide a reference for the automated testing and simulation software they use for network changes and implementations as well as the headcount for that department.

 

[PaulMurkin]

Have you ever upgraded systems of this size?

Yes

 

[PaulMurkin]

Cutting corners to save money

Its that time of the year... CEO is eyeing a new Maybach... gotta cut corners to make extra dosh because the call drop trick is no longer effective

 

[Jet-Fighter7700]

typical tech mentality, upgrade now, think later,
Change control nonexistent, roll backs non existent and just hurry up and fix it mentality.

 

[Conack]

Supersonic (on MTN) is still offline in several Garden Route towns it seems. Have seen multiple reports of continued outages all over - and currently still ongoing.

 

[SubtleBeast]

Poorly planned upgrade to national infrastructure. Should've staggered the upgrades. Where were the fail safes and redundancies? Upgrades probably rushed due to the December freeze approaching.

 

[PaulMurkin]

typical tech mentality, upgrade now, think later,
Change control nonexistent, roll backs non existent and just hurry up and fix it mentality.

Just like at the previous company I worked for. One of the major reasons why I left.

 

[PaulMurkin]

Poorly planned upgrade to national infrastructure. Should've staggered the upgrades. Where were the fail safes and redundancies? Upgrades probably rushed due to the December freeze approaching.

Of course ja... f**ked around all year then suddenly skrikked wakker in November... because too busy leafing thru the Mercedes catalogues or just plain loafing.

 

[Jet-Fighter7700]

Just like at the previous company I worked for. One of the major reasons why I left.

agree with you, but its a worldwide problem as @r00igev@@r mentioned,
when project managers and other tech managers barely understand what techies do,

they make mistakes and dont do enough testing and dont do enough due diligence.
and just put pressure on things they barely understand.

 

[PaulMurkin]

agree with you, but its a worldwide problem as @r00igev@@r mentioned,
when project managers and other tech managers barely understand what techies do,

they make mistakes and dont do enough testing and dont do enough due diligence.
and just put pressure on things they barely understand.

In other words, the incompetents lording over the skilled people.

 

[Jet-Fighter7700]

In other words, the incompetents lording over the skilled people.

its the same everywhere, even outside SA, just wait and see.
its a cancer of the tech environment arguably.

 

[r00igev@@r]

its the same everywhere, even outside SA, just wait and see.
its a cancer of the tech environment arguably.

Take that back! VLAN Musk is a great coder.

Edit: Fire the buggers that don't listen. Damn woke left wingers.

 

[PaulMurkin]

its the same everywhere, even outside SA, just wait and see.
its a cancer of the tech environment arguably.

And like cancer it varies by degrees... I know its here too, I just chose the non-malignant type, the one that grows for 20 years, just long enough for me to get to retirement.

 

[SilverCode]

Would be interesting to have more information. Why did the upgrade fail? Why was an untested upgrade pushed into the network? If it was tested why did the testing not pick up the problem?

Testing in a lab will never catch all potential issues. For something like network infrastructure, you are never going to have the amount of devices/traffic in your lab setup that you have in the real world.

It is easy to test something in a lab and have it pass with say 1000 real or simulated devices. But put that out in the real world and suddenly something like an innocent routing rule or change to a cache size is overwhelmed by the sheer amount of traffic.

You don't notice a problem when pushing 10 GB/s through your equipment, but you do when that becomes 1 TB/s

 

[r00igev@@r]

Testing in a lab will never catch all potential issues. For something like network infrastructure, you are never going to have the amount of devices/traffic in your lab setup that you have in the real world.

It is easy to test something in a lab and have it pass with say 1000 real or simulated devices. But put that out in the real world and suddenly something like an innocent routing rule or change to a cache size is overwhelmed by the sheer amount of traffic.

You don't notice a problem when pushing 10 GB/s through your equipment, but you do when that becomes 1 TB/s

That does not negate the reason for testing to see that the process works and a company with the resources and saddlebags of moola like MTN can afford simulation and load testing software.

Unless you yourself work for MTN and decided that you not going to test because testing doesn't pick up all potential issues so we'll sommer crash in live. Would explain the TITSUP.

Edit: What innocent thing broke and why didn't it happen with previous upgrades?