Vumatel | How Load-Shedding may affect your fibre connection

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,470
Some or all Vumatel Trenched (grey box on your boundary wall and a Raycore CPE) clients across all ISPs may have experienced long delays in re-establishing connectivity after load shedding - this is not isolated and is affecting clients throughout South Africa. We received reports that some clients took up to 5 hours to come online after yesterday's Stage 4 load shedding.

Unfortunately load shedding is here to stay, so we thought it best to post an update in this regard. Vumatel sent out this memo to clients today:

HOW LOAD-SHEDDING MAY AFFECT YOUR FIBRE CONNECTION

07c10432-f2c0-41a1-a76f-b31ac52a4227.jpg

Vumatel's network is backed up by generators and UPS hardware so if you have backup power* at home, you will be able to surf the internet during load-shedding.

Load-shedding is currently occurring on a mass scale across many suburbs at once. This means that at any given time, thousands of Vumatel customers (without back up power) are going down due to a loss of power to their CPE (white fibre Raycore device mounted to their wall). When the power is restored, thousands of CPEs try to restore their connection with the network at once. This can cause a delay for some customers in the restoration of services immediately after load-shedding. It is similar to thousands of people suddenly trying to enter a stadium at the same time and only so many people can fit through the gates at once. If you find that your internet service does not come back up straight after load-shedding, please be patient. The CPE is trying to authenticate a connection, along with all the others that have just come on-line. If after 2 hours, the service has still not been restored, please contact your ISP for assistance.

*Backup Power Requirements for the CPE & Wi-Fi router:

To keep your internet connection up during the current Stage 2 load-shedding (up to 4 hours), by installing a 1000Va UPS, you will be able to keep the following devices powered up during this period:
1. Vumatel CPE (white Raycore device)
2. ISP Wi-Fi Router
3. One Mobile device based power supply (1 - 5 amp) which will typically charge smartphones and tablets during this period

Please note: this is an indicative calculation and will vary based on the different Wi-Fi routers and power supplies of mobile devices connected to the UPS during this period.

To keep a television powered during this the recommended size of the UPS would be no less than 3000Va.

Kind regards
The VUMA team

Now if you're interested in a technical explanation rather than the stadium analogy:

Vumatel's trenched network utilises DHCP to dynamically allocate IPs to each and every individual router connecting to the network. Each CPE's job is to hand this IP to your router. DHCP is quite a simple protocol. Each client sends a broadcast query, to see if a DHCP server is available and in turn a short negotiation leads to an IP address allocation. Simple. Now when tens of thousands of devices come online, DHCP requests begin being broadcast from all the devices. The problem is, every device on the network responds to a broadcast with a broadcast of its own saying there is or isn't a DHCP server here. I'm sure you can quickly figure out what happens next. The broadcasts flood the network with huge volumes of DHCP requests and responses, this creates a huge amount of wasteful packets, "clogging" the network (hence the packet losses) and wasting a lot of time for actual DHCP responses from the server.

I'm sure that the bright minds at Vumatel are working feverishly to find a solution to the above storms. In the meanwhile, our best recommendation is to connect your CPE and router to a UPS and avoid re-requesting a DHCP IP allocation as best as possible, IE, keeping everything on throughout.
 

daveza

Honorary Master
Joined
Apr 5, 2004
Messages
45,178
So the small print - unless you have a ups there will be a two hour reconnection time after every power outage.

And this will happen with every fiber provider, I don't buy it.

If this was the US, the vultures would already be circling - here we just put up.
 

EADC

Executive Member
Joined
Apr 10, 2018
Messages
7,684
The mentality Vuma provide feedback sorry you talking **** I don't buy it.

Vuma doesn't provide feedback why is Vuma so quiet cry cry

Never happy
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,470
So the small print - unless you have a ups there will be a two hour reconnection time after every power outage.

And this will happen with every fiber provider, I don't buy it.

If this was the US, the vultures would already be circling - here we just put up.

Not every fibre provider. 1 fibre provider (FNO), Vumatel on one half of their network, the active Ethernet portion - their GPON network is unaffected because the ISPs networks handle the PPPoE requests and I don't think any ISP networks struggle with that. We're just helping you understand the issue at hand, it's a DHCP network. DHCP doesn't work well when tens of thousands of devices request at once. We're told Vumatel are working on a remedy.
 
  • Like
Reactions: Yuu

daveza

Honorary Master
Joined
Apr 5, 2004
Messages
45,178
We're told Vumatel are working on a remedy.

Which is doing whatever the other fiber providers who don't have a two hour reconnection time are doing ?
 

3WA

Honorary Master
Joined
Sep 25, 2012
Messages
13,165
So the small print - unless you have a ups there will be a two hour reconnection time after every power outage.

Sounds like a pre-emptive notification to me. My whole building is on Vuma and haven’t heard of anyone getting delayed reconnections.
 

wrathdelivery

Active Member
Joined
Aug 7, 2006
Messages
51
So why was mine up and down for 4pm thru to 10am then? Their "explanation" sounds like BS to me. I got an IP at some point, but then "lost" it again with no power interruption. Meaning their UPS suggestion wouldn't have helped me at all.
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,470
So why was mine up and down for 4pm thru to 10am then? Their "explanation" sounds like BS to me. I got an IP at some point, but then "lost" it again with no power interruption. Meaning their UPS suggestion wouldn't have helped me at all.

We think it was the broadcast storm wreaking havoc on the network; though we are waiting for more feedback and an incident report (we may be left waiting and waiting).
 

MightyQuin

Not amused...
Joined
Oct 6, 2010
Messages
25,647
I have put mine on my inverter system for today, as I will be out and won't need it for my TV etc.

We shall see.....
 

irBosOtter

Expert Member
Joined
Feb 14, 2014
Messages
2,802
Bullcrap, mine got a IP immediately, after every 50 reboots I did yesterday, but could not even ping the gateway, DHCP storm se gat.
 

tibbydudeza

Active Member
Joined
Feb 12, 2019
Messages
51
So why was mine up and down for 4pm thru to 10am then? Their "explanation" sounds like BS to me. I got an IP at some point, but then "lost" it again with no power interruption. Meaning their UPS suggestion wouldn't have helped me at all.

It sounds like BS to me … suspect routers had to be rebooted (hence why we some folks lost IP connection) because they froze up.

Think of it as a massive DDOS attack.
Clearly using DHCP for the CPE devices instead of a static IP was a bad decision.

Also it does not help if you are on a UPS like me and they restart their routers and you join the flood of DHCP requests like the rest.
 
Last edited:

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,470
It sounds like BS to me … suspect routers had to be rebooted (hence why we some folks lost IP connection) because they froze up.

Think of it as a massive DDOS attack.
Clearly using DHCP for the CPE devices instead of a static IP was a bad decision.

Also it does not help if you are on a UPS like me and they restart their routers and you join the flood of DHCP requests like the rest.

We're still trying to piece together what happened as a concerned ISP, and we're not getting much feedback from the FNO for the time being.

We saw some clients stay up in various areas (we're assuming these clients had UPSs) and those clients didn't go down after. Then we saw clients who stayed up through the load shedding, but went down after. We also saw clients get IPs after some time, but see huge packet loss (making their lines unusable). From this, we've deduced that the broadcast flood led to huge packet volumes across the network, making the network unusable while broadcast continued. At this point, a few clients have told us they rebooted (leading to joining the DHCP queue), making matters worse.

Nevertheless, we agree that waiting it out isn't a comforting solution, and for now, the UPS is a suggestion to alleviate the number of DHCP requests after an outage. While this problem only affects Vumatel clients on the trenched network, the effect is felt more in Cape Town because a larger proportion of the network is trenched than GPON.

A second broadcast flood happened this morning. Vumatel has sent a notice out saying they have implemented a measure to reduce the effect of these floods. We'll need to wait until the next load shedding to see if it worked.
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,470
Bullcrap, mine got a IP immediately, after every 50 reboots I did yesterday, but could not even ping the gateway, DHCP storm se gat.

DHCP storm is a broadcast storm; so nothing gets through the noise. We saw latencies as high as 900ms between clients and the Core Nodes (up from 1ms).
 

daveza

Honorary Master
Joined
Apr 5, 2004
Messages
45,178
Why did this only effect Vumatel fiber clients and not other fiber providers ?
 

pinball wizard

Honorary Master
Joined
Feb 9, 2010
Messages
28,897
So the small print - unless you have a ups there will be a two hour reconnection time after every power outage.

And this will happen with every fiber provider, I don't buy it.

.

Doesn't happen on my network.
 

tibbydudeza

Active Member
Joined
Feb 12, 2019
Messages
51
Rather strange there is no headline about this very serious issue on mybroadband.co.za leader page.
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,470
Why did this only effect Vumatel fiber clients and not other fiber providers ?

As I explained, this only affected Vumatel clients on the trenched DHCP network for all ISPs, not the GPON network. Vumatel is the only Fibre Operator that operates a DHCP network (@pinball wizard mentioned TT Connect as well, though we can't comment as we don't deliver services on their network).
 
Top