Web sQuad ISP - Feedback Thread #2

websquadza said:
Seems to be resolving itself, seeing a number of peers on the ZANOG channel saying they're still seeing some issues.

Every network in SA (and pretty much south of the equator) peers at NAP JHB, or has an upstream who does, plus this is where all the big cloud vendors hand over traffic, so very bad day when NAP has a wobbly like this.

We're also on INX-ZA's (JINX, DINX, CINX) fabrics, but we need more peers there to make sure that traffic jumps smoothly. Also bear in mind that the standard timeout on BGP sessions is 180s (without BFD), so that means even with multiple fabrics, it can take up to 3 minutes to drop a bad prefix route and install a good one.

NAP / INX are fabrics, where peers set up direct connections (BLP) to one another to exchange routing information, or use the route servers (RS/MLP) to exchange routes with one another. Today's outage resulted in RS1 and 2 going offline simultaneously, and at least for us, multiple BLP peers dropping for almost 10 minutes. It seems there was something on the fabric itself which caused traffic to stop flowing, but the great team at NAP Africa will most certainly do a deep analysis and confirm what happened today.
Click to expand...
Nice - thanks for the insight
 
thanks for the updates. noticed some "slowness/timeouts" . Checked this forum, glad it's been monitored.
 
my internet has pooped the bed
is there anything going on with evotel? @websquadza
Screenshot 2025-05-22 182654.png

edit: my router is cooked or something else, i'm going crazy :/
 
Last edited:
websquadza said:
Howdy, looks like a last mile issue to me. You still seeing the same?
Click to expand...
please take a look at my ticket because it seems like casper the ghost is affecting every router i try and test with.
kinda going crazy
Ticket #907872
 
Ugh, trying to get some work finished , and apparently we down , this generally doesn’t last too long, but is becoming a headache
 
BouncyNinja said:
Ugh, trying to get some work finished , and apparently we down , this generally doesn’t last too long, but is becoming a headache
Click to expand...
I also had a bit of an outage on Openserve about 15mins ago - rebooted the ONT and that seemed to resolve it
 
BouncyNinja said:
Ugh, trying to get some work finished , and apparently we down , this generally doesn’t last too long, but is becoming a headache
Click to expand...
You still seeing your service down? We're not aware of any large scale Vuma outages currently. Also checked Vuma events board for anything for your area, nothing yet.
 
Hey @websquadza ,
Please can you chat to your team about the internet quality in our estate (perhaps check the calls logged), we are getting random disconnections and packetloss frequently (during working hours, teams meetings etc), we have logged numerous calls and even though I largely gave up on the issue, this is my last resort before seeking a new ISP. last night at 9:35 till 10:00pm was another bad one

Tickets logged in the past:
#865568
#859964
#262078

I am giving up because I have supplied all the MTR results etc etc, and no resolution in sight yet.

Last night example:
1747983607627.png
 
websquadza said:
You still seeing your service down? We're not aware of any large scale Vuma outages currently. Also checked Vuma events board for anything for your area, nothing yet.
Click to expand...
apologies for only coming back now, it was back on again about 10 minutes later
 
Gozzy said:
Hey @websquadza ,
Please can you chat to your team about the internet quality in our estate (perhaps check the calls logged), we are getting random disconnections and packetloss frequently (during working hours, teams meetings etc), we have logged numerous calls and even though I largely gave up on the issue, this is my last resort before seeking a new ISP. last night at 9:35 till 10:00pm was another bad one

Tickets logged in the past:
#865568
#859964
#262078

I am giving up because I have supplied all the MTR results etc etc, and no resolution in sight yet.

Last night example:
View attachment 1822478
Click to expand...
Hi There

Thanks for your patience and continued support, and for reaching out and bringing this to my attention.

Could you please add one more IP to your monitoring, the gateway IP you're seeing on your PPPOE (you can pick this up on your router), as this will help us see if your connection is routable your home and our network handover. While I see your PPPOE is staying up, I'd like to see if traffic is flowing. PPPOE is usually quite sensitive, so if L2 is down, PPPOE drops, but I want to see if there isn't something causing issues with the actual flow of PPPOE packets. We have noted that an NLD flapped last night, however this would not have affected you.

Looking at your tickets below, there have been varied issues and I see you've had some headaches here. But we're certainly keen to investigate these, and ensure that these are resolved permanently.

I have checked on these tickets, and I see the following.

#865568 - This one was us. We experienced a fault on our BNG which you connect to, a memory exhaustion error, and routes didn't clear out of the routing table quickly enough (or at all). We have made adjustments to our devices to ensure this behaviour is prevented in future.

#859964 - FNO (Comtel Connect) upgraded a device at their POP without any prior notification, affecting all the services linked to POP. Unfortunately this was done during production traffic, which was not ideal. I note you mentioned on the ticket that this only affected Web sQuad off this POP. This is odd, as the FNO confirmed all services were affected, and the RFO was shared with you.

#262078 - Part of a larger issue that stemmed from a faulty configuration issue. Web sQuad had a legacy VLAN at the POP, and new VLAN configs introduced during 2024 by the FNO. While you were on the newer VLAN, 3 clients connecting to this POP were on the legacy VLAN and the FNO had bridged both VLANs towards us as a single service. The OLTs at the POP saw these duplicate MACs as potential loops and would randomly drop traffic it incorrectly thought was broadcast. This issue could only affect Web sQuad due to the VLANs being Web sQuad specific and caused major headaches for our clients and us while we worked with the FNO to resolve the issue. Unfortunately as per our RFO to you, we don't get to see what the FNOs do on their end and this particular one caused a lot of distress.

We're not an ISP to point fingers, and will always acknowledge if a fault lies with us, and hound our FNOs if the issue is on their side, until issues are resolved.

I'm also asking the team to review all your tickets and correlate them to any other network incidents we may have experienced.

Please PM me any other issues you may have and I'll be happy to investigate further.
 
anything wonky ? my twitch and webmail not working.

yet youtube seems fine.
 
You must log in or register to reply here.
Back
Top
Sign up to the MyBroadband newsletter
X