Web Squad ISP

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,167
NAP JHB Update (very preliminary): Looks like a loop was induced across one of the NAP switch fabrics just before 6pm (details are still very scarce, but we do see that most peers dropped off for a while). This dropped all traffic across the affected fabric and in turn most connected networks (+- 900 Gbps if graphs are accurate). Some networks who have presence dual presence at NAP and JINX weren't affected as @JustAnotherSouthAfrican picked up. Google was affected because both their sessions went down. Whatever caused the loop has been removed and all affected peers at NAP JHB are back up. Will provide further updates once we have a clearer picture.
 

joburg_guy

Active Member
Joined
May 13, 2009
Messages
39
Very strange routing behavior just observed on the WS network - please check the routes to Telkom/SAIX, they are currently running via London
 

joburg_guy

Active Member
Joined
May 13, 2009
Messages
39
Very strange routing behavior just observed on the WS network - please check the routes to Telkom/SAIX, they are currently running via London
2 5 ms 4 ms 6 ms core.bng-xe-02.jb1.za.wecom.net.za [160.119.230.1] 3 5 ms 5 ms 5 ms core.cr-xe-01.jb1.za.ws.net.za [160.119.224.29] 4 7 ms 6 ms 4 ms core.pe-xe-ip01.jb1.za.ws.net.za [160.119.224.18] 5 6 ms 7 ms 5 ms gi0-0-1-0.agr11.jnb01.atlas.cogentco.com [206.249.0.201] 6 5 ms 5 ms 6 ms be2355.ccr51.jnb01.atlas.cogentco.com [154.54.43.37] 7 167 ms 165 ms 165 ms be2436.ccr21.lon02.atlas.cogentco.com [130.117.0.89] 8 167 ms 167 ms 165 ms be2573.ccr42.lon13.atlas.cogentco.com [154.54.62.5] 9 322 ms 178 ms 177 ms be12488.ccr42.ams03.atlas.cogentco.com [130.117.51.42] 10 180 ms 179 ms 181 ms be2814.ccr42.fra03.atlas.cogentco.com [130.117.0.142] 11 184 ms 184 ms 185 ms be2846.rcr22.fra06.atlas.cogentco.com [154.54.37.30] 12 180 ms 179 ms 181 ms be2844.agr21.fra06.atlas.cogentco.com [130.117.0.30] 13 182 ms 180 ms 180 ms 149.14.211.194 14 * * * Request timed out. 15 185 ms 184 ms 183 ms igubu.saix.net [196.25.1.1]
 

PsyWulf

Honorary Master
Joined
Nov 22, 2006
Messages
11,323
Very strange routing behavior just observed on the WS network - please check the routes to Telkom/SAIX, they are currently running via London
Wasn't this related to the Paid peering thing a while ago? IE if you don't pay you gotta route internationally *cackles maniacally*
 

kayvee

Expert Member
Joined
Jan 3, 2008
Messages
3,532
Don't you hate it when they close unresolved tickets!

Massive issues with my connection up and down, going on for 4 days. Not happy.
#603008
 

cavedog

Honorary Master
Joined
Oct 19, 2007
Messages
18,984
Wasn't this related to the Paid peering thing a while ago? IE if you don't pay you gotta route internationally *cackles maniacally*

Yeap only Telkom ISP ranges you can pick up locally. Saix and the rest nope they have selected peering. Example telkomsa.net and telkom.co.za. You can only get all of it locally if you have Openserve transit or other major partners like Seacom, IS, Vodacom, MTN ect.
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,167
Don't you hate it when they close unresolved tickets!

Massive issues with my connection up and down, going on for 4 days. Not happy.
#603008

looks to me like you cleared your issue earlier this week which is why that ticket was closed. You can always open a new ticket- we’ll investigate it and make sure you confirm it’s resolved.. This outage seems to be part of an unrelated larger outage which we’re waiting for Vuma to confirm... We’ve responded on your ticket.
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,167
Very strange routing behavior just observed on the WS network - please check the routes to Telkom/SAIX, they are currently running via London

Nothing really strange here. Just SAIX’s (like seacom’s) closed peering policy. Web sQuad remains one of the most peered local networks- both with open peering and over 300 direct peering sessions with other networks in SA. We fight hard to make sure the quickest paths between two networks in the same building (or on the same fabric), is direct. Archaic policies driven by some idea that we’ll turn around and pay them premium for IP transit drive their decisions not to peer openly. This when Tier 1 transit providers are available locally

That said, we do want the best possible routes for our clients and we are working on a 3rd party arrangement to pick these up locally. We’re just waiting for the 3rd party to complete some adjustments to only provide us with these routes and we should be sorted in the next week or so.
Yeap only Telkom ISP ranges you can pick up locally. Saix and the rest nope they have selected peering. Example telkomsa.net and telkom.co.za. You can only get all of it locally if you have Openserve transit or other major partners like Seacom, IS, Vodacom, MTN ect.

Yup. This means that SAIX just seem to be making life tough for their own wholesale clients- not all of whom have their own peering. Much like seacom to their own clients.
 

JustinB

Senior Member
Joined
Aug 6, 2008
Messages
910
Sigh. Don't you hate it when your internet goes down and then you remember you haven't paid for the month yet... Whoops....
 

kayvee

Expert Member
Joined
Jan 3, 2008
Messages
3,532
Thanks websquadza.

Well I cleared it prematurely... next time I'll wait at least a day :)

Awaiting Vuma investigation anxiously.
 

DrJohnZoidberg

Honorary Master
Joined
Jul 24, 2006
Messages
22,406
Curious if this happened?
It definitely seems better than last week but it's still not where it should be.

Code:
     Server: Vox Telecom - Johannesburg (id = 7254)
        ISP: Web-Squad-Connect
    Latency:    19.87 ms   (0.20 ms jitter)
   Download:   310.19 Mbps (data used: 434.3 MB)
     Upload:    95.59 Mbps (data used: 85.7 MB)
Packet Loss: Not available.
 Result URL: https://www.speedtest.net/result/c/67f3bec9-23f0-403e-a2d4-0aeef1cdecb2
tim@miniplex:~$ speedtest -s 12286 -p yes

   Speedtest by Ookla

     Server: Afrihost - Johannesburg (id = 12286)
        ISP: Web-Squad-Connect
    Latency:    21.51 ms   (0.83 ms jitter)
   Download:   448.18 Mbps (data used: 480.3 MB)
     Upload:    95.57 Mbps (data used: 87.5 MB)
Packet Loss: Not available.
 Result URL: https://www.speedtest.net/result/c/2afbf83f-fb38-4290-9249-d8da4a13828d
tim@miniplex:~$ speedtest -s 1817 -p yes

   Speedtest by Ookla

     Server: Mweb - Johannesburg (id = 1817)
        ISP: Web-Squad-Connect
    Latency:    22.54 ms   (0.39 ms jitter)
   Download:   454.55 Mbps (data used: 687.3 MB)
     Upload:    95.61 Mbps (data used: 96.4 MB)
Packet Loss:     0.0%
 Result URL: https://www.speedtest.net/result/c/b0668e6a-55ec-485b-b8a5-a5732aa48550
tim@miniplex:~$ speedtest -s 2990 -p yes

   Speedtest by Ookla

     Server: MTN SA - Johannesburg (id = 2990)
        ISP: Web-Squad-Connect
    Latency:    21.41 ms   (0.23 ms jitter)
   Download:   480.01 Mbps (data used: 850.4 MB)
     Upload:    95.59 Mbps (data used: 84.6 MB)
Packet Loss:     0.0%
 Result URL: https://www.speedtest.net/result/c/3c93e813-2a81-48b1-9624-3c8ca27fba4c
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,167
Curious if this happened?
It definitely seems better than last week but it's still not where it should be.

Code:
     Server: Vox Telecom - Johannesburg (id = 7254)
        ISP: Web-Squad-Connect
    Latency:    19.87 ms   (0.20 ms jitter)
   Download:   310.19 Mbps (data used: 434.3 MB)
     Upload:    95.59 Mbps (data used: 85.7 MB)
Packet Loss: Not available.
Result URL: https://www.speedtest.net/result/c/67f3bec9-23f0-403e-a2d4-0aeef1cdecb2
tim@miniplex:~$ speedtest -s 12286 -p yes

   Speedtest by Ookla

     Server: Afrihost - Johannesburg (id = 12286)
        ISP: Web-Squad-Connect
    Latency:    21.51 ms   (0.83 ms jitter)
   Download:   448.18 Mbps (data used: 480.3 MB)
     Upload:    95.57 Mbps (data used: 87.5 MB)
Packet Loss: Not available.
Result URL: https://www.speedtest.net/result/c/2afbf83f-fb38-4290-9249-d8da4a13828d
tim@miniplex:~$ speedtest -s 1817 -p yes

   Speedtest by Ookla

     Server: Mweb - Johannesburg (id = 1817)
        ISP: Web-Squad-Connect
    Latency:    22.54 ms   (0.39 ms jitter)
   Download:   454.55 Mbps (data used: 687.3 MB)
     Upload:    95.61 Mbps (data used: 96.4 MB)
Packet Loss:     0.0%
Result URL: https://www.speedtest.net/result/c/b0668e6a-55ec-485b-b8a5-a5732aa48550
tim@miniplex:~$ speedtest -s 2990 -p yes

   Speedtest by Ookla

     Server: MTN SA - Johannesburg (id = 2990)
        ISP: Web-Squad-Connect
    Latency:    21.41 ms   (0.23 ms jitter)
   Download:   480.01 Mbps (data used: 850.4 MB)
     Upload:    95.59 Mbps (data used: 84.6 MB)
Packet Loss:     0.0%
Result URL: https://www.speedtest.net/result/c/3c93e813-2a81-48b1-9624-3c8ca27fba4c

Yup, firmware update seems to have had a few small improvements (and dropped CPU usage on our boxes, which is a good sign). However, still waiting on vendors to make some minor adjustments their end. One vendor has maintenance scheduled for later this week. More updates to follow.
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,167
Question to Vumatel users: Would an add-on (IE a small extra cost per month) SLA interest you?
  • Assuming it provided you with the following:
    • Quicker access to support and escalations (We're talking from the Vuma end here) and;
    • Guaranteed Mean Time to Repair (ie, your line is repaired within a certain time frame between Monday and Saturday)
This would be a pseudo SLA (with proposed numbers that aren't far off what the FTTB providers provide). Comment below whether you would be interested in this add on and what you would be willing to pay for a service like this.
 

DrJohnZoidberg

Honorary Master
Joined
Jul 24, 2006
Messages
22,406
I still find this hilarious though, although I'm really not complaining :laugh:

Code:
   Speedtest by Ookla

     Server: 31173 Services AB - Amsterdam (id = 23094)
        ISP: Web-Squad-Connect
    Latency:   149.82 ms   (0.66 ms jitter)
   Download:   927.08 Mbps (data used: 1.4 GB)
     Upload:    97.07 Mbps (data used: 131.8 MB)
Packet Loss:     0.0%
 Result URL: https://www.speedtest.net/result/c/4e7c925c-acd1-480f-bbd6-6a4f003d9405

Maybe we should just route everything through Amsterdam.

1615810998247.png
 

netstrider

Expert Member
Joined
Nov 1, 2006
Messages
2,657
Question to Vumatel users: Would an add-on (IE a small extra cost per month) SLA interest you?
  • Assuming it provided you with the following:
    • Quicker access to support and escalations (We're talking from the Vuma end here) and;
    • Guaranteed Mean Time to Repair (ie, your line is repaired within a certain time frame between Monday and Saturday)
This would be a pseudo SLA (with proposed numbers that aren't far off what the FTTB providers provide). Comment below whether you would be interested in this add on and what you would be willing to pay for a service like this.
What is a realistic amount (ballpark)? It's all good and dandy saying R100, but we have no idea how realistic that is for such a service considering the massive price differences between FTTH and FTTB.
 
Top