Web Squad ISP

[daelm]

Evening all, busy investigating this- will have an update shortly.

it's not necessarily you. my backup service won't load gmail either, for example

 

[websquadza]

it's not necessarily you. my backup service won't load gmail either, for example

Thanks for the update. Yeah, the NOG telegram group is lighting up like mad at the moment. Seems quite wide spread

 

[daelm]

it's not just google services, though

 

[PsyWulf]

Thanks for the update. Yeah, the NOG telegram group is lighting up like mad at the moment. Seems quite wide spread

Indeed

Multiple providers

 

[daelm]

someone dun ****ed up, basically

 

[BouncyNinja]

No go

 

[Concentric]

No go

Im back up. youtube loads fine

 

[BouncyNinja]

Is baaack

 

[BouncyNinja]

Slightly off topic,

Who thinks it’ll stick the landing this time

 

[GoneWithTheWindows]

Update - looks like multiple Google services are down - seems pretty widespread as we're hearing multiple networks seeing the same. Google 2 (there are 2 peering cores) in JHB seems to be down. I think Google will do a little load shifting and re-routing shortly

If you're struggling to load anything, make sure you aren't stuck on just Google DNS (ie 8.8.8.8 and 8.8.4.4). Try add Cloudflare (1.1.1.1, 1.0.0.1) or Quad9 (9.9.9.9) into the mix.

Also remember, a bunch of 3rd party services run on Google Cloud, so it may explain other services giving issues.

more updates to follow once we hear more.

This has nothing to do with google services? My issues is completely un related.

 

[websquadza]

Is baaack

Seeing services restore - seems Google are on it.

 

[websquadza]

This has nothing to do with google services? My issues is completely un related.

View attachment 1030284

Definitely Google related outage that affected everyone.

Your traffic trace there seems to be getting very far into the interwebs. Leaves us, heads onto 2 other networks and drops off. Let me see what's potting here.

Edit (and correction) - Looks like there was a flap on WACS around the same time that we didn't see (most traffic moved to alternate routes fairly quickly)

 

[websquadza]

Further update. Google services seem mostly restored (for all affected networks). EASSy is down for some (break in the Med), WACS seems to have taken a dive for a few and ISPs using Seacom (cable) also reported issues (which may or may not be related). This means some local CDNs are probably going stale too (hence certain content not loading across other services).

A lot of investigations going on all round. About 200 Gbps of traffic died off at NAP JHB (https://www.napafrica.net/peering-traffic).

 

[daelm]

Seeing services restore - seems Google are on it.

back up

 

[DrJohnZoidberg]

Getting back to my slow speeds with Steam. I decided to look a little deeper and even when I have the Cape Town server selected in Steam it's downloading from a server in Joburg.

Checking Wireshark and found the IP being used by Steam on the download which is (155.133.238.146)

Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                      Packets               Pings
Host                                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
1. Internet                                                         0.0%     5    0.5   0.6   0.5   0.6   0.0
2. core.as-01.cp1.za.ws.net.za                                      0.0%     5    2.2   2.4   1.9   2.7   0.3
3. core.cr-01.cp1.za.ws.net.za                                      0.0%     5    1.9   2.2   1.9   2.8   0.4
4. core.pe-xe-ix01.jb1.za.ws.net.za                                 0.0%     4   21.8  22.0  21.7  22.6   0.4
5. 196-60-9-22.ixp.joburg                                           0.0%     4   22.4  24.8  21.6  32.5   5.1
6. 155.133.238.146                                                  0.0%     4   21.8  22.0  21.7  22.4   0.3

Second problem is that for some reason it's slower than downloading from the Belgium server.

@websquadza Any ideas on what's going on here or how I could start to resolve this. I didn't have this problem with my Rocketnet connection, it always maxed out my 200Mbit line now I've got a gigabit line and can't even get 200Mbps.

EDIT: Also just confirmed that switching between Cape Town and Joburg servers in Steam has no effect, they both end up resolving the same IP.

 

[DrJohnZoidberg]

Managed to get around 500Mbps+ to the Belgium server but cannot hit 200Mbps to Johannesburg.

This is the route for the Belgium server.

Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                      Packets               Pings
Host                                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
1. Internet                                                         0.0%     9    0.6   0.5   0.5   0.6   0.0
2. core.as-01.cp1.za.ws.net.za                                      0.0%     9    2.6   2.1   1.7   2.6   0.3
3. core.cr-01.cp1.za.ws.net.za                                      0.0%     8    2.2   2.1   1.7   2.5   0.3
4. 165-69-148-197.as37497.za.net                                    0.0%     8    2.0   2.0   1.8   2.5   0.2
5. 162-68-148-197.as37497.za.net                                    0.0%     8    2.3   8.5   2.2  48.8  16.3
6. 14-71-148-197.as37497.za.net                                     0.0%     8  144.8 145.3 144.7 148.5   1.3
7. (waiting for reply)
8. (waiting for reply)
9. VALVE-CORPO.ear3.Amsterdam1.Level3.net                           0.0%     8  174.1 174.1 173.6 174.7   0.3
10. 155.133.248.28                                                   0.0%     8  152.3 165.6 151.6 174.1  11.3

 

[DeatheCore]

Getting back to my slow speeds with Steam. I decided to look a little deeper and even when I have the Cape Town server selected in Steam it's downloading from a server in Joburg.

Checking Wireshark and found the IP being used by Steam on the download which is (155.133.238.146)

Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                      Packets               Pings
Host                                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
1. Internet                                                         0.0%     5    0.5   0.6   0.5   0.6   0.0
2. core.as-01.cp1.za.ws.net.za                                      0.0%     5    2.2   2.4   1.9   2.7   0.3
3. core.cr-01.cp1.za.ws.net.za                                      0.0%     5    1.9   2.2   1.9   2.8   0.4
4. core.pe-xe-ix01.jb1.za.ws.net.za                                 0.0%     4   21.8  22.0  21.7  22.6   0.4
5. 196-60-9-22.ixp.joburg                                           0.0%     4   22.4  24.8  21.6  32.5   5.1
6. 155.133.238.146                                                  0.0%     4   21.8  22.0  21.7  22.4   0.3

Second problem is that for some reason it's slower than downloading from the Belgium server.

@websquadza Any ideas on what's going on here or how I could start to resolve this. I didn't have this problem with my Rocketnet connection, it always maxed out my 200Mbit line now I've got a gigabit line and can't even get 200Mbps.

EDIT: Also just confirmed that switching between Cape Town and Joburg servers in Steam has no effect, they both end up resolving the same IP.

I've noticed this too with my Vuma 1Gbps + CISP connection. Perhaps Steam ZA have some sort of issues and/or server side rate limiting in effect

 

[websquadza]

Getting back to my slow speeds with Steam. I decided to look a little deeper and even when I have the Cape Town server selected in Steam it's downloading from a server in Joburg.

Checking Wireshark and found the IP being used by Steam on the download which is (155.133.238.146)

Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                      Packets               Pings
Host                                                               Loss%   Snt   Last   Avg  Best  Wrst StDev
1. Internet                                                         0.0%     5    0.5   0.6   0.5   0.6   0.0
2. core.as-01.cp1.za.ws.net.za                                      0.0%     5    2.2   2.4   1.9   2.7   0.3
3. core.cr-01.cp1.za.ws.net.za                                      0.0%     5    1.9   2.2   1.9   2.8   0.4
4. core.pe-xe-ix01.jb1.za.ws.net.za                                 0.0%     4   21.8  22.0  21.7  22.6   0.4
5. 196-60-9-22.ixp.joburg                                           0.0%     4   22.4  24.8  21.6  32.5   5.1
6. 155.133.238.146                                                  0.0%     4   21.8  22.0  21.7  22.4   0.3

Second problem is that for some reason it's slower than downloading from the Belgium server.

@websquadza Any ideas on what's going on here or how I could start to resolve this. I didn't have this problem with my Rocketnet connection, it always maxed out my 200Mbit line now I've got a gigabit line and can't even get 200Mbps.

EDIT: Also just confirmed that switching between Cape Town and Joburg servers in Steam has no effect, they both end up resolving the same IP.

Steam/valve don’t have any servers in CPT (or peering rather - so no network presence for servers there)- not sure why they would list these. Potentially 3rd party servers acting as caching nodes. If you have an IP for a CPT server, I can confirm if this server is in fact CPT based or JHB based and listed and JHB.

With regards to JHB, we peer with steam/valve at NAP- no issues on our end here. So I think as @DeatheCore mentioned above- these may be server side restrictions. Valve only have 20G of peering capacity (according to peeringDB) in JHB- which can easily come under strain on big days. That said, we’re running a few tests just to make sure.

 

[CrypticZA]

Just some stats i pulled from steam

 

[DrJohnZoidberg]

Steam/valve don’t have any servers in CPT (or peering rather - so no network presence for servers there)- not sure why they would list these. Potentially 3rd party servers acting as caching nodes. If you have an IP for a CPT server, I can confirm if this server is in fact CPT based or JHB based and listed and JHB.

With regards to JHB, we peer with steam/valve at NAP- no issues on our end here. So I think as @DeatheCore mentioned above- these may be server side restrictions. Valve only have 20G of peering capacity (according to peeringDB) in JHB- which can easily come under strain on big days. That said, we’re running a few tests just to make sure.

Thanks, I looked into this again after seeing this post: https://mybroadband.co.za/forum/index.php?posts/27036882

Clearly the servers are capable enough so I'm wondering why my speeds to are so poor.

I did also look inside some steam config files and i don't see any references to Cape town servers so i assume that's maybe just a placeholder maybe or only available sometimes, dunno.