Web Squad ISP

Status
Not open for further replies.
ghostRgg said:
JHB -> MEA (Overwatch 15.185.58.194) = 124ms
DBN -> MEA = 152ms

Noticing a lot of other ISP's are also having MEA issues from Durban. when it should technically be the lowest latency. If its just an Amazon hiccup then let's hope it doesn't turn into Google re-announcing its routes every 10 hours :/
Click to expand...

Amazon doesn’t have a routing core in KZN, just L2 paths between mtunzini and JHB (which explains the higher latency in KZN. This means ISPs need to pick up AWS traffic using their NLD or transit capacity from Amazon’s nearest core- JHB.

Unfortunately AWS currently controls their entire local routing table themselves (including what they advertise and how they route return traffic), they also favour peering traffic.. And for a cloud provider, stability and speed outweighs latency when it comes to routing decisions. We’ve seen this with all the major providers when something goes wrong or capacity is somewhat constrained.
AWS also tends to take much longer to restore paths after an outage (WACS had a blip or two around the new year) which means they usually leave traffic on longer, less preferred routes for a while after an outage before switching back.
 
SFW007 said:
Thanks bud .....please keep me posted as middle east also up to sh*t
Click to expand...
websquadza said:
Evening all. Apologies for the tardy response today. We have picked up on these losses throughout the day. They seem to be on the WACS path to EU. We have escalated to our upstream for further investigation. Will provide feedback shortly.
Click to expand...

AWS traffic aside, some interface issues have been picked up on our upstream side. They’re planning non-service affecting maintenance to resolve this, time and date to be confirmed. We’re also adding some fresh routes into the mix next week as soon as our core is out of freeze.
 
websquadza said:
Amazon doesn’t have a routing core in KZN, just L2 paths between mtunzini and JHB (which explains the higher latency in KZN. This means ISPs need to pick up AWS traffic using their NLD or transit capacity from Amazon’s nearest core- JHB.

Unfortunately AWS currently controls their entire local routing table themselves (including what they advertise and how they route return traffic), they also favour peering traffic.. And for a cloud provider, stability and speed outweighs latency when it comes to routing decisions. We’ve seen this with all the major providers when something goes wrong or capacity is somewhat constrained.
AWS also tends to take much longer to restore paths after an outage (WACS had a blip or two around the new year) which means they usually leave traffic on longer, less preferred routes for a while after an outage before switching back.
Click to expand...
Just makes me sad. WE ARE RIGHT HERE DAMN IT.

I totally understand AWS's routing due to stability and all the issues we have seen in some KZN backhauls, just curious to see if we could get lower than 120 here, doubt it but oh well. Imagine the latency to MEA if we had a direct link from Teraco -> Mtunzini -> EASSy. One can only dream I guess.

Also can't wait to see all the fresh new routes!
 
Getting a lot of packet loss tonight to the wow servers. It's been manageable for the past few hours but its heading towards being unplayable.

1610135806317.png
 
DarkSt0rm said:
Getting a lot of packet loss tonight to the wow servers. It's been manageable for the past few hours but its heading towards being unplayable.

View attachment 990182
Click to expand...
Loss starts at your router, and also possible it's your fno. Websquad rep did mention an international issue, but I reckon start by seeing if you can clear the packet loss from the first hop.
 
MDE said:
Loss starts at your router, and also possible it's your fno. Websquad rep did mention an international issue, but I reckon start by seeing if you can clear the packet loss from the first hop.
Click to expand...
Hop 12 loss lines up with the loss on hop 15. The first 2 hops is probably just routers dropping icmp packets, 8 & 9 the same thing. Router reboot couldn't hurt tho. Not sure why router would be rejecting pings, seeing its 98% loss. I've often seen loss on hop 2 with no loss and destination.
 
websquadza said:
AWS traffic aside, some interface issues have been picked up on our upstream side. They’re planning non-service affecting maintenance to resolve this, time and date to be confirmed. We’re also adding some fresh routes into the mix next week as soon as our core is out of freeze.
Click to expand...

These just for JHB again, or are there some being implemented for CPT too? Maybe some more optimal routes from London to the rest of EU?
 
routing netflix to international servers - something seems borked.
 
Seeyou said:
These just for JHB again, or are there some being implemented for CPT too? Maybe some more optimal routes from London to the rest of EU?
Click to expand...

Changes happened in CPT, JHB and LON. Maintenance was confirmed completed Friday and we’ve been monitoring it closely. Packet loss has cleared.
Sopbeen said:
routing netflix to international servers - something seems borked.
Click to expand...

Any chance you have an IP for me to trace to? Netflix are pretty much peering only (they have little actual transit presence), so we need to be able to trace which exchange this is from and whether prefixes are being advertised and accepted.

If you’re pulling via VPN, that’s then their peering and transit. If you’re using a DNS tool, then Netflix’s distribution still sees our prefix in SA and chooses to route that way.

Netflix also have forceful policies to push content to the nearest CDN, remember caches don’t necessarily correlate with regional restrictions, rather asynchronous loading should the content be requested by client. There are some pretty cool vids explaining how Netflix does this- will find some and post the link in case you’re interested.
 
@websquadza just logged a support ticket 792650. Not sure if there's issues just on my link or in general.

Getting packet loss to some international sites including random ping spikes in games (Apex, Overwatch) on EU servers.

Could you check please?
 
websquadza said:
Changes happened in CPT, JHB and LON. Maintenance was confirmed completed Friday and we’ve been monitoring it closely. Packet loss has cleared.


Any chance you have an IP for me to trace to? Netflix are pretty much peering only (they have little actual transit presence), so we need to be able to trace which exchange this is from and whether prefixes are being advertised and accepted.

If you’re pulling via VPN, that’s then their peering and transit. If you’re using a DNS tool, then Netflix’s distribution still sees our prefix in SA and chooses to route that way.

Netflix also have forceful policies to push content to the nearest CDN, remember caches don’t necessarily correlate with regional restrictions, rather asynchronous loading should the content be requested by client. There are some pretty cool vids explaining how Netflix does this- will find some and post the link in case you’re interested.
Click to expand...
Code: 
Target Name: netflix.com
         IP: 52.213.155.117
  Date/Time: 10 Jan 2021 16:32:21 - 10 Jan 2021 16:42:21

Hop  Sent  PL%     Min     Max     Avg  Host Name / [IP]
  1    11    0    0,28    1,71    0,49  192.168.88.1 [192.168.88.1]
  2    11    0    2,01    3,08    2,57  core.as-01.cp1.za.ws.net.za [160.119.238.1]
  3    11    0    2,17    3,29    2,62  core.cr-01.cp1.za.ws.net.za [160.119.233.189]
  4    11    0    2,22    3,36    2,59  core.pe-xe-ix01.cp1.za.ws.net.za [160.119.232.20]
  5    11    0    1,80   18,45    3,96  196.10.140.105 [196.10.140.105]
  6    11    0    2,96    9,03    4,31  52.93.57.88 [52.93.57.88]
  7    11    0    2,15    8,57    3,12  52.93.57.103 [52.93.57.103]
  8    11    0  151,22  152,48  151,85  52.93.57.170 [52.93.57.170]
  9    11    0  151,38  156,76  152,82  52.93.134.66 [52.93.134.66]
 10    11    0  151,55  153,37  152,00  52.93.134.189 [52.93.134.189]
 11    11  100       0       0       0   [-]
 12    11  100       0       0       0   [-]
 13    11    0  151,22  155,86  152,83  52.93.128.234 [52.93.128.234]
 14    11  100       0       0       0  netflix.com [52.213.155.117]
 
Sopbeen said:
Code: 
Target Name: netflix.com
         IP: 52.213.155.117
  Date/Time: 10 Jan 2021 16:32:21 - 10 Jan 2021 16:42:21

Hop  Sent  PL%     Min     Max     Avg  Host Name / [IP]
  1    11    0    0,28    1,71    0,49  192.168.88.1 [192.168.88.1]
  2    11    0    2,01    3,08    2,57  core.as-01.cp1.za.ws.net.za [160.119.238.1]
  3    11    0    2,17    3,29    2,62  core.cr-01.cp1.za.ws.net.za [160.119.233.189]
  4    11    0    2,22    3,36    2,59  core.pe-xe-ix01.cp1.za.ws.net.za [160.119.232.20]
  5    11    0    1,80   18,45    3,96  196.10.140.105 [196.10.140.105]
  6    11    0    2,96    9,03    4,31  52.93.57.88 [52.93.57.88]
  7    11    0    2,15    8,57    3,12  52.93.57.103 [52.93.57.103]
  8    11    0  151,22  152,48  151,85  52.93.57.170 [52.93.57.170]
  9    11    0  151,38  156,76  152,82  52.93.134.66 [52.93.134.66]
10    11    0  151,55  153,37  152,00  52.93.134.189 [52.93.134.189]
11    11  100       0       0       0   [-]
12    11  100       0       0       0   [-]
13    11    0  151,22  155,86  152,83  52.93.128.234 [52.93.128.234]
14    11  100       0       0       0  netflix.com [52.213.155.117]
Click to expand...

This IP is on AWS and behind AWS peering (hop 5). Doesn’t look like a source CDN server though, more a HTTP front end web server. Any chance you can pick up on the CDN server(s) you’re trying to pull content from? (I know this is a bit harder to find).
 
Senshi said:
@websquadza just logged a support ticket 792650. Not sure if there's issues just on my link or in general.

Getting packet loss to some international sites including random ping spikes in games (Apex, Overwatch) on EU servers.

Could you check please?
Click to expand...

We’ve responded on your ticket
 
websquadza said:
Netflix also have forceful policies to push content to the nearest CDN, remember caches don’t necessarily correlate with regional restrictions, rather asynchronous loading should the content be requested by client. There are some pretty cool vids explaining how Netflix does this- will find some and post the link in case you’re interested.
Click to expand...
id be very interested to watch these if you can please post some links :)
 
websquadza said:
This IP is on AWS and behind AWS peering (hop 5). Doesn’t look like a source CDN server though, more a HTTP front end web server. Any chance you can pick up on the CDN server(s) you’re trying to pull content from? (I know this is a bit harder to find).
Click to expand...
Yep the CDN is pointing to jhb. The problem seems to have resolved itself. Sooooo I think we can stop wasting time and have a beer. Cheers.
 
Seeyou said:
I meant the incoming new routes :)
Click to expand...
Seeyou said:
I guess I'll have to wait and see :p
Click to expand...

Sorry for the late response here. Most timelines we were given last year have fallen behind by months (Various reasons for these delays) - which is frustrating to say the least. We are obviously working hard to implement these asap, and as soon as we have feedback from vendors we can start implementing accordingly.
 
Status
Not open for further replies.
Back
Top
Sign up to the MyBroadband newsletter
X