Web Squad ISP

Status
Not open for further replies.
websquadza said:
Thanks for this
MTR looks clear. Definitely not imagining things, just trying to isolate the potential cause. Desktop youtube loading fine? Do you have IPv6 enabled on your router? Also try swapping DNS to cloudflare 1.1.1.1
Click to expand...
Desktop youtube seems fine, don't have IPv6 enabled, never gotten around to it.
Swapping DNS on mobile to 1.1.1.1 didn't make a difference to loading,
Have swapped desktop to 1.1.1.1 also to see if the Google meets break ups go away.
 
Chiller89 said:
Desktop youtube seems fine, don't have IPv6 enabled, never gotten around to it.
Swapping DNS on mobile to 1.1.1.1 didn't make a difference to loading,
Have swapped desktop to 1.1.1.1 also to see if the Google meets break ups go away.
Click to expand...
Oh god, Google Meets have been a nightmare for a couple of weeks with random stoppages every now and again, 1.1.1.1 won't help with those, I'm using CF for DNS as it is. We (company) suspect that something iffy happened on Google's side, because there are only issues when we are in a meet with people from EU/UK. US/Asia/Africa is fine.
 
DarkSt0rm said:
Oh god, Google Meets have been a nightmare for a couple of weeks with random stoppages every now and again, 1.1.1.1 won't help with those, I'm using CF for DNS as it is. We (company) suspect that something iffy happened on Google's side, because there are only issues when we are in a meet with people from EU/UK. US/Asia/Africa is fine.
Click to expand...
I was just about to set up a test call to an offnet server and see how long it lasts, but you reckon it's more the international leg?
 
Chiller89 said:
Desktop youtube seems fine, don't have IPv6 enabled, never gotten around to it.
Swapping DNS on mobile to 1.1.1.1 didn't make a difference to loading,
Have swapped desktop to 1.1.1.1 also to see if the Google meets break ups go away.
Click to expand...
What router are you using?
 
websquadza said:
I was just about to set up a test call to an offnet server and see how long it lasts, but you reckon it's more the international leg?
Click to expand...
It's something in the EU. Spent a good 3 hours with infrastructure guys from Canada last week with the only drop being loadshedding switchover. A few hours earlier we had the CTO in the UK on the same call and was dropping (Canadian guys were also dropping) every 20ish minutes. Moving to teams this week because of it.
 
DarkSt0rm said:
Oh god, Google Meets have been a nightmare for a couple of weeks with random stoppages every now and again, 1.1.1.1 won't help with those, I'm using CF for DNS as it is. We (company) suspect that something iffy happened on Google's side, because there are only issues when we are in a meet with people from EU/UK. US/Asia/Africa is fine.
Click to expand...
Follow up here, any luck on those tests?
 
DarkSt0rm said:
It's something in the EU. Spent a good 3 hours with infrastructure guys from Canada last week with the only drop being loadshedding switchover. A few hours earlier we had the CTO in the UK on the same call and was dropping (Canadian guys were also dropping) every 20ish minutes
Click to expand...
If there are error outputs on Google meets (similar to Youtube), please forward these and I'll post a ticket on the Google ISP portal. This sounds like a large issue on their internal routes?
 
websquadza said:
If there are error outputs on Google meets (similar to Youtube), please forward these and I'll post a ticket on the Google ISP portal. This sounds like a large issue on their internal routes?
Click to expand...
Nothing, diagnostics only shows a slight ms bump (20-30ms) before a huge stoppage. Got a 2 hour call with the infrastructure guys in the UAE in 20 minutes, so will keep an eye on it again.
 
LON1 is stable again. Issues are so transitory. Wonder if it is SFP related. I just don't have time to come out and collect a new one from you right now :sob:

1676280872493.png
 
DarkSt0rm said:
Nothing, diagnostics only shows a slight ms bump (20-30ms) before a huge stoppage. Got a 2 hour call with the infrastructure guys in the UAE in 20 minutes, so will keep an eye on it again.
Click to expand...
Please do, Google NOC is painful, so any and all stats at call fail will help.

DarkSt0rm said:
LON1 is stable again. Issues are so transitory. Wonder if it is SFP related. I just don't have time to come out and collect a new one from you right now :sob:

View attachment 1474915
Click to expand...
Thanks for confirming. Hop 5 is the handover to us (2,3,4 are our IPs configured in a VRF within Vuma's core network, with 2 being your area L3 headend, and 3 and 4 being internal routing before reaching us on 5. 3 and 4 are severely ICMP rate limited), and in your previous plot, 5 showed almost 50% drops. We do also implement ICMP rate limiting on certain devices, but not 50%. Obviously with first hop not being clear, we can't see if that SFP in the CPE was at play today.
 
The YouTube issues are back and I confirmed a VPN resolves it. I shared an update in my ticket.

The issue seems to be specific to YouTube (or Google) since other websites are working
 
JOEY_8 said:
The YouTube issues are back and I confirmed a VPN resolves it. I shared an update in my ticket.

The issue seems to be specific to YouTube (or Google) since other websites are working
Click to expand...
We do think this has something to do with Google peering in JHB, but trying to isolate it. VPN usually helps for one of two reasons:
1. MTU issues: VPNs auto clamp MSS for connections, so they resolve MTU issues
2. Bypass local routes: VPNs tend to carry traffic from elsewhere, so might be that Google has an issue on JHB peering, and the VPN is picking up the traffic elsewhere.
 
JOEY_8 said:
The YouTube issues are back and I confirmed a VPN resolves it. I shared an update in my ticket.

The issue seems to be specific to YouTube (or Google) since other websites are working
Click to expand...
Chiller89 said:
Desktop youtube seems fine, don't have IPv6 enabled, never gotten around to it.
Swapping DNS on mobile to 1.1.1.1 didn't make a difference to loading,
Have swapped desktop to 1.1.1.1 also to see if the Google meets break ups go away.
Click to expand...

Can you check on this again? We've updated announces and trying to steer to different Google egress ports to see if it makes a difference
 
websquadza said:
Please do, Google NOC is painful, so any and all stats at call fail will help.


Thanks for confirming. Hop 5 is the handover to us (2,3,4 are our IPs configured in a VRF within Vuma's core network, with 2 being your area L3 headend, and 3 and 4 being internal routing before reaching us on 5. 3 and 4 are severely ICMP rate limited), and in your previous plot, 5 showed almost 50% drops. We do also implement ICMP rate limiting on certain devices, but not 50%. Obviously with first hop not being clear, we can't see if that SFP in the CPE was at play today.
Click to expand...
I'll disable ICMP filtering on the router for a few days tonight.
 
websquadza said:
Can you check on this again? We've updated announces and trying to steer to different Google egress ports to see if it makes a difference
Click to expand...
I noticed now that my YouTube app was still giving problems, but opening YouTube in a browser was working.

Update: issue is back
 
Status
Not open for further replies.
Back
Top
Sign up to the MyBroadband newsletter
X