Web Squad ISP

Senshi

Well-Known Member
Joined
Mar 15, 2005
Messages
439
Thank you for this feedback! Passed it onto him!


Seeing lots of physical link drops on eth 3 and 4 in these logs; could be a bad cable? I'm assuming one of these ports connects to your ONT? Or the ONT could be restarting. Wouldn't see issues on the ethernet physical interfaces if the issue is fibre related.

What line speed? these all seem stuck at about 100 Mbps? Could be related to the above eth phy issue?

I'll change out the cable (I have spares) with a new one, however doubt that it's the cause as then the issues wouldn't be as random (perfectly fine for most the day, then between 4 & 5pm super weirdness).

I'm on a 200mb package, getting half that.
 

Krazee

Member
Joined
Nov 28, 2019
Messages
28
I'll change out the cable (I have spares) with a new one, however doubt that it's the cause as then the issues wouldn't be as random (perfectly fine for most the day, then between 4 & 5pm super weirdness).

I'm on a 200mb package, getting half that.
Are you by chance on the Radiokop OLT? , if so do you have any random bursts of loss?

I've been battling with Vuma for the last 3 months with issues on my line.
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,593
I'll change out the cable (I have spares) with a new one, however doubt that it's the cause as then the issues wouldn't be as random (perfectly fine for most the day, then between 4 & 5pm super weirdness).

I'm on a 200mb package, getting half that.

If you can clear those eth up/downs, that will help narrow it down. Seeing as the only active portion is between the ONT and the router and router and your PC, it's important to clear those. Even a bad line won't cause eth ports to flap like that.

Take this for example, 1G duplex on the phy, but 4 flaps in the 12s before it settles.

Code:
Dec  1 16:02:45 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) (phyId: b) Link DOWN.
Dec  1 16:02:46 roamast: sta A8:DB:03:86:95:DA ap 04:D9:F5:E6:3F:D4 rcpi 74
Dec  1 16:02:46 roamast: after rssi -73
Dec  1 16:02:47 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) (phyId: b) Link UP at 1000 mbps full duplex
Dec  1 16:02:48 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) (phyId: b) Link DOWN.
Dec  1 16:02:51 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) (phyId: b) Link UP at 1000 mbps full duplex
Dec  1 16:02:52 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) (phyId: b) Link DOWN.
Dec  1 16:02:54 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) (phyId: b) Link UP at 1000 mbps full duplex
Dec  1 16:02:56 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) (phyId: b) Link DOWN.
Dec  1 16:02:58 kernel: eth4 (Ext switch port: 3) (Logical Port: 11) (phyId: b) Link UP at 1000 mbps full duplex

That and a bunch of wifi disassociations.

Usually when a link greater than 100M settles on 100M (94-95M), this is a sign that a port somewhere is negotiating 100 Mbps on the phy instead of 1000 Mbps. This could be between ONT and router or Router and PC.
 

Senshi

Well-Known Member
Joined
Mar 15, 2005
Messages
439
Are you by chance on the Radiokop OLT? , if so do you have any random bursts of loss?

I've been battling with Vuma for the last 3 months with issues on my line.

Yep, in Radiokop as well.

Happens every now and then, just random connectivity loss for a few seconds then comes back up.

@websquadza

I've changed the uplink cable between router & ONT with brand new cat6 shielded, upgraded firmware of both router & access points (for good measure), did a full shutdown and restart and speeds have improved but not back to 200/200 yet.

Will monitor and post feedback.
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,593
Some good news for Openserve Users: 50 Mbps packages are now symmetrical (50 Mbps down and 50 Mbps up). Existing clients, please allow a week or so for Openserve to finish pushing these upgrades across the board.
 

Napalm

Expert Member
Joined
Jun 22, 2006
Messages
3,005
Whats going on with gaming latency to europe please. Its been pretty bad last week or so.
 

Napalm

Expert Member
Joined
Jun 22, 2006
Messages
3,005
Do you have an example or ticket I can look at? Area and FNO?
No ticket no. Just been seeing 260 to 270ms ping to EU server 8.209.112.253 - Genshin impact

Im in bellville, capetown. Vuma trenched.

You can test pings from your side too.

It used to be solid 139ms , 145ms or on a bad day 169ms
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,593
No ticket no. Just been seeing 260 to 270ms ping to EU server 8.209.112.253 - Genshin impact

Im in bellville, capetown. Vuma trenched.

You can test pings from your side too.

It used to be solid 139ms , 145ms or on a bad day 169ms

Checked this IP:

Code:
|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                            172.16.106.1 -    0 |   72 |   72 |    0 |    0 |    1 |    0 |
|          core.as-01-xe.cp1.za.ws.net.za -    0 |   72 |   72 |    0 |    0 |    5 |    0 |
|                            100.99.197.1 -    0 |   72 |   72 |    0 |    0 |    2 |    1 |
|        core.pe-xe-ip01.cp1.za.ws.net.za -    0 |   72 |   72 |    0 |    0 |    4 |    0 |
|           165-69-148-197.as37497.za.net -    0 |   72 |   72 |    0 |    0 |    3 |    1 |
|           162-68-148-197.as37497.za.net -    0 |   72 |   72 |    0 |    0 |    5 |    0 |
|            14-71-148-197.as37497.za.net -    0 |   72 |   72 |  143 |  143 |  148 |  143 |
|           5-1-8.ear4.London2.Level3.net -   87 |   15 |    2 |    0 |  144 |  145 |  144 |
|                   No response from host -  100 |   14 |    0 |    0 |    0 |    0 |    0 |
|             ldn-b7-link.ip.twelve99.net -    0 |   72 |   72 |  144 |  155 |  425 |  144 |
|            ldn-bb1-link.ip.twelve99.net -    0 |   72 |   72 |  144 |  144 |  145 |  145 |
|            prs-bb1-link.ip.twelve99.net -   40 |   28 |   17 |    0 |  158 |  160 |  159 |
|            ffm-bb1-link.ip.twelve99.net -    0 |   72 |   72 |  158 |  158 |  161 |  158 |
|             ffm-b1-link.ip.twelve99.net -    0 |   72 |   72 |  158 |  160 |  173 |  160 |
|alicloud-svc068046-ic352160.ip.twelve99-cust.net -    0 |   72 |   72 |  229 |  229 |  240 |  229 |
|                   No response from host -  100 |   14 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   14 |    0 |    0 |    0 |    0 |    0 |
|                   No response from host -  100 |   14 |    0 |    0 |    0 |    0 |    0 |
|                           8.209.112.253 -    0 |   72 |   72 |  224 |  225 |  229 |  227 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider

Issue seems to be server's host side, it's hosted in Alibaba's Cloud. Reaching London in 143ms, and next hop (Telia 1299) at 144ms.. From there it's off to Marseilles (france - hop 13 and 14) and then gets handed over to alibaba's cloud there. That's 3+ networks away and routing from Alibaba cloud seems to favour a return path via Singapore. EU latency is good though.

We spun up a VM on Alicloud in Frankfurt and it does indeed route back via Singapore:
Code:
websquad-ali-de (172.31.216.101)                                                                                  2021-12-02T06:39:26+0800
Keys:  Help   Display mode   Restart statistics   Order of fields   quit
                                                                                                  Packets               Pings
 Host                                                                                           Loss%   Snt   Last   Avg  Best  Wrst StDev
 1. (waiting for reply)
 2. 11.201.251.13                                                                                0.0%   280    0.2   0.2   0.2   0.4   0.0
 3. 11.223.248.253                                                                               0.0%   280    5.2   2.3   1.6   6.1   0.6
 4. 11.168.63.254                                                                                0.0%   280    1.0   0.8   0.7   4.0   0.2
 5. 10.54.149.90                                                                                 0.0%   280    0.9   2.0   0.8  32.7   3.6
 6. ix-xe-11-0-0-0.tcore2.fr0-frankfurt.as6453.net                                               0.0%   280    2.3   2.0   0.8  27.1   2.8
 7. if-ae-54-2.tcore1.fr0-frankfurt.as6453.net                                                   0.0%   280  156.6 157.1 156.4 180.7   2.5
 8. if-ae-55-2.tcore2.pvu-paris.as6453.net                                                       0.0%   280  155.0 155.0 154.4 172.9   1.8
 9. if-ae-49-2.tcore1.pvu-paris.as6453.net                                                       0.0%   280  154.8 155.0 154.6 171.9   1.8
10. if-ae-55-2.tcore1.pye-paris.as6453.net                                                       0.0%   280  152.7 152.8 152.5 162.1   0.7
11. if-ae-8-1602.tcore1.wyn-marseille.as6453.net                                                46.2%   280  152.7 153.4 152.6 172.9   2.9
12. if-ae-31-6.tcore1.svw-singapore.as6453.net                                                   0.0%   279  152.9 153.3 152.7 178.7   2.6
13. if-ae-2-2.tcore2.svw-singapore.as6453.net                                                    0.0%   279  155.6 155.1 154.4 174.6   2.3
14. core.cr-01.cp1.za.ws.net.za                                                                  0.0%   279  222.9 223.3 222.9 342.1   7.1
15. core.as-01.cp1.za.ws.net.za                                                                  0.0%   279  223.2 223.5 223.1 276.3   3.9

We'll drop alicloud a ticket to see if they can resolve this; but not much we can do to steer here as this is an Alicloud routing choice.

Edit: confirmed that this is Alicloud preferring routes via our asia paths, which is kind of counter intuitive. Unfortunately adjusting advertisements could affect routes from Singapore, which will affect gamers who prefer that region. We'll wait for Alicloud to respond to our ticket.
 
Last edited:

Seeyou

Expert Member
Joined
May 1, 2007
Messages
2,380
So day 3 of these super slow international speeds. Seems I'm the only one affected? (or complaining about it)

@websquadza - I'd really like to know when I can expect the accelerator to be back up and running and your feedback regarding my question about why my international is so terrible without it, when my iperfs show no loss up to 500mbps. (haven't tested higher)
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,593
So day 3 of these super slow international speeds. Seems I'm the only one affected? (or complaining about it)

@websquadza - I'd really like to know when I can expect the accelerator to be back up and running and your feedback regarding my question about why my international is so terrible without it, when my iperfs show no loss up to 500mbps. (haven't tested higher)
Accelerator has been back on since last night. We're definitely not seeing any issues from our side. Have you checked the usual TCP optimisation techniques (tcp window scaling etc)?
 

Seeyou

Expert Member
Joined
May 1, 2007
Messages
2,380
Accelerator has been back on since last night. We're definitely not seeing any issues from our side. Have you checked the usual TCP optimisation techniques (tcp window scaling etc)?

Absolutely nothing has changed my side.

Speed is better than it was this morning, but I'm not seeing the speeds I was prior to this accelerator issue. NNTP is also still slow. Prior to this issue I had to throttle SABnzbd to 45MB/sec so as not to hit the DDoS protection - now I can barely sustain 40MB/sec. Again, nothing has changed my side. You can scroll back a few pages and find my speed tests of 700-800mbps internationally. Now it struggled to reach this:



Also unsure why you're dodging the question RE un-accelerated speeds. Haven't heard back from you RE the alternate transit as discussed either. I'm not crazy enough to go for an ISP change on Jan 1, but as things stand I may be going elsewhere.
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,593
Absolutely nothing has changed my side.

Speed is better than it was this morning, but I'm not seeing the speeds I was prior to this accelerator issue. NNTP is also still slow. Prior to this issue I had to throttle SABnzbd to 45MB/sec so as not to hit the DDoS protection - now I can barely sustain 40MB/sec. Again, nothing has changed my side. You can scroll back a few pages and find my speed tests of 700-800mbps internationally. Now it struggled to reach this:



Also unsure why you're dodging the question RE un-accelerated speeds. Haven't heard back from you RE the alternate transit as discussed either. I'm not crazy enough to go for an ISP change on Jan 1, but as things stand I may be going elsewhere.
Not dodging un-accelerated speed question- the fact that the box is now on and the speeds have improved shows that it helps maximise utility of services - which it's meant to do. The accelerator doesn't manufacture capacity out of nowhere, it makes the available capacity more usable. So to answer your question, factors beyond our control make it hard to get great speeds without the accelerator, even though the capacity is on hand. The accelerator improves this. That said, it's not guaranteed to assist with every site, every server and every connection. There are many factors that influence TCP at higher latencies, and the accelerator's job is to assist (as best it can) with those. The issue could be server side, peering, transmission, local peering, or last mile - this is why we installed the accelerator; it mitigates against most perceptible issues.

It has the knock on effect of assisting with FNO issues (and sometimes can be said to be there for that); but without an accelerator, surplus capacity generally sits unused on the network as TCP simply won't take advantage of it. You can have a 10G empty pipe from the EU to SA, but without some TCP trickery, the perceived speeds will always be in the low 100s of mbps. One server side tweak which is rolling out (which we have no control over) is something like TCP BBR, which organically improves high latency connections. The accelerator's job is to kind of work out many of these kinks for other connections. Add to that loss factors (even a 0.001%) with higher latency and while the accelerator helps, we can't beat the physics of time (140+ms) and a lost packet). With regards to why you're not seeing last week's speeds - these will usually move based on a number of factors, not the least server load and similar factors on the peering side, transmission network, transti two networks away, in the EU; there are a number of factors at play, but the issue certainly isn't congestion on our network our our upstream. These are speeds we usually see to CoreIX's servers (which are usually a good measure, but not a guarantee) and well within range. We try our best to ensure that we get the best performance (latency and speed) we can to most destinations, but unless it's a peered link and directly contactable via a NOC, we can't make this promise for every destination on the planet.

With regards to the TCP rate limiting by our DDOS protection tool, this will remain in place to protect all our clients. We don't have feedback on what measures our secondary provider is taking to assist in maximising the utility of their service - just that they are working on it; something on their packet processing side can't even be fixed by our TCP accelerator; which is why it's not in production yet.
 

Deadmanza

Honorary Master
Joined
Sep 13, 2013
Messages
12,417
What is going on tonight? Its up and down the whole time both local and international!
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,593
What is going on tonight? Its up and down the whole time both local and international!
@websquadza Also noticed everything drop twice in the last 2 hours. You guys seeing any issues? Vumatel Trenched in Bryanston
Looking into this quickly. We haven’t seen any alarms triggered, so seems fairly isolated. @Deadmanza remind me what area please?

Update: Saw a short sharp drop in traffic volumes (+-30-40%) on Vuma NNI around 20:38. No notices from Vuma in this regard. Seems to be isolated to a Vuma region (Bryanston we imagine).
 
Last edited:

Kryptonite

Senior Member
Joined
Mar 14, 2006
Messages
527
What’s going on with you guys tonight, hard down in kenridge area, network disconnected, cable up
 

websquadza

WebSquad
Company Rep
Joined
Mar 26, 2018
Messages
2,593
What’s going on with you guys tonight, hard down in kenridge area, network disconnected, cable up
FNO? If you're hard down and there aren't any network notices, you're going to need to log it with support; you can do it via the portal or send a mail to support [at] websquad.co.za
 
Last edited:

r00igev@@r

Executive Member
Joined
Dec 14, 2009
Messages
7,950
Not dodging un-accelerated speed question- the fact that the box is now on and the speeds have improved shows that it helps maximise utility of services - which it's meant to do. The accelerator doesn't manufacture capacity out of nowhere, it makes the available capacity more usable. So to answer your question, factors beyond our control make it hard to get great speeds without the accelerator, even though the capacity is on hand. The accelerator improves this. That said, it's not guaranteed to assist with every site, every server and every connection. There are many factors that influence TCP at higher latencies, and the accelerator's job is to assist (as best it can) with those. The issue could be server side, peering, transmission, local peering, or last mile - this is why we installed the accelerator; it mitigates against most perceptible issues.

It has the knock on effect of assisting with FNO issues (and sometimes can be said to be there for that); but without an accelerator, surplus capacity generally sits unused on the network as TCP simply won't take advantage of it. You can have a 10G empty pipe from the EU to SA, but without some TCP trickery, the perceived speeds will always be in the low 100s of mbps. One server side tweak which is rolling out (which we have no control over) is something like TCP BBR, which organically improves high latency connections. The accelerator's job is to kind of work out many of these kinks for other connections. Add to that loss factors (even a 0.001%) with higher latency and while the accelerator helps, we can't beat the physics of time (140+ms) and a lost packet). With regards to why you're not seeing last week's speeds - these will usually move based on a number of factors, not the least server load and similar factors on the peering side, transmission network, transti two networks away, in the EU; there are a number of factors at play, but the issue certainly isn't congestion on our network our our upstream. These are speeds we usually see to CoreIX's servers (which are usually a good measure, but not a guarantee) and well within range. We try our best to ensure that we get the best performance (latency and speed) we can to most destinations, but unless it's a peered link and directly contactable via a NOC, we can't make this promise for every destination on the planet.

With regards to the TCP rate limiting by our DDOS protection tool, this will remain in place to protect all our clients. We don't have feedback on what measures our secondary provider is taking to assist in maximising the utility of their service - just that they are working on it; something on their packet processing side can't even be fixed by our TCP accelerator; which is why it's not in production yet.
Good explanation. You should write a blog so that info is in a search place. It goes missing in the forums.
 
Top