Web Squad ISP

websquadza

WebSquad
Joined
Mar 26, 2018
Messages
1,277
@websquadza

All looks good, websites load fast, Youtube and Twitch stream well, download speeds seem good from a short test and latencies are far better than AH. However, I do seem to have a consistent 1% PL along pretty much the whole route to EU. It's not constant, but I do notice it in-game as small microstutters that appear and then disappear for a while. You can see here it below - it wasn't present on Afrihost at all. Any ideas?

Code:
|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                             192.168.1.1 -    0 | 1802 | 1802 |    0 |    0 |    6 |    0 |
|             core.as-01.cp1.za.ws.net.za -    0 | 1802 | 1802 |    0 |    0 |   13 |    0 |
|             core.cr-01.cp1.za.ws.net.za -    1 | 1775 | 1768 |    0 |    1 |   78 |    1 |
|        core.pe-xe-ip01.cp1.za.ws.net.za -    1 | 1771 | 1763 |    0 |    1 |   76 |    1 |
|           165-69-148-197.as37497.za.net -    1 | 1783 | 1778 |    0 |    1 |   77 |    1 |
|           162-68-148-197.as37497.za.net -    1 | 1755 | 1743 |  141 |  142 |  230 |  142 |
|            54-71-148-197.as37497.za.net -    1 | 1751 | 1738 |  141 |  143 |  234 |  141 |
|            33-66-148-197.as37497.za.net -    2 | 1731 | 1713 |  141 |  141 |  217 |  142 |
|ge-0-1-0-68.peering1.the.lon1.coreix.net -    1 | 1768 | 1759 |  141 |  142 |  228 |  142 |
|        xe-5-1.edge4.enf.lon5.coreix.net -    1 | 1763 | 1753 |  142 |  144 |  230 |  142 |
|        85.13.221.178.reverse.coreix.net -    1 | 1755 | 1743 |  141 |  142 |  281 |  142 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
Addendum to the above - does seem to be affecting my NNTP downloads too. May be nitpicking, but my line maxes to EU NNTP servers at ~19.8MB/sec on 50 connections, whereas before it would hit 21-22MB/sec. Presumably the PL. Could this be an MTU issue? My ER-X is showing MTU is auto set to 1492 on the PPPoE connection.
Welcome to Web sQuad, thanks for the feedback. We're looking into this one. Looks less like packet loss (which would make anything with a high latency unusably slow) and more like a packet drop every now and again (and I identify between the two as the latter doesn't seem to be causing performance headaches, but is noticeable in game) - we're working to see where this is coming from..

With regards to your downloads running at 19.8 vs 21 MB, that sounds more like the processing head on PPPOE - we provision all our PPPOE accounts with 10% extra (so a 200 Mbps profile is actually 225 280 kbps), but not all the line providers do the same. It seems Octotel hard limit to 200 Mbps (and I'm not sure whether they limit speeds by dropping packets or queuing them), which means packet overhead and retransmits on dropped packets at linewire will bring that speed down about 5-10%. Will send you some settings for the ER-X.
 

cavedog

Honorary Master
Joined
Oct 19, 2007
Messages
17,735
@websquadza

All looks good, websites load fast, Youtube and Twitch stream well, download speeds seem good from a short test and latencies are far better than AH. However, I do seem to have a consistent 1% PL along pretty much the whole route to EU. It's not constant, but I do notice it in-game as small microstutters that appear and then disappear for a while. You can see here it below - it wasn't present on Afrihost at all. Any ideas?

Code:
|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                             192.168.1.1 -    0 | 1802 | 1802 |    0 |    0 |    6 |    0 |
|             core.as-01.cp1.za.ws.net.za -    0 | 1802 | 1802 |    0 |    0 |   13 |    0 |
|             core.cr-01.cp1.za.ws.net.za -    1 | 1775 | 1768 |    0 |    1 |   78 |    1 |
|        core.pe-xe-ip01.cp1.za.ws.net.za -    1 | 1771 | 1763 |    0 |    1 |   76 |    1 |
|           165-69-148-197.as37497.za.net -    1 | 1783 | 1778 |    0 |    1 |   77 |    1 |
|           162-68-148-197.as37497.za.net -    1 | 1755 | 1743 |  141 |  142 |  230 |  142 |
|            54-71-148-197.as37497.za.net -    1 | 1751 | 1738 |  141 |  143 |  234 |  141 |
|            33-66-148-197.as37497.za.net -    2 | 1731 | 1713 |  141 |  141 |  217 |  142 |
|ge-0-1-0-68.peering1.the.lon1.coreix.net -    1 | 1768 | 1759 |  141 |  142 |  228 |  142 |
|        xe-5-1.edge4.enf.lon5.coreix.net -    1 | 1763 | 1753 |  142 |  144 |  230 |  142 |
|        85.13.221.178.reverse.coreix.net -    1 | 1755 | 1743 |  141 |  142 |  281 |  142 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
You cancelled with Afrihost :(
 

Seeyou

Expert Member
Joined
May 1, 2007
Messages
1,751
@websquadza

Seeing some weird jitter and occasional crazy latency to Cloudflare services in CPT. I'd expect the average latency to be a flat 1ms:

Code:
|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                             192.168.1.1 -    0 |   60 |   60 |    0 |    0 |    0 |    0 |
|             core.as-01.cp1.za.ws.net.za -    0 |   60 |   60 |    0 |    0 |    2 |    1 |
|             core.cr-01.cp1.za.ws.net.za -    0 |   60 |   60 |    0 |    0 |    1 |    1 |
|        core.pe-xe-ix01.cp1.za.ws.net.za -    0 |   60 |   60 |    0 |    4 |   61 |    1 |
|                 cloudflare.ixp.capetown -    0 |   60 |   60 |    1 |    7 |   78 |    2 |
|                           104.20.10.169 -    0 |   60 |   60 |    1 |    4 |   62 |    1 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
 

websquadza

WebSquad
Joined
Mar 26, 2018
Messages
1,277
@websquadza

Seeing some weird jitter and occasional crazy latency to Cloudflare services in CPT. I'd expect the average latency to be a flat 1ms:

Code:
|------------------------------------------------------------------------------------------|
|                                      WinMTR statistics                                   |
|                       Host              -   %  | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
|                             192.168.1.1 -    0 |   60 |   60 |    0 |    0 |    0 |    0 |
|             core.as-01.cp1.za.ws.net.za -    0 |   60 |   60 |    0 |    0 |    2 |    1 |
|             core.cr-01.cp1.za.ws.net.za -    0 |   60 |   60 |    0 |    0 |    1 |    1 |
|        core.pe-xe-ix01.cp1.za.ws.net.za -    0 |   60 |   60 |    0 |    4 |   61 |    1 |
|                 cloudflare.ixp.capetown -    0 |   60 |   60 |    1 |    7 |   78 |    2 |
|                           104.20.10.169 -    0 |   60 |   60 |    1 |    4 |   62 |    1 |
|________________________________________________|______|______|______|______|______|______|
   WinMTR v0.92 GPL V2 by Appnor MSP - Fully Managed Hosting & Cloud Provider
Remember, ICMP isn't always prioritised on routers - they'd much rather be pushing packets than replying to every ICMP request - especially if there are a large number of ICMP requests being received on the hardware at once (think of the hundreds/thousands of constant monitors on these devices - everyone monitors quad1, quad8 etc) - at which point they rate limit or deprioritise even more- this even happens in large LAN environments. No issues along these routes.

Furthermore, networks sometimes load balance these requests - run a test to 8.8.8.8 for long enough and you'll see additional IPs pop up as requests are routed elsewhere. Trace above is clear to our peering edge, and an average of 4ms on 60 requests with one or two outliers is good. No issues on NAP, levels are good etc - so this points to deprioritisation of ICMP.
 

Seeyou

Expert Member
Joined
May 1, 2007
Messages
1,751
Welcome to Web sQuad, thanks for the feedback. We're looking into this one. Looks less like packet loss (which would make anything with a high latency unusably slow) and more like a packet drop every now and again (and I identify between the two as the latter doesn't seem to be causing performance headaches, but is noticeable in game) - we're working to see where this is coming from..

With regards to your downloads running at 19.8 vs 21 MB, that sounds more like the processing head on PPPOE - we provision all our PPPOE accounts with 10% extra (so a 200 Mbps profile is actually 225 280 kbps), but not all the line providers do the same. It seems Octotel hard limit to 200 Mbps (and I'm not sure whether they limit speeds by dropping packets or queuing them), which means packet overhead and retransmits on dropped packets at linewire will bring that speed down about 5-10%. Will send you some settings for the ER-X.
Seems more pronounced today. Getting a little frustrating in game. Latency is great but the packet loss / dropped packets are quite noticeable. Tried setting MTU to 1480 but no difference.
 

Seeyou

Expert Member
Joined
May 1, 2007
Messages
1,751
This is crazy impressive:


If we can just sort out this bizarre packet behaviour I'll be extremely happy.
 

websquadza

WebSquad
Joined
Mar 26, 2018
Messages
1,277
This is crazy impressive:


If we can just sort out this bizarre packet behaviour I'll be extremely happy.
Good News: We've picked up which device may be dropping packets randomly - Bad News: unfortunately it's not a hitless repair. We'll be sending out comms for a scheduled maintenance procedure in CPT during this week. The process will take about 5 minutes and will be done way out of hours to minimise impact.
 

Hingisp

Active Member
Joined
Jul 4, 2019
Messages
88
Good News: We've picked up which device may be dropping packets randomly - Bad News: unfortunately it's not a hitless repair. We'll be sending out comms for a scheduled maintenance procedure in CPT during this week. The process will take about 5 minutes and will be done way out of hours to minimise impact.
Would this have any effect on the your Durban based clients? I have submitted a ticket on 30/05 and yet to have any response with my slow speed results and also dropped packets.
 

Kilerhotdog

Well-Known Member
Joined
Dec 28, 2011
Messages
127
@websquadza, You mentioned something about Frogfoot and Openserve possibly being available this month. Do you know what the pricing for the 100 and 200mbps packages will cost for frogfoot or openserve?

If not I will wait :)
 

ijacobs3

Expert Member
Joined
Oct 15, 2009
Messages
2,690
This is crazy impressive:


If we can just sort out this bizarre packet behaviour I'll be extremely happy.

Im very confused, i must be doing something very wrong as i dont get anywhere near that on a 200/200 connection, locally to jhb , yes, durban , cape town, not even close, and the UK is even worse


Afrihost JHb


Cape Town



London - Coreix
 

websquadza

WebSquad
Joined
Mar 26, 2018
Messages
1,277
Im very confused, i must be doing something very wrong as i dont get anywhere near that on a 200/200 connection, locally to jhb , yes, durban , cape town, not even close, and the UK is even worse


Afrihost JHb


Cape Town



London - Coreix
Degradation of speed on a 19ms link tells us there's packet loss on your last mile. Should reach CPT at 200 Mbps with ease. Have you run tests to eliminate local packet loss? LAN cable to router, LAN cable to CPE etc? iperf results?
 

ijacobs3

Expert Member
Joined
Oct 15, 2009
Messages
2,690
Degradation of speed on a 19ms link tells us there's packet loss on your last mile. Should reach CPT at 200 Mbps with ease. Have you run tests to eliminate local packet loss? LAN cable to router, LAN cable to CPE etc? iperf results?
fresh cable, directly into rebooted CPE

Capetown ( using Vodacom as a site to test to)


Capetown ( using vox as the site to test to)


Coreix London

1591202349668.png


If it were something on my direct side ( cables, etc) would i not then get the same result to jhb as well? my local lan cable doesnt know the difference from capetown to jhb to london

using https://perfops.net/mtr-from-cape-town to my IP


Cape Town
Start: Wed Jun 3 18:34:00 2020
Loss% Snt Last Avg Best Wrst StDev
1.|-- 197.85.7.1 0.0% 2 0.8 0.8 0.8 0.8 0.0
2.|-- 197.85.5.245 0.0% 2 2.4 2.4 2.4 2.5 0.0
3.|-- 196.41.144.51 0.0% 2 2.2 2.4 2.2 2.6 0.0
4.|-- 197.84.120.65 0.0% 2 8.8 6.9 5.1 8.8 2.4
5.|-- 197.84.4.243 0.0% 2 3.9 3.8 3.7 3.9 0.0
6.|-- 197.84.8.226 0.0% 2 3.8 3.9 3.8 4.1 0.0
7.|-- 196.60.70.137 0.0% 2 6.2 4.7 3.2 6.2 2.0
8.|-- 160.119.224.21 0.0% 2 21.2 21.4 21.2 21.5 0.0
9.|-- 160.119.224.30 0.0% 2 21.1 21.3 21.1 21.6 0.0
10.|-- 160.119.224.78 0.0% 2 24.2 23.2 22.1 24.2 1.4
11.|-- 160.119.224.102 0.0% 2 22.9 22.6 22.2 22.9 0.0
12.|-- 160.119.232.145 0.0% 2 21.7 22.0 21.7 22.3 0.0

not sure if this helps or means anything?

and from jhb to cpt
1591202217860.png
 

websquadza

WebSquad
Joined
Mar 26, 2018
Messages
1,277
fresh cable, directly into rebooted CPE

Capetown ( using Vodacom as a site to test to)


Capetown ( using vox as the site to test to)


Coreix London

View attachment 850597


If it were something on my direct side ( cables, etc) would i not then get the same result to jhb as well? my local lan cable doesnt know the difference from capetown to jhb to london

using https://perfops.net/mtr-from-cape-town to my IP


Cape Town
Start: Wed Jun 3 18:34:00 2020
Loss% Snt Last Avg Best Wrst StDev
1.|-- 197.85.7.1 0.0% 2 0.8 0.8 0.8 0.8 0.0
2.|-- 197.85.5.245 0.0% 2 2.4 2.4 2.4 2.5 0.0
3.|-- 196.41.144.51 0.0% 2 2.2 2.4 2.2 2.6 0.0
4.|-- 197.84.120.65 0.0% 2 8.8 6.9 5.1 8.8 2.4
5.|-- 197.84.4.243 0.0% 2 3.9 3.8 3.7 3.9 0.0
6.|-- 197.84.8.226 0.0% 2 3.8 3.9 3.8 4.1 0.0
7.|-- 196.60.70.137 0.0% 2 6.2 4.7 3.2 6.2 2.0
8.|-- 160.119.224.21 0.0% 2 21.2 21.4 21.2 21.5 0.0
9.|-- 160.119.224.30 0.0% 2 21.1 21.3 21.1 21.6 0.0
10.|-- 160.119.224.78 0.0% 2 24.2 23.2 22.1 24.2 1.4
11.|-- 160.119.224.102 0.0% 2 22.9 22.6 22.2 22.9 0.0
12.|-- 160.119.232.145 0.0% 2 21.7 22.0 21.7 22.3 0.0

not sure if this helps or means anything?

and from jhb to cpt
View attachment 850595
Not that easy, if something your side is causing TCP retransmits, your speed will be inversely proportional to the distance (latency). Basically, the higher the latency, the lower your speed. TCP always performs slower with higher latency, that's expected. Add packet loss or retransmits, and you have an exponential degradation of the line (like we're seeing here). Think of it this way - if every TCP packet needs an acknowledgement - at <3ms, the retransmits are barely visible. At 20ms, they start becoming noticeable and at 150ms- good luck.

This can be caused by TCP issues in your environment (even faulty drivers), a bad NIC, a bad router, a bad line (though you don't show immediate loss on your MTRs - there may still be an issue (which is why iperfs exist).

Getting FNOs to admit issues is like pulling a needle through your eye. So first start by eliminating everything local: NIC and PC: test another PC. Router: bypass and test to CPE. Swap cables etc.

It's not present on the rest of the network, so it's not AEX or NNI.

JHB NAP Iperf: iperf3 -4 -V -t 10 -O 3 -u -b 180M -l 1440 -c iperf.jb1.napafrica.net
CPT NAP Iperf: iperf3 -4 -V -t 10 -O 3 -u -b 180M -l 1440 -c iperf.ct1.napafrica.net

It's also worth whipping out wireshark and monitoring your lan interface for any TCP issues (usually quite quick and easy to pick up issues that way).
 
Top