Cool Ideas / Zoom Fibre – Single TCP upload ~20 Mbps, 16 streams ~200 Mbps

[Prineshan_Amichand]

Hi @CoolEscalator, @PBCool

I'm hoping someone can assist with a strange upload performance issue that Cool Ideas and Zoom Fibre have been investigating.

I have a symmetrical fibre connection, and the connection is clearly capable of approximately 200 Mbps upstream. However, single-stream TCP upload performance is consistently extremely low.

Testing against the Cool Ideas Johannesburg iPerf3 server (ter-jhb-iperf-1.cisp.co.za, port 17001) has produced the following:

Windows

• 1 TCP stream upload: ~14–25 Mbps
• 16 parallel TCP streams upload: ~193–200 Mbps
• 1 TCP stream download: ~200 Mbps

Today I wanted to completely eliminate Windows, the Killer Windows driver, VPN/filter drivers and other Windows software as variables.

I therefore booted the same Dell laptop from a Zorin Linux Live USB, disabled Wi-Fi and confirmed the Ethernet connection at 1 Gbps / Full Duplex.

Without changing the network connection, I repeated the Cool Ideas iPerf3 tests:

Zorin Linux

• 1 TCP stream upload: 22.9 Mbps
• 16 parallel TCP streams upload: ~200 Mbps
• 1 TCP stream download: ~200 Mbps

So essentially the same behaviour is reproduced under two completely different operating systems and TCP stacks.

Other troubleshooting already completed includes:

• Testing directly via the ONT/CPE, bypassing my normal router.
• Zoom Fibre technician onsite testing.
• Physical Ethernet links confirmed at 1 Gbps/full duplex with no CRC/FCS errors observed on the normal router WAN.
• PPPoE/path MTU tested at 1480. A 1452-byte ICMP payload with DF passes; 1453 requires fragmentation.
• Temporarily reducing the Windows Ethernet MTU to 1480 made no material difference to the single-stream upload result.
• Multiple Speedtest servers can achieve approximately 200 Mbps upload, while the Cool Ideas Speedtest server has repeatedly been around ~85 Mbps upload.
• Zoom's checks so far have not identified a physical fibre fault.

The part I'm struggling to explain is this:

Why can one upstream TCP flow only achieve approximately 20 Mbps, while 16 parallel upstream TCP flows collectively achieve the full ~200 Mbps — yet one downstream TCP flow can achieve ~200 Mbps?

At this stage the behaviour has survived different routers/direct ONT testing and now two completely different operating systems.

Cool Ideas and Zoom Fibre are still investigating the matter. I'm not trying to point fingers at either party — I'm hoping someone here, particularly the Cool Ideas network guys, may recognise this pattern or suggest where we should be looking next.

@CoolEscalator — I can PM you my existing Cool Ideas ticket/reference and any additional iPerf results required.

 

[PBCool]

Hi, can you post your Iperf command that you're testing with?

It is typically a packet loss issue for single thread performance.

 

[Prineshan_Amichand]

Hi, can you post your Iperf command that you're testing with?

It is typically a packet loss issue for single thread performance.

Hi, sure.

Single-stream upload: iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -t 30

Single-stream download: iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -t 30 -R

16-stream upload: iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -t 30 -P 16

I repeated the same tests under Windows and Zorin Linux.

Single-stream upload is around 20–23 Mbps, single-stream download around 200 Mbps, while 16 parallel upload streams reach around 200 Mbps.

Happy to run any additional tests if you want to check specifically for packet loss/retransmissions.

 

[PBCool]

Hi, sure.

Single-stream upload: iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -t 30

Single-stream download: iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -t 30 -R

16-stream upload: iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -t 30 -P 16

I repeated the same tests under Windows and Zorin Linux.

Single-stream upload is around 20–23 Mbps, single-stream download around 200 Mbps, while 16 parallel upload streams reach around 200 Mbps.

Happy to run any additional tests if you want to check specifically for packet loss/retransmissions.

Thanks and you get the same results with doing an Okla speedtest?

 

[PBCool]

To test loss on the line with iperf please see this thread:

Thread 'How to do iPerf tests.'

Jun 9, 2021

Sometimes it's required to determine whether packet loss is the culprit for slow up or download speeds. Packet loss can cause TCP retransmissions to occur, because the data has to be retransmitted which will cause TCP to start with a slower up/download window.

This will affect performance in either the upload, or the download direction, and is typically seen on a bandwidth graph as "valleys and peaks" in a download or upload graph, as well as lower than expected speedtests etc.

(example of a graph showing the valley and peak pattern)

In this case, an iPerf test to one of...

• TheRoDent
• Replies: 20
• Forum: Cool Ideas Support Forum

• 

 

[Prineshan_Amichand]

Thanks and you get the same results with doing an Okla speedtest?

Not exactly. Ookla results vary depending on the selected server.

Using the Cool Ideas Speedtest server I consistently get around 200 Mbps download but only ~85 Mbps upload.

However, using Cell C, Neurotel and 123NET I can get approximately 200 Mbps upload.

This is what led me to test with iPerf3. With the Cool Ideas iPerf server, a single upload stream is only around 20 Mbps, but increasing to 16 parallel streams allows it to reach approximately 200 Mbps.

So the connection can definitely deliver the full upstream bandwidth; the unusual part seems to be the single-stream TCP performance.

 

[PBCool]

Not exactly. Ookla results vary depending on the selected server.

Using the Cool Ideas Speedtest server I consistently get around 200 Mbps download but only ~85 Mbps upload.

However, using Cell C, Neurotel and 123NET I can get approximately 200 Mbps upload.

This is what led me to test with iPerf3. With the Cool Ideas iPerf server, a single upload stream is only around 20 Mbps, but increasing to 16 parallel streams allows it to reach approximately 200 Mbps.

So the connection can definitely deliver the full upstream bandwidth; the unusual part seems to be the single-stream TCP performance.

Well on Speedtest you should be able to select a single thread test? What results do you get there?

But what you are describing is typically some sort of loss.

 

[Prineshan_Amichand]

Well on Speedtest you should be able to select a single thread test? What results do you get there?

But what you are describing is typically some sort of loss.

You were spot on regarding testing Ookla in Single Connection mode.

I’ve just run both tests against the same Cool Ideas Johannesburg server:

Single Connection: 164.74 Mbps down / 21.67 Mbps up

Multi Connection: 201.09 Mbps down / 106.34 Mbps up

Interestingly, the 21.67 Mbps single-connection Ookla upload is almost identical to the 22.9 Mbps single-stream iPerf3 upload I measured under Zorin Linux.

So the single vs multi behaviour is definitely reproducible outside of iPerf as well.

I’ll attach screenshots of both results.

If you suspect packet loss, I'm happy to run whatever test you recommend to try to identify where it's occurring.

 

[PBCool]

You were spot on regarding testing Ookla in Single Connection mode.

I’ve just run both tests against the same Cool Ideas Johannesburg server:

Single Connection: 164.74 Mbps down / 21.67 Mbps up

Multi Connection: 201.09 Mbps down / 106.34 Mbps up

Interestingly, the 21.67 Mbps single-connection Ookla upload is almost identical to the 22.9 Mbps single-stream iPerf3 upload I measured under Zorin Linux.

So the single vs multi behaviour is definitely reproducible outside of iPerf as well.

I’ll attach screenshots of both results.

If you suspect packet loss, I'm happy to run whatever test you recommend to try to identify where it's occurring.

Well its almost always FNO related, so those iperfs should show the % of loss.

 

[Prineshan_Amichand]

Well its almost always FNO related, so those iperfs should show the % of loss.

Did some further testing this morning based on the packet-loss suggestion.

I used the Cool Ideas iPerf server with --get-server-output so that I could get the actual UDP loss reported by the receiving server.

Normal setup – PC → Router → ONT

10 Mbps: 0.31% loss
50 Mbps: 0.29% loss
100 Mbps: 0.34% loss
190 Mbps: 0.31% loss

I then completely bypassed the Router and connected the same PC directly to the ONT using Windows PPPoE.

Direct ONT – PC → ONT → PPPoE

10 Mbps: 0.34% loss
50 Mbps: 0.37% loss
100 Mbps: 0.28% loss
190 Mbps: 0.32% loss

So the ~0.3% UDP loss remains almost unchanged with the router and internal network completely removed.

While still directly connected to the ONT, I also repeated the TCP tests against the same Cool Ideas iPerf server:

Single TCP upload: 22.4 Mbps
Single TCP download: 206 Mbps

The download server also reported 17,895 retransmissions during the 30-second test.

This also matches the earlier behaviour where a single upstream TCP stream is around 14–23 Mbps, while 16 parallel upstream streams can reach ~193–200 Mbps.

So at this stage we have:

• ~0.3% UDP loss even when directly connected to the ONT
• Single TCP upload ~22 Mbps
• Single TCP download ~206 Mbps
• Multi-stream TCP upload capable of ~200 Mbps

I think this now rules out my internal LAN as the source of the measured packet loss.

I already have an open ticket with Cool Ideas for this issue, which has been escalated and is being investigated with Zoom Fibre. Zoom has also attended onsite, and testing has been ongoing for some time.

@CoolEscalator , @PBCool — does the ~0.3% loss seen directly through the ONT explain or contribute to the single-stream TCP behaviour we're seeing, and is there anything else you would like me to test from my side?

 

[PBCool]

Did some further testing this morning based on the packet-loss suggestion.

I used the Cool Ideas iPerf server with --get-server-output so that I could get the actual UDP loss reported by the receiving server.

Normal setup – PC → Router → ONT

10 Mbps: 0.31% loss
50 Mbps: 0.29% loss
100 Mbps: 0.34% loss
190 Mbps: 0.31% loss

I then completely bypassed the Router and connected the same PC directly to the ONT using Windows PPPoE.

Direct ONT – PC → ONT → PPPoE

10 Mbps: 0.34% loss
50 Mbps: 0.37% loss
100 Mbps: 0.28% loss
190 Mbps: 0.32% loss

So the ~0.3% UDP loss remains almost unchanged with the router and internal network completely removed.

While still directly connected to the ONT, I also repeated the TCP tests against the same Cool Ideas iPerf server:

Single TCP upload: 22.4 Mbps
Single TCP download: 206 Mbps

The download server also reported 17,895 retransmissions during the 30-second test.

This also matches the earlier behaviour where a single upstream TCP stream is around 14–23 Mbps, while 16 parallel upstream streams can reach ~193–200 Mbps.

So at this stage we have:

• ~0.3% UDP loss even when directly connected to the ONT
• Single TCP upload ~22 Mbps
• Single TCP download ~206 Mbps
• Multi-stream TCP upload capable of ~200 Mbps

I think this now rules out my internal LAN as the source of the measured packet loss.

I already have an open ticket with Cool Ideas for this issue, which has been escalated and is being investigated with Zoom Fibre. Zoom has also attended onsite, and testing has been ongoing for some time.

@CoolEscalator , @PBCool — does the ~0.3% loss seen directly through the ONT explain or contribute to the single-stream TCP behaviour we're seeing, and is there anything else you would like me to test from my side?

Hi, please post the iperf commands you ran. You can run upload and download tests

 

[Prineshan_Amichand]

Hi, please post the iperf commands you ran. You can run upload and download tests

Sure. All tests were against the Cool Ideas Johannesburg iPerf server.

Single-stream TCP upload:

iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -t 30
Result direct to ONT: 22.5 Mbps sender / 22.4 Mbps receiver

Single-stream TCP download:

iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -t 30 -R
Result direct to ONT: 206 Mbps sender / 206 Mbps receiver

For the UDP packet-loss tests I used --get-server-output so I could see the loss reported by the receiving Cool Ideas server.

UDP uploads:

iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -u -b 10M -O 2 -t 10 --get-server-output
iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -u -b 50M -O 2 -t 10 --get-server-output
iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -u -b 100M -O 2 -t 10 --get-server-output
iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -u -b 190M -O 2 -t 10 --get-server-output
I ran those UDP tests both through the Router and again with the PC directly connected to the ONT using PPPoE.

If you'd like me to run the equivalent UDP download (-R) tests as well, let me know what rates you would like tested and I'll post the results.

 

[CoolEscalator]

Sure. All tests were against the Cool Ideas Johannesburg iPerf server.

Single-stream TCP upload:

iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -t 30
Result direct to ONT: 22.5 Mbps sender / 22.4 Mbps receiver

Single-stream TCP download:

iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -t 30 -R
Result direct to ONT: 206 Mbps sender / 206 Mbps receiver

For the UDP packet-loss tests I used --get-server-output so I could see the loss reported by the receiving Cool Ideas server.

UDP uploads:

iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -u -b 10M -O 2 -t 10 --get-server-output
iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -u -b 50M -O 2 -t 10 --get-server-output
iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -u -b 100M -O 2 -t 10 --get-server-output
iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -u -b 190M -O 2 -t 10 --get-server-output
I ran those UDP tests both through the Router and again with the PC directly connected to the ONT using PPPoE.

If you'd like me to run the equivalent UDP download (-R) tests as well, let me know what rates you would like tested and I'll post the results.

HI there, so i tested from the BNG that you connect on, to our Iperf server in JHB, and could not see a Single thread issue
I shared my findings with the escallations Analyst working on your ticket.

The main point I want to focus on , is before the outage you had , you had no such issues, after the Zoom outage, these speed issues started to occur. I believe you also mentioned on your ticket, that other customers in the area on different ISP's also started to experience speed related issues since after the outage was resolved.
It is strange that to some destinations you get bad uploads, and to others not. However the only time I have seen a similar issue to this, was when the FNO had a bad switch, and after moving the customer to a different switch, was the issue resolved.
I have asked our Escalaltions Analyst to recommend this to Zoom

TCP Upload (500M-1000M-2000M)

iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -b 500M
Connecting to host ter-jhb-iperf-1.cisp.co.za, port 17001
[  5] local 2c0f:f9a8::1:27:31:b2 port 34918 connected to 2c0f:f9a8:1::1:2 port 17001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  59.6 MBytes   500 Mbits/sec    0   3.66 MBytes
[  5]   1.00-2.00   sec  59.6 MBytes   500 Mbits/sec    0   3.66 MBytes
[  5]   2.00-3.00   sec  59.6 MBytes   500 Mbits/sec    0   3.66 MBytes
[  5]   3.00-4.00   sec  59.5 MBytes   499 Mbits/sec    0   3.66 MBytes
[  5]   4.00-5.00   sec  59.6 MBytes   500 Mbits/sec    0   3.66 MBytes
[  5]   5.00-6.00   sec  59.6 MBytes   500 Mbits/sec    0   3.66 MBytes
[  5]   6.00-7.00   sec  59.6 MBytes   500 Mbits/sec    0   3.66 MBytes
[  5]   7.00-8.00   sec  59.6 MBytes   500 Mbits/sec    0   3.66 MBytes
[  5]   8.00-9.00   sec  59.6 MBytes   500 Mbits/sec    0   3.66 MBytes
[  5]   9.00-10.00  sec  59.5 MBytes   499 Mbits/sec    0   3.66 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   596 MBytes   500 Mbits/sec    0             sender
[  5]   0.00-10.02  sec   596 MBytes   499 Mbits/sec                  receiver

iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -b 1000M
Connecting to host ter-jhb-iperf-1.cisp.co.za, port 17001
[  5] local 2c0f:f9a8::1:27:31:b2 port 43044 connected to 2c0f:f9a8:1::1:2 port 17001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   119 MBytes   999 Mbits/sec   19   3.19 MBytes
[  5]   1.00-2.00   sec   119 MBytes  1.00 Gbits/sec    0   3.19 MBytes
[  5]   2.00-3.00   sec   119 MBytes  1.00 Gbits/sec    0   3.19 MBytes
[  5]   3.00-4.00   sec   119 MBytes   999 Mbits/sec    0   3.19 MBytes
[  5]   4.00-5.00   sec   119 MBytes  1.00 Gbits/sec    0   3.19 MBytes
[  5]   5.00-6.00   sec   119 MBytes  1.00 Gbits/sec    0   3.19 MBytes
[  5]   6.00-7.00   sec   119 MBytes   999 Mbits/sec    0   3.19 MBytes
[  5]   7.00-8.00   sec   119 MBytes  1.00 Gbits/sec    0   3.19 MBytes
[  5]   8.00-9.00   sec   119 MBytes  1.00 Gbits/sec    0   3.19 MBytes
[  5]   9.00-10.00  sec   119 MBytes   999 Mbits/sec    0   3.19 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.16 GBytes  1000 Mbits/sec   19             sender
[  5]   0.00-10.02  sec  1.16 GBytes   997 Mbits/sec                  receiver


@TRV-DB1-DC1-I06-BNG2%default:~$ iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -b 2000M
Connecting to host ter-jhb-iperf-1.cisp.co.za, port 17001
[  5] local 2c0f:f9a8::1:27:31:b2 port 42384 connected to 2c0f:f9a8:1::1:2 port 17001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   209 MBytes  1.75 Gbits/sec   11   3.17 MBytes
[  5]   1.00-2.00   sec   220 MBytes  1.85 Gbits/sec    0   3.17 MBytes
[  5]   2.00-3.00   sec   234 MBytes  1.97 Gbits/sec    0   3.17 MBytes
[  5]   3.00-4.00   sec   257 MBytes  2.15 Gbits/sec    0   3.17 MBytes
[  5]   4.00-5.00   sec   272 MBytes  2.28 Gbits/sec    0   3.17 MBytes
[  5]   5.00-6.00   sec   238 MBytes  2.00 Gbits/sec    0   3.17 MBytes
[  5]   6.00-7.00   sec   238 MBytes  2.00 Gbits/sec    0   3.17 MBytes
[  5]   7.00-8.00   sec   238 MBytes  2.00 Gbits/sec    0   3.17 MBytes
[  5]   8.00-9.00   sec   238 MBytes  2.00 Gbits/sec    0   3.17 MBytes
[  5]   9.00-10.00  sec   238 MBytes  2.00 Gbits/sec    0   3.17 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.33 GBytes  2.00 Gbits/sec   11             sender
[  5]   0.00-10.03  sec  2.33 GBytes  1.99 Gbits/sec                  receiver

TCP Download (500M-1000M-2000M)

iperf3 -R -c ter-jhb-iperf-1.cisp.co.za -p 17001 -b 500M
Connecting to host ter-jhb-iperf-1.cisp.co.za, port 17001
Reverse mode, remote host ter-jhb-iperf-1.cisp.co.za is sending
[  5] local 2c0f:f9a8::1:27:31:b2 port 49338 connected to 2c0f:f9a8:1::1:2 port 17001
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  60.8 MBytes   510 Mbits/sec
[  5]   1.00-2.00   sec  59.6 MBytes   500 Mbits/sec
[  5]   2.00-3.00   sec  59.5 MBytes   499 Mbits/sec
[  5]   3.00-4.00   sec  59.6 MBytes   500 Mbits/sec
[  5]   4.00-5.00   sec  59.6 MBytes   500 Mbits/sec
[  5]   5.00-6.00   sec  59.6 MBytes   500 Mbits/sec
[  5]   6.00-7.00   sec  59.6 MBytes   500 Mbits/sec
[  5]   7.00-8.00   sec  59.7 MBytes   500 Mbits/sec
[  5]   8.00-9.00   sec  59.6 MBytes   500 Mbits/sec
[  5]   9.00-10.00  sec  59.5 MBytes   499 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.03  sec   598 MBytes   500 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   597 MBytes   501 Mbits/sec                  receiver

iperf3 -R -c ter-jhb-iperf-1.cisp.co.za -p 17001 -b 1000M
Connecting to host ter-jhb-iperf-1.cisp.co.za, port 17001
Reverse mode, remote host ter-jhb-iperf-1.cisp.co.za is sending
[  5] local 2c0f:f9a8::1:27:31:b2 port 43430 connected to 2c0f:f9a8:1::1:2 port 17001
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   121 MBytes  1.02 Gbits/sec
[  5]   1.00-2.00   sec   119 MBytes  1.00 Gbits/sec
[  5]   2.00-3.00   sec   119 MBytes   998 Mbits/sec
[  5]   3.00-4.00   sec   119 MBytes  1.00 Gbits/sec
[  5]   4.00-5.00   sec   119 MBytes  1000 Mbits/sec
[  5]   5.00-6.00   sec   119 MBytes   999 Mbits/sec
[  5]   6.00-7.00   sec   119 MBytes  1.00 Gbits/sec
[  5]   7.00-8.00   sec   119 MBytes  1.00 Gbits/sec
[  5]   8.00-9.00   sec   119 MBytes  1.00 Gbits/sec
[  5]   9.00-10.00  sec   119 MBytes  1.00 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec  1.17 GBytes  1.00 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  1.17 GBytes  1.00 Gbits/sec                  receiver


iperf3 -R -c ter-jhb-iperf-1.cisp.co.za -p 17001 -b 2000M
Connecting to host ter-jhb-iperf-1.cisp.co.za, port 17001
Reverse mode, remote host ter-jhb-iperf-1.cisp.co.za is sending
[  5] local 2c0f:f9a8::1:27:31:b2 port 52652 connected to 2c0f:f9a8:1::1:2 port 17001
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   241 MBytes  2.02 Gbits/sec
[  5]   1.00-2.00   sec   238 MBytes  2.00 Gbits/sec
[  5]   2.00-3.00   sec   238 MBytes  2.00 Gbits/sec
[  5]   3.00-4.00   sec   239 MBytes  2.00 Gbits/sec
[  5]   4.00-5.00   sec   238 MBytes  2.00 Gbits/sec
[  5]   5.00-6.00   sec   238 MBytes  2.00 Gbits/sec
[  5]   6.00-7.00   sec   238 MBytes  2.00 Gbits/sec
[  5]   7.00-8.00   sec   239 MBytes  2.00 Gbits/sec
[  5]   8.00-9.00   sec   238 MBytes  2.00 Gbits/sec
[  5]   9.00-10.00  sec   238 MBytes  2.00 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec  2.33 GBytes  2.00 Gbits/sec    4             sender
[  5]   0.00-10.00  sec  2.33 GBytes  2.00 Gbits/sec                  receiver

 

[Prineshan_Amichand]

HI there, so i tested from the BNG that you connect on, to our Iperf server in JHB, and could not see a Single thread issue
I shared my findings with the escallations Analyst working on your ticket.

The main point I want to focus on , is before the outage you had , you had no such issues, after the Zoom outage, these speed issues started to occur. I believe you also mentioned on your ticket, that other customers in the area on different ISP's also started to experience speed related issues since after the outage was resolved.
It is strange that to some destinations you get bad uploads, and to others not. However the only time I have seen a similar issue to this, was when the FNO had a bad switch, and after moving the customer to a different switch, was the issue resolved.
I have asked our Escalaltions Analyst to recommend this to Zoom

TCP Upload (500M-1000M-2000M)

iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -b 500M
Connecting to host ter-jhb-iperf-1.cisp.co.za, port 17001
[  5] local 2c0f:f9a8::1:27:31:b2 port 34918 connected to 2c0f:f9a8:1::1:2 port 17001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  59.6 MBytes   500 Mbits/sec    0   3.66 MBytes
[  5]   1.00-2.00   sec  59.6 MBytes   500 Mbits/sec    0   3.66 MBytes
[  5]   2.00-3.00   sec  59.6 MBytes   500 Mbits/sec    0   3.66 MBytes
[  5]   3.00-4.00   sec  59.5 MBytes   499 Mbits/sec    0   3.66 MBytes
[  5]   4.00-5.00   sec  59.6 MBytes   500 Mbits/sec    0   3.66 MBytes
[  5]   5.00-6.00   sec  59.6 MBytes   500 Mbits/sec    0   3.66 MBytes
[  5]   6.00-7.00   sec  59.6 MBytes   500 Mbits/sec    0   3.66 MBytes
[  5]   7.00-8.00   sec  59.6 MBytes   500 Mbits/sec    0   3.66 MBytes
[  5]   8.00-9.00   sec  59.6 MBytes   500 Mbits/sec    0   3.66 MBytes
[  5]   9.00-10.00  sec  59.5 MBytes   499 Mbits/sec    0   3.66 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   596 MBytes   500 Mbits/sec    0             sender
[  5]   0.00-10.02  sec   596 MBytes   499 Mbits/sec                  receiver

iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -b 1000M
Connecting to host ter-jhb-iperf-1.cisp.co.za, port 17001
[  5] local 2c0f:f9a8::1:27:31:b2 port 43044 connected to 2c0f:f9a8:1::1:2 port 17001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   119 MBytes   999 Mbits/sec   19   3.19 MBytes
[  5]   1.00-2.00   sec   119 MBytes  1.00 Gbits/sec    0   3.19 MBytes
[  5]   2.00-3.00   sec   119 MBytes  1.00 Gbits/sec    0   3.19 MBytes
[  5]   3.00-4.00   sec   119 MBytes   999 Mbits/sec    0   3.19 MBytes
[  5]   4.00-5.00   sec   119 MBytes  1.00 Gbits/sec    0   3.19 MBytes
[  5]   5.00-6.00   sec   119 MBytes  1.00 Gbits/sec    0   3.19 MBytes
[  5]   6.00-7.00   sec   119 MBytes   999 Mbits/sec    0   3.19 MBytes
[  5]   7.00-8.00   sec   119 MBytes  1.00 Gbits/sec    0   3.19 MBytes
[  5]   8.00-9.00   sec   119 MBytes  1.00 Gbits/sec    0   3.19 MBytes
[  5]   9.00-10.00  sec   119 MBytes   999 Mbits/sec    0   3.19 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.16 GBytes  1000 Mbits/sec   19             sender
[  5]   0.00-10.02  sec  1.16 GBytes   997 Mbits/sec                  receiver


@TRV-DB1-DC1-I06-BNG2%default:~$ iperf3 -c ter-jhb-iperf-1.cisp.co.za -p 17001 -b 2000M
Connecting to host ter-jhb-iperf-1.cisp.co.za, port 17001
[  5] local 2c0f:f9a8::1:27:31:b2 port 42384 connected to 2c0f:f9a8:1::1:2 port 17001
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   209 MBytes  1.75 Gbits/sec   11   3.17 MBytes
[  5]   1.00-2.00   sec   220 MBytes  1.85 Gbits/sec    0   3.17 MBytes
[  5]   2.00-3.00   sec   234 MBytes  1.97 Gbits/sec    0   3.17 MBytes
[  5]   3.00-4.00   sec   257 MBytes  2.15 Gbits/sec    0   3.17 MBytes
[  5]   4.00-5.00   sec   272 MBytes  2.28 Gbits/sec    0   3.17 MBytes
[  5]   5.00-6.00   sec   238 MBytes  2.00 Gbits/sec    0   3.17 MBytes
[  5]   6.00-7.00   sec   238 MBytes  2.00 Gbits/sec    0   3.17 MBytes
[  5]   7.00-8.00   sec   238 MBytes  2.00 Gbits/sec    0   3.17 MBytes
[  5]   8.00-9.00   sec   238 MBytes  2.00 Gbits/sec    0   3.17 MBytes
[  5]   9.00-10.00  sec   238 MBytes  2.00 Gbits/sec    0   3.17 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  2.33 GBytes  2.00 Gbits/sec   11             sender
[  5]   0.00-10.03  sec  2.33 GBytes  1.99 Gbits/sec                  receiver

TCP Download (500M-1000M-2000M)

iperf3 -R -c ter-jhb-iperf-1.cisp.co.za -p 17001 -b 500M
Connecting to host ter-jhb-iperf-1.cisp.co.za, port 17001
Reverse mode, remote host ter-jhb-iperf-1.cisp.co.za is sending
[  5] local 2c0f:f9a8::1:27:31:b2 port 49338 connected to 2c0f:f9a8:1::1:2 port 17001
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  60.8 MBytes   510 Mbits/sec
[  5]   1.00-2.00   sec  59.6 MBytes   500 Mbits/sec
[  5]   2.00-3.00   sec  59.5 MBytes   499 Mbits/sec
[  5]   3.00-4.00   sec  59.6 MBytes   500 Mbits/sec
[  5]   4.00-5.00   sec  59.6 MBytes   500 Mbits/sec
[  5]   5.00-6.00   sec  59.6 MBytes   500 Mbits/sec
[  5]   6.00-7.00   sec  59.6 MBytes   500 Mbits/sec
[  5]   7.00-8.00   sec  59.7 MBytes   500 Mbits/sec
[  5]   8.00-9.00   sec  59.6 MBytes   500 Mbits/sec
[  5]   9.00-10.00  sec  59.5 MBytes   499 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.03  sec   598 MBytes   500 Mbits/sec    0             sender
[  5]   0.00-10.00  sec   597 MBytes   501 Mbits/sec                  receiver

iperf3 -R -c ter-jhb-iperf-1.cisp.co.za -p 17001 -b 1000M
Connecting to host ter-jhb-iperf-1.cisp.co.za, port 17001
Reverse mode, remote host ter-jhb-iperf-1.cisp.co.za is sending
[  5] local 2c0f:f9a8::1:27:31:b2 port 43430 connected to 2c0f:f9a8:1::1:2 port 17001
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   121 MBytes  1.02 Gbits/sec
[  5]   1.00-2.00   sec   119 MBytes  1.00 Gbits/sec
[  5]   2.00-3.00   sec   119 MBytes   998 Mbits/sec
[  5]   3.00-4.00   sec   119 MBytes  1.00 Gbits/sec
[  5]   4.00-5.00   sec   119 MBytes  1000 Mbits/sec
[  5]   5.00-6.00   sec   119 MBytes   999 Mbits/sec
[  5]   6.00-7.00   sec   119 MBytes  1.00 Gbits/sec
[  5]   7.00-8.00   sec   119 MBytes  1.00 Gbits/sec
[  5]   8.00-9.00   sec   119 MBytes  1.00 Gbits/sec
[  5]   9.00-10.00  sec   119 MBytes  1.00 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec  1.17 GBytes  1.00 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  1.17 GBytes  1.00 Gbits/sec                  receiver


iperf3 -R -c ter-jhb-iperf-1.cisp.co.za -p 17001 -b 2000M
Connecting to host ter-jhb-iperf-1.cisp.co.za, port 17001
Reverse mode, remote host ter-jhb-iperf-1.cisp.co.za is sending
[  5] local 2c0f:f9a8::1:27:31:b2 port 52652 connected to 2c0f:f9a8:1::1:2 port 17001
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   241 MBytes  2.02 Gbits/sec
[  5]   1.00-2.00   sec   238 MBytes  2.00 Gbits/sec
[  5]   2.00-3.00   sec   238 MBytes  2.00 Gbits/sec
[  5]   3.00-4.00   sec   239 MBytes  2.00 Gbits/sec
[  5]   4.00-5.00   sec   238 MBytes  2.00 Gbits/sec
[  5]   5.00-6.00   sec   238 MBytes  2.00 Gbits/sec
[  5]   6.00-7.00   sec   238 MBytes  2.00 Gbits/sec
[  5]   7.00-8.00   sec   239 MBytes  2.00 Gbits/sec
[  5]   8.00-9.00   sec   238 MBytes  2.00 Gbits/sec
[  5]   9.00-10.00  sec   238 MBytes  2.00 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.02  sec  2.33 GBytes  2.00 Gbits/sec    4             sender
[  5]   0.00-10.00  sec  2.33 GBytes  2.00 Gbits/sec                  receiver

Thanks for taking the time to investigate and for sharing your findings.

Given that you've already shared your results and recommendation with the escalation analyst handling my ticket, I understand the next step is for Cool Ideas to formally engage Zoom Fibre regarding the suspected access equipment issue and the possibility of moving my connection to another switch.

The ticket has been open since 13 September, so I would appreciate it if Cool Ideas could follow through on this recommendation and obtain a clear response and action plan from Zoom.

Hopefully, this will finally move us towards resolving the issue.

Thanks again for your assistance.

 

[CoolEscalator]

Thanks for taking the time to investigate and for sharing your findings.

Given that you've already shared your results and recommendation with the escalation analyst handling my ticket, I understand the next step is for Cool Ideas to formally engage Zoom Fibre regarding the suspected access equipment issue and the possibility of moving my connection to another switch.

The ticket has been open since 13 September, so I would appreciate it if Cool Ideas could follow through on this recommendation and obtain a clear response and action plan from Zoom.

Hopefully, this will finally move us towards resolving the issue.

Thanks again for your assistance.

Only a pleasure
We will be sure to keep you updated