Poor international speeds, ISP (Cool Ideas) or line issue?

[jonathanj]

My fibre, a 50/50 package with Cool Ideas on Metrofibre's network from my home in Fricker Road, Illovo, Johannesburg, has been active for about a week now. I didn't run many tests (just a local speedtest) initially and things seemed alright, day to day usage seemed okay but maybe I was expecting too much coming from 40Mbps VDSL.

Last night was particularly bad, international HTTP downloads (hosted on Amazon's infrastructure) were crawling (yoyo-ing between 1Mbps and 2Mbps!) so I ran some speedtests local and international (to a randomly picked London server), they don't look good:

Local speedtest to Cool Ideas' Randburg server, not quite as close to 50/50 as I would have expected but maybe that is a signal quality issue:

International speedtest to randomly chosen London server, disappointing:

(Tests were done while the line was idle and from a wired machine.)

I ran some traceroutes for the downloads and the routes, through LINX, seemed pretty direct; nothing funny.

My conclusion is that this shows that it's (probably) not a Metrofibre issue since the local test is giving me 10 times better results. I also plugged my VDSL back in and easily maxed out the 40Mbps on local and international downloads and speedtests, so I can rule out any kind of local network issue and get confirmation that I'm not crazy.

I know these were done at peak time but 2Mbps on international (compared to ~40Mbps via VDSL) is not what I would consider a good result. I'm assuming this isn't the norm based on reading other parts of the forums.

@PBCool I'd really appreciate any ideas, suggestions or working together to find a resolution.

 

[PBCool]

Hi jonathanj, have you logged this with support as yet? It looks like our WACS capacity was flapping at around 1am last night for a few minutes that being said at 1am the network is operating at around 15% so the SAT3 and IS capacity wouldn't have been a problem.

We also peer with Amazon in London, so quite possibly something on their side? Did you try doing speedtests to multiple London servers as the results can vary quite a bit depending on which host you choose. Were there any other services you were having issues with?

 

[jonathanj]

Hi jonathanj, have you logged this with support as yet? It looks like our WACS capacity was flapping at around 1am last night for a few minutes that being said at 1am the network is operating at around 15% so the SAT3 and IS capacity wouldn't have been a problem.

We also peer with Amazon in London, so quite possibly something on their side? Did you try doing speedtests to multiple London servers as the results can vary quite a bit depending on which host you choose. Were there any other services you were having issues with?

Hi Paul,

Thanks for the prompt response. <3

I haven't logged it with support yet. I can definitely do that, are there any other details I should include?

I noticed the issue around 19h00 and did several speedtests (you can see the UTC times on them) around that same time and every 30-60 minutes until about 23h00, speeds improved gradually (a few Mbps) as it got later. I tried a variety of other speedtest servers (London, New York, etc.) and there was some variance but nothing major (a few Mbps at best.) If I run a speedtest from the line now, it's definitely better but we're still talking about 20-25Mbps. HTTP downloads are also better but in the same ballpark as the speedtest: 17Mbps average for a 100MB file, to a Linode in London.

Here's a download (done right now) from cloudfront.debian.net (with the DNS forced to resolve to 54.192.9.31, randomly chosen from the possible CDN edges, which appears to be Seattle, USA), average speed 13Mbps:

$ curl -L --resolve cloudfront.debian.net:80:54.192.9.31 [url]http://cloudfront.debian.net/debian-cd/8.7.1-live/amd64/iso-hybrid/debian-live-8.7.1-amd64-standard.iso[/url] > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  417M  100  417M    0     0  1704k      0  0:04:10  0:04:10 --:--:-- 1586k

I didn't try a lot of other services, bittorrent looked like it was peaking at about 15-20Mbps (I gave it about 30-45 minutes to find it's stride and then killed it) but mostly the international speed tests and HTTP downloads agreed on numbers so I stopped looking.

 

[PBCool]

Hi Paul,

Thanks for the prompt response. <3

I haven't logged it with support yet. I can definitely do that, are there any other details I should include?

I noticed the issue around 19h00 and did several speedtests (you can see the UTC times on them) around that same time and every 30-60 minutes until about 23h00, speeds improved gradually (a few Mbps) as it got later. I tried a variety of other speedtest servers (London, New York, etc.) and there was some variance but nothing major (a few Mbps at best.) If I run a speedtest from the line now, it's definitely better but we're still talking about 20-25Mbps. HTTP downloads are also better but in the same ballpark as the speedtest: 17Mbps average for a 100MB file, to a Linode in London.

Here's a download (done right now) from cloudfront.debian.net (with the DNS forced to resolve to 54.192.9.31, randomly chosen from the possible CDN edges, which appears to be Seattle, USA), average speed 13Mbps:

$ curl -L --resolve cloudfront.debian.net:80:54.192.9.31 [url]http://cloudfront.debian.net/debian-cd/8.7.1-live/amd64/iso-hybrid/debian-live-8.7.1-amd64-standard.iso[/url] > /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  417M  100  417M    0     0  1704k      0  0:04:10  0:04:10 --:--:-- 1586k

I didn't try a lot of other services, bittorrent looked it was peaking at about 2.5-3.5Mbps but mostly the international speed tests and HTTP downloads agreed on numbers so I stopped looking.

Wish I could report on there being an issue, if I do the same get from within the network I get the following performance: 9.49M/s eta 46s which seems they limit connections to 100Mbps per session.

The fibrehoods network isn't particularly busy either, running at about 30% of its peak. Can you send me the trace via PM to that IP?

 

[irBosOtter]

Well, that's kinda normal. You will not really get close to full line speed on fibre to international. You would be lucky if you get full speed locally as well most of the time. Our business 100Mb fibre (well, 1Gb local in SA) only get 4Mbps download speed from international servers...
If you really want your full speed (1:1 contention line) you will be paying upwards of R30 000 per month...

 

[PBCool]

Well, that's kinda normal. You will not really get close to full line speed on fibre to international. You would be lucky if you get full speed locally as well most of the time. Our business 100Mb fibre (well, 1Gb local in SA) only get 4Mbps download speed from international servers...
If you really want your full speed (1:1 contention line) you will be paying upwards of R30 000 per month...

Not really as our network is never really saturated, which means there is more than enough overhead for a 100Mbps speedtest to the UK, I can get up to 300Mbps down to the UK servers on a 1Gbps Vumatel line.

 

[Savvoh]

Hey Jonathan

I know this was a while back, but did you ever get any answers or solutions from Cool Ideas.

I just got a 100meg line with them and the internationals speeds hand around 5megs, which is now worse than my adsl.

Let me know,

Thanks, Sav.

 

[PBCool]

Hey Jonathan

I know this was a while back, but did you ever get any answers or solutions from Cool Ideas.

I just got a 100meg line with them and the internationals speeds hand around 5megs, which is now worse than my adsl.

Let me know,

Thanks, Sav.

Do you have a ticket for me to investigate? We recently basically doubled international capacity so it's a process of troubleshooting your issue. In most cases if it's international only it's a TCP window scaling issue or packet loss on your line.

 

[jonathanj]

Hey Jonathan

I know this was a while back, but did you ever get any answers or solutions from Cool Ideas.

I just got a 100meg line with them and the internationals speeds hand around 5megs, which is now worse than my adsl.

Let me know,

Thanks, Sav.

Sav, sorry to hear that, it's not a great feeling at all. Who is your fibre infrastructure provider? It might be worth asking them (or escalating via Cool Ideas) to come out and test for faults, I never managed to get my provider to do this for me. My issue turned out to be bad join and not a Cool Ideas problem.

One (or more) of the joins in my fibre is not the best, the result of this error (on a GPON/passive fibre setup) is barely noticeable for short/local routes and very noticeable for long/international routes, made even worse by peak usage times (since the OLT has less time to spend resending my packets.) PBCool wrote a post a while ago about this but unfortunately I can't find it now.

Good luck!

 

[jonathanj]

Do you have a ticket for me to investigate? We recently basically doubled international capacity so it's a process of troubleshooting your issue. In most cases if it's international only it's a TCP window scaling issue or packet loss on your line.

PBCool, is the TCP window scaling issue something you can look at on your end? If so, what kind of thing would I need to send to Cool Ideas support to have my problems looked at or escalated with MFN?

 

[PBCool]

PBCool, is the TCP window scaling issue something you can look at on your end? If so, what kind of thing would I need to send to Cool Ideas support to have my problems looked at or escalated with MFN?

Nah TCP Scaling is dependant on the OS and Application, we can give you a process to check for packet loss on your line. We have had a few MFN lines with packet loss attached to it. Do you have a ticket number for me?

 

[jonathanj]

Nah TCP Scaling is dependant on the OS and Application, we can give you a process to check for packet loss on your line. We have had a few MFN lines with packet loss attached to it. Do you have a ticket number for me?

I'd appreciate that. The only ticket number I have about this issue is the original one from months ago: COOL-20170605-16487

 

[PBCool]

I'd appreciate that. The only ticket number I have about this issue is the original one from months ago: COOL-20170605-16487

Cool thanks I'll have the support manager pick this up, and will escalate to MFN again.

 

[Savvoh]

Thanks for the reply. I am going to have to look at this seriously, as the only reason i went fiber is to stream international and 5 megs wont cut it. Route to NYC is now running at 2 meg.

HAHA, but will get the Cool idea guys to look before i complain to much.

 

[Savvoh]

Do you have a ticket for me to investigate? We recently basically doubled international capacity so it's a process of troubleshooting your issue. In most cases if it's international only it's a TCP window scaling issue or packet loss on your line.

I have Opened a ticket in the CP, so hopefully ill get some feedback soon.

20171114-53580

 

[LuanRousseau]

Im having the same issues now for 3 months where my issues get delay with test test test and again test but no solution im paying R1009 for my fiber 100mb/100/mb local is fine but international avg 20-30mb i have test the same with other friends with cool ideas and tt conect same issue one the other hand afrihost test show full line speeds international and local which makes me think they are looking for excuses and still keep my money but no sir i want credit on my line till this issue is resolved i have logged several tickets
#COOL-20180925-150028 #COOL-20181017-160436 #COOL-20181017-160436 #COOL-20181018-161215 #COOL-20181019-16150 #COOL-20181019-161509

 

[Sinbad]

That's a symptom of packet loss.

 

[LuanRousseau]

which side as cool ideas keeping me on a string

 

[Sinbad]

which side as cool ideas keeping me on a string

Between your PC and the destination somewhere. Can't really narrow it down unfortunately.

When I had issues like this it was at the ISP interconnect to Vumatel, but I doubt this is your issue as it would affect all clients on that interconnect. More likely it's loss between your PC and the TT connect breakout to CISP. So your network, or your individual fibre.

 

[BizzCool]

Well i support multiple client home user which i have move to cool ideas and it the same issue exact same result