CellC routes to I.S. hosted akamai CDN servers via London ??

[Q-Tech]

I discovered this while waiting for the mybb page to load via my CellC 3G connection ...

mybb uses effectivemeasure.net for banner ads etc.

effectivemeasure.net, like any number of other global content providers including apple, mcafee, symantec, etc. uses akamai content delivery network (CDN) to locally host content, thereby reducing latency.

akamai uses I.S. to host its CDN servers in South Africa.

When an nslookup is done for content that is available from a local akamai server, CellC DNS correctly responds with the IP address of one of the I.S. hosted servers, however, that's where the wheels fall off :

The routing within the CellC network sends traffic destined for the I.S. hosted server off to London where it crosses on to the I.S. network at some peering point and then comes all the way back to South Africa - strangely, traffic to www.is.co.za does not do this - it goes directly to IS via JINX

No wonder the CellC network is slow and congested - could the CellC rep on here please take this info back to the engineers and ask them to take a look.

For the TL;DR types, you can stop here

For the rest, below are some traceroute samples showing the difference in route between an ISDSL connection and CellC. (The first few hops have been left out to protect the innocent.)

cdn.effectivemeasure.net :

ISDSL :

2 196-210-138-129.dynamic.isadsl.co.za (196.210.138.129) 18.182 ms 20.610 ms 22.691 ms
3 cdsl2-rba-vl2563.ip.isnet.net (196.38.73.213) 26.095 ms 28.719 ms 30.942 ms
4 cdsl2-rba-vl150.ip.isnet.net (196.38.73.9) 33.726 ms 35.852 ms 38.264 ms
5 168.209.1.140 (168.209.1.140) 42.021 ms 44.259 ms 46.422 ms
6 * * *
7 a196-26-223-9.deploy.akamaitechnologies.com (196.26.223.9) 39.095 ms 39.892 ms 40.233 ms

CellC :

3 41.50.20.33 (41.50.20.33) 63.735 ms 79.244 ms 96.364 ms
4 10.220.231.70 (10.220.231.70) 111.906 ms 129.490 ms 144.757 ms
5 41.50.16.1 (41.50.16.1) 161.528 ms 193.178 ms 176.313 ms
6 * 41.50.0.3 (41.50.0.3) 223.960 ms 238.838 ms
7 41.50.0.2 (41.50.0.2) 207.161 ms 192.352 ms 191.636 ms
8 41.50.7.35 (41.50.7.35) 287.205 ms 286.390 ms 303.648 ms
9 41.50.7.33 (41.50.7.33) 142.919 ms 140.847 ms 142.860 ms
10 if-15-0-1.mcore3.L78-London.as6453.net (195.219.144.25) 290.836 ms 291.709 ms 299.446 ms
11 if-3-3-0-2.tcore1.L78-London.as6453.net (80.231.130.29) 347.130 ms 313.167 ms 283.043 ms
12 Vlan704.icore1.LDN-London.as6453.net (80.231.130.10) 315.891 ms * *
13 Vlan533.icore1.LDN-London.as6453.net (195.219.83.102) 300.578 ms 301.306 ms 290.559 ms
14 195.50.124.34 (195.50.124.34) 332.598 ms 299.816 ms 260.063 ms
15 core2b-pkl-tenge-0-7-0-0-126.ip.isnet.net (168.209.201.82) 290.192 ms 290.394 ms 290.652 ms
16 csw5-pkl-tg2-7.ip.isnet.net (168.209.1.188) 288.989 ms 279.558 ms 279.418 ms
17 a196-33-166-201.deploy.akamaitechnologies.com (196.33.166.201) 290.065 ms 279.767 ms 284.502 ms


appldnld.apple.com :

ISDSL :

2 196-210-138-129.dynamic.isadsl.co.za (196.210.138.129) 17.739 ms 20.173 ms 22.628 ms
3 cdsl2-rba-vl2563.ip.isnet.net (196.38.73.213) 25.899 ms 28.289 ms 30.978 ms
4 cdsl2-rba-vl150.ip.isnet.net (196.38.73.9) 33.021 ms 35.128 ms 37.565 ms
5 168.209.1.140 (168.209.1.140) 41.589 ms 43.789 ms 46.230 ms
6 * csw6-pkl-tg2-7.ip.isnet.net (168.209.1.189) 48.030 ms *
7 a196-26-223-17.deploy.akamaitechnologies.com (196.26.223.17) 37.151 ms 38.403 ms 38.137 ms

CellC :

3 41.50.20.33 (41.50.20.33) 93.420 ms 108.910 ms 125.856 ms
4 10.220.231.70 (10.220.231.70) 141.717 ms 159.258 ms 206.117 ms
5 41.50.16.1 (41.50.16.1) 190.811 ms 173.944 ms 222.800 ms
6 41.50.0.3 (41.50.0.3) 238.528 ms 252.899 ms 269.622 ms
7 41.50.0.2 (41.50.0.2) 284.285 ms 223.066 ms 193.039 ms
8 * 41.50.7.35 (41.50.7.35) 271.243 ms 303.694 ms
9 41.50.7.33 (41.50.7.33) 159.165 ms 157.200 ms 158.734 ms
10 if-15-0-1.mcore3.L78-London.as6453.net (195.219.144.25) 290.999 ms 290.892 ms 293.636 ms
11 if-3-3-0-2.tcore1.L78-London.as6453.net (80.231.130.29) 293.874 ms 294.678 ms 295.303 ms
12 * * *
13 Vlan533.icore1.LDN-London.as6453.net (195.219.83.102) 278.595 ms 300.513 ms 290.562 ms
14 195.50.124.34 (195.50.124.34) 289.672 ms 289.524 ms 289.441 ms
15 core2b-pkl-tenge-0-7-0-0-126.ip.isnet.net (168.209.201.82) 288.171 ms 290.376 ms 288.430 ms
16 csw5-pkl-tg2-7.ip.isnet.net (168.209.1.188) 260.420 ms 279.911 ms 280.302 ms
17 a196-33-166-201.deploy.akamaitechnologies.com (196.33.166.201) 290.034 ms 290.222 ms 272.447 ms

 

[Saajid]

Interesting find. Well done. I'd really like to know what Cell C have to say about this.

 

[ginggs]

The routing within the CellC network sends traffic destined for the I.S. hosted server off to London where it crosses on to the I.S. network at some peering point and then comes all the way back to South Africa - strangely, traffic to www.is.co.za does not do this - it goes directly to IS via JINX

It may come down to cost. IS might be charging more for local peering than Cell-C can buy international bandwidth.

See:
http://mybroadband.co.za/news/telecoms/65866-who-peers-for-free-and-who-doesnt.html

If this is the case, I support Cell-C 100%, and would rather wait a few milliseconds more for data from IS.

 

[Q-Tech]

Hi ginggs,

I agree that ISP's charging for local peering goes against the principles the internet was founded on. If this is indeed the reason for CellC doing this though, surely it would make sense for them to redirect akamai and other CDN traffic to a UK or european destination rather than allowing it go full circle and end up back in SA anyway. The CellC network is congested already and I can only think that the additional load from retries and timeouts is making it worse.

Cheers,
Q.

 

[ginggs]

The CellC network is congested already and I can only think that the additional load from retries and timeouts is making it worse.

What makes you think the Cell-C is network is congested?

Have a look at some of the speedtest results in this thread:
http://mybroadband.co.za/vb/showthread.php/258764?p=9877780#post9877780

Sure, there may be some base stations suffering from congestion, or what ever the problem is that Huawei and Cell-C found, but I wouldn't say their international links are congested.

 

[The_Unbeliever]

See my speedtest done a few minutes ago :

This is where Vodacom and MTN both have weak signal and poor transfer rates.

 

[Q-Tech]

A quick update on this - as per this mybb article today, CellC is now using local open peering.
I don't know if the local peering links are active yet, but a check on the two destinations in my first post now shows much more sensible routing and response times via JINX and on to MWEB's network.

Well done CellC - you guys obviously identified the problem and did something about it.

 

[Q-Tech]

Route to CellC Web site from CellC mobile data connection goes via London ??

WTF dudes -? How is it possible to trash your routing tables so badly that your own clients can't even access your web site ??

Trace route taken at 22h45 08/07/2013 :

3 57 ms 39 ms 39 ms 41.48.23.1
4 46 ms 46 ms 40 ms 10.228.221.37
5 67 ms 79 ms 78 ms 10.228.223.70
6 44 ms 149 ms 49 ms 41.48.16.1
7 58 ms 79 ms 59 ms 41.48.0.3
8 306 ms 239 ms 219 ms 41.48.21.100
9 50 ms 49 ms 39 ms 41.48.21.97
10 227 ms 219 ms 226 ms 83.231.233.141
11 225 ms 219 ms 219 ms xe-0-0-0-0.r00.londen10.uk.bb.gin.ntt.net [129.250.6.53]
12 227 ms 219 ms 207 ms ae-0.level3.londen10.uk.bb.gin.ntt.net [129.250.9.126]
13 245 ms 239 ms 229 ms ae-52-52.csw2.London1.Level3.net [4.69.139.120]
14 246 ms 229 ms 229 ms 4.69.166.29
15 247 ms 219 ms 228 ms 195.50.122.182
16 236 ms 240 ms 229 ms ls-cr-2.uk--ls-pr-2.uk.mtnns.net [209.212.111.186]
17 266 ms 228 ms 249 ms rb-cr-1.za--ls-cr-2.uk.mtnns.net [196.44.31.115]
18 276 ms 239 ms 309 ms 196.44.31.97
19 327 ms 279 ms 259 ms 196.31.220.10
20 288 ms 339 ms 239 ms tengigabitethernet1-1.gw20.jnb6.za.mtnbusiness.net [196.31.220.23]
21 246 ms 238 ms 239 ms tengigabitethernet5-1.hr9.jnb6.za.mtnbusiness.net [196.31.63.146]
22 297 ms 279 ms 239 ms core-router1.jnb2.host-h.net [196.30.213.108]
23 267 ms 269 ms 229 ms firewall5.jnb2.host-h.net [41.72.136.23]

 

[alternate]

Sometimes it's cheaper to route local traffic overseas than nationally.

 

[Segg]

from what I understand its much cheaper to send traffic from sa to london and back to sa than it is to send is locally, IIRC there was an article about a month or two back talking about it

 

[mercenary]

CellC CDN Routing

Hi Guys,

Any idea why CellC would route traffic like this?

Who can I call to fix this?

traceroute to a41.dscg10.akamai.net (165.165.46.17), 64 hops max, 52 byte packets
1 * * *
2 41.50.20.X (41.50.20.X) 65.718 ms 59.306 ms 120.195 ms
3 10.220.231.70 (10.220.231.70) 100.138 ms 54.868 ms 49.935 ms
4 41.50.16.1 (41.50.16.1) 50.112 ms 39.580 ms 50.354 ms
5 41.50.0.3 (41.50.0.3) 49.503 ms 49.322 ms 50.230 ms
6 41.48.253.34 (41.48.253.34) 89.932 ms 88.755 ms 60.027 ms
7 41.48.21.100 (41.48.21.100) 250.064 ms 219.102 ms 229.737 ms
8 41.48.21.97 (41.48.21.97) 59.596 ms 49.220 ms 50.130 ms
9 83.231.233.141 (83.231.233.141) 240.107 ms 239.495 ms 220.010 ms
10 ae-1.r02.londen03.uk.bb.gin.ntt.net (129.250.5.25) 239.987 ms 248.996 ms 230.031 ms
11 xe-2-0-2.ar1.lhr1.uk.nlayer.net (69.22.139.42) 289.901 ms 228.750 ms 240.435 ms
12 92.60.244.178 (92.60.244.178) 249.593 ms 219.371 ms 239.596 ms
13 196.43.11.218 (196.43.11.218) 240.109 ms 239.037 ms 239.520 ms
14 196.43.26.58 (196.43.26.58) 249.720 ms 238.811 ms 239.928 ms
15 akamai-ip17.telkom-ipnet.co.za (165.165.46.17) 270.110 ms 248.878 ms 219.646 ms

They are routing traffic out and back into South Africa.

Thanks

 

[mercenary]

thanks for the merge admin

Why doesn't CellC simply direct traffic to MWebs akamai node? I would imagine they would peer according the to article;

http://mybroadband.co.za/news/telecoms/65866-who-peers-for-free-and-who-doesnt.html

 

[Q-Tech]

I will do some more checks on this tonight - as mentioned way up top in this thread, the destinations I initially identified were being routed locally when I checked a couple of weeks later, but now it seems different akamai hosts have different rules, or CellC local peering is completely broken and everything is going overseas again.

 

[Q-Tech]

I have done a few more checks tonight and the traces show CellC is not peering locally any more. I tested a number of CDN destinations and they either route to servers hosted in Europe or go via London and back to SA. For the CellC marketing types who may be monitoring the forums - here's a little irony for you :

As mentioned previously in this thread, the route to the CellC web site which is hosted by Hetzner follows the same path to London and back to SA where it crosses on to the MTN Business backbone and then on to Hetzner. Ping time is ~240ms at the moment.
The route to the MTN web site goes directly off the CellC network on to MTN Network Services. Ping time is ~60ms at the moment
The route to the Vodacom web site goes directly off the CellC network on to Telkom's backbone and then on to Vodacom's network. Ping time is ~61ms at the moment
The route to Telkom Mobile's web site goes directly off the CellC network on to Telkom's backbone. Ping time is ~60ms at the moment

The real impact of this technical stuff, dear marketing types, is that CellC's competitors can provide information about their products and services to CellC clients four times faster than your own web site does.

For the rest of us who just want an ISP with a network that works, please could someone from CellC explain the logic behind this - as mentioned by mercenary above, there are free local peers who do host akamai and other CDN services, why not use them ?

Thank you.

 

[ginggs]

Related, HTTP (port 80) traffic to http://www.cellc.co.za routed locally, HTTPS (port 443) traffic to https://www.cellc.co.za routed via London:
http://mybroadband.co.za/vb/showthread.php/504196?p=9789456#post9789456

@Q-Tech: are you doing ICMP traceroutes? It could be that ICMP is routed via London, but HTTP is not.

 

[Q-Tech]

Hi ginggs,

Thanks for pointing that out - I am doing ICMP traceroutes and it is entirely possible that ICMP traffic is being handled differently due to transparent proxies, load balancers or other clever gadgets sitting in the path - I'll try to find a way to get a definitive answer on http and https routes as these are the protocols used by most of the CDN services.

Cheers,
Q.

 

[Q-Tech]

I have now done TCP traceroutes on port 80 and 443 to the same CDN destinations and they are showing the same routes as the ICMP traces I did earlier. The CellC web site routes via London for both http and https.