Afrihost Capped ADSL Feedback (MTN) - New beginning

[AfriMan]

2% packetloss and ping spikes around a year after I fist reported it to your "CORE NETWORK TEAM"

I am at a loss no pun intended on how a issue as big as packetloss can still not be resolved?

I don't believe we have any general packet loss issues on the network. Aside from some specific reports around in-game packet loss, and we known problems relating to peak demand right now (going back possibly to mid-March, if I remember correctly), we haven't had general packet loss problems over the entire network.

If you've had problems for a year consistently, it would be very unusual and we'd need to troubleshoot with you. Chances are there is something else contributing to this, but we'll have to test methodically to find the possible cause.

You're welcome to PM me and we'll get that process underway.

 

[Razotter]

I see you're thread hopping here. So as I mentioned to you on the Uncapped thread where you pointed to deliberate shaping or throttling, I just want to point out that this does not make sense. If this was a shaping issue then it would never manifest on unshaped products. These are separate data pools, with entirely different rulesets, so the lifting of shaping on an Uncapped account making the symptoms go away, as you reported, would make no sense at all. So again, this requires more investigation to find out what could possibly work to get around this once we understand what is actually happening here.

I think the big question is why would we want to throttle Chrome traffic. With P2P and other types of data affecting demand, focussing on Chrome is not going to benefit us in any possible way. The amount of bandwidth will be negligible (in general terms) so it would be a complete waste of time to limit traffic on this service.

Just in general, if you're happy to make sarky comments to me, I would assume that you can handle one or two thrown back. If not's the case, I think I've failed Forums 101

I think there is a misunderstanding in my use of the word 'deliberate'. I did not mean Afrihost is specifically choosing to throttle the traffic. You are right in that that would not make sense. It would not apply to capped accounts which by your own testament are unshapped and unthrottled. And yet the behaviour we are seeing all points to traffic being limited severely and regularly, on both capped and uncapped accounts.

And by your own admission Afrihost doesn't know what is causing it. Is it then so easy to say "It cannot be shaping because we know it isn't"?

I have asked three friends who use Afrihost accounts, both capped and uncapped. Two of those hadn't noticed they were having a problem but on testing with google maps and inbox in chrome, we found all of us are experiencing the same problem. Four samples isn't a great size, but I'd hazard a guess that the numbers are greater than that.

All we're asking is for someone to look at our problem seriously Afriman. I get that this is a difficult problem to resolve, but we aren't going to get anywhere by simply saying "It cannot be" when collected evidence by your customers points in a different direction.

 

[AfriMan]

I think there is a misunderstanding in my use of the word 'deliberate'. I did not mean Afrihost is specifically choosing to throttle the traffic. You are right in that that would not make sense. It would not apply to capped accounts which by your own testament are unshapped and unthrottled. And yet the behaviour we are seeing all points to traffic being limited severely and regularly, on both capped and uncapped accounts.

And by your own admission Afrihost doesn't know what is causing it. Is it then so easy to say "It cannot be shaping because we know it isn't"?

I have asked three friends who use Afrihost accounts, both capped and uncapped. Two of those hadn't noticed they were having a problem but on testing with google maps and inbox in chrome, we found all of us are experiencing the same problem. Four samples isn't a great size, but I'd hazard a guess that the numbers are greater than that.

All we're asking is for someone to look at our problem seriously Afriman. I get that this is a difficult problem to resolve, but we aren't going to get anywhere by simply saying "It cannot be" when collected evidence by your customers points in a different direction.

I totally understand you POV here, and I really appreciate your willingness to engage and find solutions. Right now, we are all in agreement that this is happening, and it seems to happen on our network. The number of clients affected is unknown, we simply don't have data to support how many clients use Chrome and how many clients have experienced this problem. We also do not know the specific cause of the issue, but when we test with every variation of shaped, unshaped, rate limited accounts, we aren't able to identify a commonality that we can change in order to remove the symptom from the user experience. Until we find what is causing this, it is going to be difficult to propose a solution, other than removing the setting which we know will work for now. We are definitely continuing to investigate and work with this, and I can tell you that we are taking this seriously in every aspect. We definitely not blaming Google/Chrome, they have every right to change their settings whether we like it or not, but until we know where the conflict is, we'll have to keep testing and investigating until we do.

 

[Worskos]

What time did you run these tests?

We did see a significant in latency in the South last night during the peak period demand, but that would have started dying off before 10pm, according to my reports. Current latency in the south should be mostly normal, but we do usually see a brief spike in demand at 8am as businesses log on, though that is usually not accompanied by latency.

I still see latency on your line on both traces, so I still want to convince you that while our latency is sometimes poor during high demand, this is compounding an existing latency issue on your line. It may just be that the latency is far less noticeable when not added to by high demand on our network, but it may mean that your experience may be more affected than others who do not have this.

I ran this test at 7.30 pm yesterday.
So what do you suggest now?

 

[caroper]

I'll admit we don't know where the problem is. If it is on our network, we've searched through all our rules and protocols to find what could conflict with Google's experimental traffic and we've haven't found a source yet. We'll keep looking for anything that could be changed on our side that could improve our client's experience.

Definitely not related to capacity in any way.

Well have you contacted Google? As you point out, it is experimental, and I am sure that Google Would appreciate feedback if it is not working on a specific network. They may well have seen this issue else where and be able to point your Network engineers in the right direction.

Remember also that all Google products are classed as experimental or Beta for several years whilst actually in production.
If by saying " conflict with Google's experimental traffic ", you are trying to imply that the problem may simply vanish or the experiment dropped, just remember that you are "an Experimental Network" and only last year forced out software changes that broke the network so severely that you had to call in Engineers from overseas to fix it.

 

[AfriMan]

I ran this test at 7.30 pm yesterday.
So what do you suggest now?

There was definitely latency on the network at that time, which I think (from my reports) continued until between 9pm and 10pm.

I think we need to investigate why there is intermittent latency on your line regardless of which ISP you test with. If it's regardless of which ISP you test with, that would usually indicate a line problem.

 

[Worskos]

There was definitely latency on the network at that time, which I think (from my reports) continued until between 9pm and 10pm.

I think we need to investigate why there is intermittent latency on your line regardless of which ISP you test with. If it's regardless of which ISP you test with, that would usually indicate a line problem.

Tell me, what indicates to you that there is latency on the line in general?
Where on the WA test is there latency?

 

[bgotapen]

I don't believe we have any general packet loss issues on the network. Aside from some specific reports around in-game packet loss

So the specific reports of packet loss is what? A non-issue? This is the same packetloss I had when I sighned up april last year, I reported it was happening to certain websites and games. Leugue of legends had 13% that month.

It stabilized to 2% and since then it has been consistent. Packetloss is NOT A SIGN OF A HEALTHY NETWORK.

At peak times I get packetloss to afrihost.co.za at certain times during the day... 90% of the time I get packetloss to bf4, bf3, cod,lol,sc2,cs go.


Cape town cs go servers: 197.84.209.20
|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.1.1 - 0 | 200 | 200 | 0 | 0 | 4 | 0 |
| No response from host - 100 | 200 | 0 | 0 | 0 | 0 | 0 |
| 41.181.221.249 - 0 | 200 | 200 | 16 | 57 | 212 | 72 |
| 41.181.221.250 - 0 | 200 | 200 | 15 | 18 | 74 | 17 |
| qux-jh-dca-2.za-b.za.mtnbusiness.net - 0 | 200 | 200 | 16 | 20 | 78 | 17 |
| jh-dca-2.za--qux-q.za.mtnbusiness.net - 0 | 200 | 200 | 16 | 19 | 56 | 20 |
| 41.181.180.10 - 0 | 200 | 200 | 16 | 20 | 87 | 18 |
| 196.44.0.72 - 0 | 200 | 200 | 15 | 24 | 120 | 26 |
| 1122.te0-0-2-0.vic-pr-3.optinet.net - 0 | 200 | 200 | 19 | 24 | 60 | 32 |
| tengig-0-7-0-3-vic-p-1.mweb.co.za - 1 | 200 | 198 | 39 | 51 | 84 | 60 |
| te0-0-0-0.cpt-p-2.optinet.net - 1 | 200 | 198 | 45 | 48 | 58 | 48 |
| vl11.cpt-hscore-1.optinet.net - 3 | 200 | 195 | 37 | 51 | 194 | 53 |
| 196.28.178.66 - 2 | 200 | 197 | 38 | 47 | 82 | 47 |
| gig5-1-cpt-opt-65-1.optinet.net - 4 | 200 | 192 | 37 | 50 | 86 | 57 |
| No response from host - 100 | 200 | 0 | 0 | 0 | 0 | 0 |
|________________________________________________|______|______|______|______|______|______|
WinMTR - 0.8. Copyleft @2000-2002 Vasile Laurentiu Stanimir ( [email protected] )

Playing with this is bad... but with the last couple of days ping spikes it just makes it terrible. A year with this issue on your network and a lot of back and forth with your support and "critical care" and still no resolution?

 

[AfriMan]

Well have you contacted Google? As you point out, it is experimental, and I am sure that Google Would appreciate feedback if it is not working on a specific network. They may well have seen this issue else where and be able to point your Network engineers in the right direction.

Remember also that all Google products are classed as experimental or Beta for several years whilst actually in production.
If by saying " conflict with Google's experimental traffic ", you are trying to imply that the problem may simply vanish or the experiment dropped, just remember that you are "an Experimental Network" and only last year forced out software changes that broke the network so severely that you had to call in Engineers from overseas to fix it.

I'm pretty sure that people have not had problems with Chrome going back several years, as your post suggests, and this was due to recent changes they had made to how they handle Chrome traffic. MTN's network team would be in contact with Google as a network provider, not us directly.

I don't think we'd use the same logic you are using here. A network is not a static installation, it's a dynamic, living ecosystem of devices and software that is constantly changing. Yes, every ISP pushes out changes to improve and compensate for changes in the internet environment, that just makes sense. So any network would be experimental by this definition. The overseas experts you referenced here were part of the changes that affected us last year, and they worked with us on implementing the changes, figuring out what went wrong and fixing things to improve the network as quickly as possible. We are in constant contact with all our vendors and consult them as often as possible to confirm with their recommended best practice. That would just make sense, I would think?

 

[AfriMan]

Tell me, what indicates to you that there is latency on the line in general?
Where on the WA test is there latency?

From this test (and the previous tests I recall seeing in your posts):

Tracing route to www.mybroadband.co.za [197.242.89.170]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 10.0.0.2
2 * * * Request timed out.
3 13 ms 76 ms 81 ms cdsl2-ctn-vl2473-ipc.ip.isnet.net [196.38.72.194
]
4 11 ms 12 ms 25 ms vlan2473.cdsl2-ctn.isdsl.net [196.38.72.193]
5 26 ms 12 ms 12 ms 196.35.115.136
6 11 ms 12 ms 12 ms mi-za-cpt-p8-te0-0-0-0.ip.isnet.net [168.209.6.1
3]
7 19 ms 16 ms 13 ms css1-ctn-gi0-1.ip.isnet.net [168.209.2.10]
8 11 ms 10 ms 11 ms 196.36.84.154
9 38 ms 29 ms 29 ms CORE.GP-CN-HET-MEE-1.TO.GP-HV-ICT-MEE-1.DFA.P2P.
10G.za.africainx.net [41.84.13.66]
10 29 ms 29 ms 29 ms 41-66-132-246-f6.HET001-CPE-1-to-GP-CN-HET-MEE-1
.africainx.net [41.66.132.246]
11 32 ms 30 ms 30 ms core-access-switch1.jnb1.host-h.net [197.189.193
.1]
12 32 ms 38 ms 31 ms row-access-switch1-row3-4.jnb1.host-h.net [197.1
89.193.36]
13 28 ms 29 ms 35 ms www.mybroadband.co.za [197.242.89.170]

Trace complete.

This jump in latency tells me something is not great here. Looking at the rest of the trace, it could be a very mild spike in intermittent latency. But if you compound this with testing on our network when latency is poor, I think it could be more pronounced.

Generally I wouldn't be very concerned, but I would expect that you would see similar results on our network when there are no known latency problems.

 

[AfriMan]

So the specific reports of packet loss is what? A non-issue? This is the same packetloss I had when I sighned up april last year, I reported it was happening to certain websites and games. Leugue of legends had 13% that month.

It stabilized to 2% and since then it has been consistent. Packetloss is NOT A SIGN OF A HEALTHY NETWORK.

At peak times I get packetloss to afrihost.co.za at certain times during the day... 90% of the time I get packetloss to bf4, bf3, cod,lol,sc2,cs go.


Cape town cs go servers: 197.84.209.20
|------------------------------------------------------------------------------------------|
| WinMTR statistics |
| Host - % | Sent | Recv | Best | Avrg | Wrst | Last |
|------------------------------------------------|------|------|------|------|------|------|
| 192.168.1.1 - 0 | 200 | 200 | 0 | 0 | 4 | 0 |
| No response from host - 100 | 200 | 0 | 0 | 0 | 0 | 0 |
| 41.181.221.249 - 0 | 200 | 200 | 16 | 57 | 212 | 72 |
| 41.181.221.250 - 0 | 200 | 200 | 15 | 18 | 74 | 17 |
| qux-jh-dca-2.za-b.za.mtnbusiness.net - 0 | 200 | 200 | 16 | 20 | 78 | 17 |
| jh-dca-2.za--qux-q.za.mtnbusiness.net - 0 | 200 | 200 | 16 | 19 | 56 | 20 |
| 41.181.180.10 - 0 | 200 | 200 | 16 | 20 | 87 | 18 |
| 196.44.0.72 - 0 | 200 | 200 | 15 | 24 | 120 | 26 |
| 1122.te0-0-2-0.vic-pr-3.optinet.net - 0 | 200 | 200 | 19 | 24 | 60 | 32 |
| tengig-0-7-0-3-vic-p-1.mweb.co.za - 1 | 200 | 198 | 39 | 51 | 84 | 60 |
| te0-0-0-0.cpt-p-2.optinet.net - 1 | 200 | 198 | 45 | 48 | 58 | 48 |
| vl11.cpt-hscore-1.optinet.net - 3 | 200 | 195 | 37 | 51 | 194 | 53 |
| 196.28.178.66 - 2 | 200 | 197 | 38 | 47 | 82 | 47 |
| gig5-1-cpt-opt-65-1.optinet.net - 4 | 200 | 192 | 37 | 50 | 86 | 57 |
| No response from host - 100 | 200 | 0 | 0 | 0 | 0 | 0 |
|________________________________________________|______|______|______|______|______|______|
WinMTR - 0.8. Copyleft @2000-2002 Vasile Laurentiu Stanimir ( [email protected] )

Playing with this is bad... but with the last couple of days ping spikes it just makes it terrible. A year with this issue on your network and a lot of back and forth with your support and "critical care" and still no resolution?

This looks to me like packet loss occurs outside of our network at the destination host network (in this case optinet). Generally I would say that packet loss of 1 or 2% is not significant and should not really affect your overall experience, but compounded packet loss (which you seem to see on the destination network) might be different. It does seem like your worst responses in latency on our network are pretty bad, but it's hard to say how often that happens, the average seems pretty acceptable and definitely traffic seems pretty good within our network (from this test).

When was the last correspondence from Critical Care.

 

[Worskos]

From this test (and the previous tests I recall seeing in your posts):

Tracing route to www.mybroadband.co.za [197.242.89.170]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 10.0.0.2
2 * * * Request timed out.
3 13 ms 76 ms 81 ms cdsl2-ctn-vl2473-ipc.ip.isnet.net [196.38.72.194
]
4 11 ms 12 ms 25 ms vlan2473.cdsl2-ctn.isdsl.net [196.38.72.193]
5 26 ms 12 ms 12 ms 196.35.115.136
6 11 ms 12 ms 12 ms mi-za-cpt-p8-te0-0-0-0.ip.isnet.net [168.209.6.1
3]
7 19 ms 16 ms 13 ms css1-ctn-gi0-1.ip.isnet.net [168.209.2.10]
8 11 ms 10 ms 11 ms 196.36.84.154
9 38 ms 29 ms 29 ms CORE.GP-CN-HET-MEE-1.TO.GP-HV-ICT-MEE-1.DFA.P2P.
10G.za.africainx.net [41.84.13.66]
10 29 ms 29 ms 29 ms 41-66-132-246-f6.HET001-CPE-1-to-GP-CN-HET-MEE-1
.africainx.net [41.66.132.246]
11 32 ms 30 ms 30 ms core-access-switch1.jnb1.host-h.net [197.189.193
.1]
12 32 ms 38 ms 31 ms row-access-switch1-row3-4.jnb1.host-h.net [197.1
89.193.36]
13 28 ms 29 ms 35 ms www.mybroadband.co.za [197.242.89.170]

Trace complete.

This jump in latency tells me something is not great here. Looking at the rest of the trace, it could be a very mild spike in intermittent latency. But if you compound this with testing on our network when latency is poor, I think it could be more pronounced.

Generally I wouldn't be very concerned, but I would expect that you would see similar results on our network when there are no known latency problems.

Ok
So when will the latency problems on your network be sorted?

 

[eddief1]

Definitely not related to shaping, as we've mentioned several times. Capped is unshaped, so it's not relevant at all.

OK, unshaped right? What is your definition of unshaped exactly? I think the world's definition and Afrihost's are different.
My definition is that when you mark traffic & manipulate it based on type, IE torrents vs http....that is shaping! Slow one down to benefit the 'experience' prioritize/shape...there is no difference in the end.

we've searched through all our rules and protocols

What rules exactly? The rules on the shaper of course. But the capped account is unshaped, so why do you have to classify capped traffic in the first place if it's unshaped?

This what I think is going on...

You have total network capacity and pool capacity (IE Capped pool)
You don't have enough total capacity, be it IPC or whatever
When you reach high demand, you shape, but not on the pool, capped is unshaped as you say, but you HAVE to shape on a global level to manage latency etc.

The fact is when you run out of capacity you have to shape...period! Where in the line you shape is irreverent.

An the above is why QUIC is causing problems, it's classified as bulk/torrents because it's hard to match

I would call myself an "armchair network admin", so what do I know

 

[Unhappy1]

Firstly, you have a lot of time on your hands here. Would have taken me like a whole day to find all these quotes

Took me about 10 minutes, you just have to use the search function, chrome being the key word

I get that you are taking this as a personal issue because you reported it and have received feedback that you disagree with.

Lol ... Nah

But I think it's also important to remember that you seem to be coming from the perspective that we just don't feel like fixing things and we're too complacent to care.

Nope just that you make excuses and blame everyone but yourselves, cant possibly be Afrihost must be Google. Until two months later, oh wait it is us.

We're looking at this from all angles and there just isn't a quick fix or interim solution to make it go away. We've inspected traffic and interrogated the network for any rules, devices, software, etc that may conflict with this new type of traffic and so far we have not found anything from our side which is causing this. We'll keep looking and keep trying to ensure that everything works, and that our network delivers a great experience on every conceivable internet service possible.

After two months? Yeah that gives me very very little faith in you and your network.

I don't think name calling actually moves anything forward (though admittedly when I say we've tried everything, we haven't tried that, to be honest)

Wow, i see that one went over your head. Ostrich is in reference to you guys sticking your head in the sand (i know its a myth) , its an expression, not name calling.

 

[AfriGuy]

Ok
So when will the latency problems on your network be sorted?

I'd like to give you a date, unfortunately I can't though. It's not a constant issue though, but we do need to see things normalise.

 

[AfriGuy]

OK, unshaped right? What is your definition of unshaped exactly? I think the world's definition and Afrihost's are different.
My definition is that when you mark traffic & manipulate it based on type, IE torrents vs http....that is shaping! Slow one down to benefit the 'experience' prioritize/shape...there is no difference in the end.



What rules exactly? The rules on the shaper of course. But the capped account is unshaped, so why do you have to classify capped traffic in the first place if it's unshaped?

This what I think is going on...

You have total network capacity and pool capacity (IE Capped pool)
You don't have enough total capacity, be it IPC or whatever
When you reach high demand, you shape, but not on the pool, capped is unshaped as you say, but you HAVE to shape on a global level to manage latency etc.

The fact is when you run out of capacity you have to shape...period! Where in the line you shape is irreverent.

An the above is why QUIC is causing problems, it's classified as bulk/torrents because it's hard to match

I would call myself an "armchair network admin", so what do I know

I think we can probably look into this a little more from our end, but the time we've spent here should have yielded something to tweak from our end.
I'm afraid for now, disabling QUIC within Chrome fixes the issue.

For those of you having trouble with Google services and Chrome:

open a new tab and go to chrome://flags/
Find "Experimental QUIC protocol."
Set it to "Disabled" instead of "Default"

Boom, Google services working again. This was driving me nuts the past few days, and appears to be related to Google switching this protocol on for their services and having it enabled by default in Chrome. Not sure what the benefit is supposed to be, but clearly something's buggy with it right now and disabling it has no negative effects.

 

[AfriGuy]

Took me about 10 minutes, you just have to use the search function, chrome being the key word



Lol ... Nah



Nope just that you make excuses and blame everyone but yourselves, cant possibly be Afrihost must be Google. Until two months later, oh wait it is us.



After two months? Yeah that gives me very very little faith in you and your network.



Wow, i see that one went over your head. Ostrich is in reference to you guys sticking your head in the sand (i know its a myth) , its an expression, not name calling.

I don't think we're trying to run the blame game or point fingers. Let's not get personal though
I've replied to eddief1's post which should fix up QUIC issues for now.

 

[DeathStrike]

Working from home today. And so far I am happy with the performance of the network. Streaming music from sound cloud while I work and no issues. Hell it is better than at work.

 

[Unhappy1]

I don't think we're trying to run the blame game or point fingers.

You were more or less until yesterday.

Let's not get personal though
I've replied to eddief1's post which should fix up QUIC issues for now.

This again :erm:, im not trying to get personal, maybe its just the way you interpret my post style but its not my intention . Its about you guys approaching problems differently and you might be able to fix them in a timeous manner. They do say admittance is the first step of recovery, so at least Afrihost has taken its first step and i hope you're able to get through the next steps of fixing the issues quickly.

 

[AfriGuy]

Working from home today. And so far I am happy with the performance of the network. Streaming music from sound cloud while I work and no issues. Hell it is better than at work.

Great to hear the experience is an improved one