Afrihost CAPPED ADSL Feedback (MTN)

[Cynicster]

We're fine with normal day to day use, but running OS updates is especially difficult because people won't usually cancel all their other day to day online activities so the additional bandwidth is on top of daily usage.

Er... if they are running their iOS updates alongside other "online activities" it doesn't mean their lines are magically going to download at higher speeds just to handle the extra request. They are still limited to whatever their line speed is.

So just to be clear what I meant is:
Either Joe Bloggs was going to be updating his iOS device and download 1Gb of data at 4mbps over the course of 30mins or he was going to download another 1Gb of data (regular download, torrent, streaming, etc...) at 4mbps over the course of 30mins. Either way, your network is getting hit with a request for 1Gb of data at 4mbps.

 

[AfriMan]

Er... if they are running their iOS updates alongside other "online activities" it doesn't mean their lines are magically going to download at higher speeds just to handle the extra request. They are still limited to whatever their line speed is.

So just to be clear what I meant is:
Either Joe Bloggs was going to be updating his iOS device and download 1Gb of data at 4mbps over the course of 30mins or he was going to download another 1Gb of data (regular download, torrent, streaming, etc...) at 4mbps over the course of 30mins. Either way, your network is getting hit with a request for 1Gb of data at 4mbps.

That's simply not how it works, and I think I speak from experience from an ISP perspective. Very few users are trying to use their maximum capacity consistently. If a user has a need for a 1GB download, he will run that as and when he would normally submit that download. But if, in addition to scheduling the download, he also runs multiple downloads for his Apple iPhone and iPad at the same time, the overall and consistent demand is completely different.

This is an issue of scale that is difficult to conceptualise from an individual perspective, but from an overall network point of view, it's the kind of trends that we study and analyse continuously to understand how the network is being used and where it needs to be improved.

 

[MartyMarts]

Problems sorted? Will my interwebs be working this weekend or should I prepare myself?

 

[Seeyou]

The general issues in the South (not related to the iOS8 update) could be viewed as capacity related. Essentially demand during that period exceeds our resources. However, we've investigated and found that our shaping of Uncapped is not 100% efficient and also our traffic management could be improved. We are busy rolling out the updates to the software that controls these devices and we believe it will manage demand more effectively and create enough headroom on the network to bring performance back to expected levels.

We obviously have some other fallback plans, but we'll need to see the results of those actions first to determine whether we go to a plan B.

I can't quite wrap my brain around this line of logic. Let's say your capacity in the South is at 100% in the evenings with your current subscriber numbers. Clients are experiencing degraded service. Instead of securing more capacity, you implement heavier shaping on uncapped accounts. Ok, assuming that by pissing them off, you gain what - 5 to 10% capacity back? Things are ok for a short while, but the backbone is still delicately balanced. Now I'm assuming that to operate profitably, your subscriber numbers have to grow actively. Which means next month will see more usage from ADSL, not to mention mobile data users on the same backbone. Shaping included, usage quickly rises to 99% again, and service is degraded - people start complaining. Instead of starting the upgrade process, which we all remember from the beginning of the year takes MONTHS for Telkom to even start processing, you've put a band-aid on a severed limb. This is clearly foreseeable, and you're not denying that you guys need more capacity, and I have no doubt there are smart people in charge over there. So I can only assume the problem is financial limitations in securing more capacity, and perhaps a business model that doesn't support it? Obviously I'm taking shots in the dark here, but it doesn't seem to be rocket science.

 

[jannier]

When will the IP routing I reported Yesterday morning be resolved affect many users from the South and Cape Town areas?

AH:
Tracing route to webafrica.co.za [41.185.61.34]
over a maximum of 30 hops:

1 * * * Request timed out.
2 98 ms 229 ms 190 ms 41.181.178.6
3 30 ms 58 ms 93 ms ipc-recieve-jh-1a.za.mtnbusiness.net [41.181.178.5]
4 29 ms 65 ms 29 ms qux-jh-dca-2.za-b.za.mtnbusiness.net [41.181.165.115]
5 33 ms 62 ms 32 ms jh-dca-2.za--qux-q.za.mtnbusiness.net [196.31.180.4]
6 28 ms 30 ms 28 ms 41.181.180.10
7 46 ms 53 ms 30 ms 196.44.0.72
8 29 ms 30 ms 28 ms 41-208-29-125.mtnns.net [41.208.29.125]
9 31 ms 30 ms 35 ms ar2-rba-gi0-0.ip.isnet.net [196.26.0.33]
10 31 ms 30 ms 30 ms mi-za-rba-gsrpe2-gi1-1-0.ip.isnet.net [168.209.86.250]
11 27 ms 29 ms 29 ms 197.96.154.191
12 64 ms 29 ms 28 ms webafrica.co.za [41.185.61.34]

Trace complete.


WA:
Tracing route to webafrica.co.za [41.185.61.34]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 * * * Request timed out.
3 7 ms 6 ms 7 ms cdsl2-ctn-vl2473-ipc.ip.isnet.net [196.38.72.194]
4 6 ms 6 ms 6 ms vlan2473.cdsl2-ctn.isdsl.net [196.38.72.193]
5 6 ms 7 ms 6 ms 196.35.115.136
6 6 ms 6 ms 6 ms 196.26.78.33
7 8 ms 6 ms 6 ms wa-is.isdsl.net [196.38.72.84]
8 6 ms 6 ms 7 ms wnls-hr2-vl-140.wa.co.za [41.185.1.1]
9 7 ms 7 ms 6 ms webafrica.co.za [41.185.61.34]

Trace complete.

 

[AfriMan]

Problems sorted? Will my interwebs be working this weekend or should I prepare myself?

I really do hope so

 

[AfriMan]

I can't quite wrap my brain around this line of logic. Let's say your capacity in the South is at 100% in the evenings with your current subscriber numbers. Clients are experiencing degraded service. Instead of securing more capacity, you implement heavier shaping on uncapped accounts. Ok, assuming that by pissing them off, you gain what - 5 to 10% capacity back? Things are ok for a short while, but the backbone is still delicately balanced. Now I'm assuming that to operate profitably, your subscriber numbers have to grow actively. Which means next month will see more usage from ADSL, not to mention mobile data users on the same backbone. Shaping included, usage quickly rises to 99% again, and service is degraded - people start complaining. Instead of starting the upgrade process, which we all remember from the beginning of the year takes MONTHS for Telkom to even start processing, you've put a band-aid on a severed limb. This is clearly foreseeable, and you're not denying that you guys need more capacity, and I have no doubt there are smart people in charge over there. So I can only assume the problem is financial limitations in securing more capacity, and perhaps a business model that doesn't support it? Obviously I'm taking shots in the dark here, but it doesn't seem to be rocket science.

Obviously financial limitations are very real and important. All ISP have to work on very delicate balance to try to minimise overhead and upstream costs so they can keep their prices low and competitive. We could definitely order double our capacity, but that's going to mean we'll raise our prices while our competitors could drop their prices, and effectively put us out of business. We constantly balancing the constraints of doing upgrades (which I can promise you we'd love to do) and trying to bring costs down, instead of increasing them, so we can pass on savings on to clients in every way possible. We've done several upgrades in the last year, without an equivalent increase in revenue to match the increase in overheads costs, which essentially we've lost money on these improvements. We'll do these where we can, becuase we love our clients, but we don;t want to be in a situation where we either have to raise or prices (or just as bad would be not being able to drop prices to match or beat competitors) for a sustained period. So we're fighting really hard to find the right balance that puts clients first, but that we can also sustain and have a viable sustainability plan.

 

[MartyMarts]

I really do hope so

Erm... yeah, me too.

 

[MrGray]

The reality is the only other explanation for the sudden and sustained increased demand in 7pm on Wednesday would be that we signed up 10k users and they all went bananas on the network at the same time. I think the timing and nature of it, if we think back to previous years when major OS updates have landed, it's far more logical and reasonable to believe that it's more than coincidence that the timing of the iOS8 release and increase in demand must be related.

The problem with your theory is that the issues started on Tuesday evening, not Wednesday evening. Besides which, if you had any proper visibility into the network (which I am now convinced you don't, since AH's responses are only ever reactive based on complaints and your network status only changes when enough people have whinged), you wouldn't be basing your response on conjecture about a "coincidence". Not very reassuring.

 

[Seeyou]

I'd actually pay more for a rock-solid capped account. I'm sure others would too. Perhaps demand for quality over price in SA has reached the point where it would be a viable business model. I'm with AH because you're light years ahead of the other ISPs with your billing and cancellation policies, as well as the fantastic ClientZone which lets me see and manage everything. Not your prices, which although cheap, aren't the cheapest.

Anyway, I'm sure you guys have your hands full, so enough complaining from me. Looking forward to a quick resolution of the current issues.

 

[MrGray]

I'd actually pay more for a rock-solid capped account. I'm sure others would too. Perhaps demand for quality over price in SA has reached the point where it would be a viable business model. I'm with AH because you're light years ahead of the other ISPs with your billing and cancellation policies, as well as the fantastic ClientZone which lets me see and manage everything. Not your prices, which although cheap, aren't the cheapest.

Anyway, I'm sure you guys have your hands full, so enough complaining from me. Looking forward to a quick resolution of the current issues.

I agree. If it's a matter of money, I wouldn't mind paying a little more, but within reason - if they could offer a better service for, say, 20% more, I would consider it.

 

[AfriMan]

The problem with your theory is that the issues started on Tuesday evening, not Wednesday evening. Besides which, if you had any proper visibility into the network (which I am now convinced you don't, since AH's responses are only ever reactive based on complaints and your network status only changes when enough people have whinged), you wouldn't be basing your response on conjecture about a "coincidence". Not very reassuring.

It's not based on conjecture, in the context of the post I replied to, that would be the only coincidental explanation - which is entirely unplausible.

We definitely have insight into the network we monitor overall usage, trends, protocols and each by region of usage. I think it would not be correct to assume that we only become aware of problems when they are reported, though generally by the time we formulate a theory around the symptoms we would have started receiving related complaints.

 

[MrGray]

It's not based on conjecture, in the context of the post I replied to, that would be the only coincidental explanation - which is entirely unplausible.

We definitely have insight into the network we monitor overall usage, trends, protocols and each by region of usage. I think it would not be correct to assume that we only become aware of problems when they are reported, though generally by the time we formulate a theory around the symptoms we would have started receiving related complaints.

Really? So why did your network status page take two days to update and your support staff insist that it was user specific for that period? This started by at latest Tuesday evening (at least when I first noticed) but your network status page was first updated on Thursday morning. Doesn't sound like much monitoring or awareness going on at all, to be frank.

 

[bradza]

I mentioned a routing issue to Afriman ages ago and it was just returned with a textbook excuse. This is a routing issue, people aren't monitoring to optimise the routing, just playing games it seems. We've been having pathetic speeds and latency issues here with connections dropping repeatedly. I'm not happy, I'm paying for a service here not excuses. Afrihost, kick MTN into gear here. This is pathetic.

 

[AfriMan]

I agree. If it's a matter of money, I wouldn't mind paying a little more, but within reason - if they could offer a better service for, say, 20% more, I would consider it.

I'd actually pay more for a rock-solid capped account. I'm sure others would too. Perhaps demand for quality over price in SA has reached the point where it would be a viable business model. I'm with AH because you're light years ahead of the other ISPs with your billing and cancellation policies, as well as the fantastic ClientZone which lets me see and manage everything. Not your prices, which although cheap, aren't the cheapest.

Anyway, I'm sure you guys have your hands full, so enough complaining from me. Looking forward to a quick resolution of the current issues.

It's definitely noted and it's something that we'll consider in our thinking for the future.

 

[AfriMan]

I mentioned a routing issue to Afriman ages ago and it was just returned with a textbook excuse. This is a routing issue, people aren't monitoring to optimise the routing, just playing games it seems. We've been having pathetic speeds and latency issues here with connections dropping repeatedly. I'm not happy, I'm paying for a service here not excuses. Afrihost, kick MTN into gear here. This is pathetic.

I don't this issue has anything to do with routing, unless you mean throughput on local and international routes

 

[AfriMan]

When will the IP routing I reported Yesterday morning be resolved affect many users from the South and Cape Town areas?

AH:
Tracing route to webafrica.co.za [41.185.61.34]
over a maximum of 30 hops:

1 * * * Request timed out.
2 98 ms 229 ms 190 ms 41.181.178.6
3 30 ms 58 ms 93 ms ipc-recieve-jh-1a.za.mtnbusiness.net [41.181.178.5]
4 29 ms 65 ms 29 ms qux-jh-dca-2.za-b.za.mtnbusiness.net [41.181.165.115]
5 33 ms 62 ms 32 ms jh-dca-2.za--qux-q.za.mtnbusiness.net [196.31.180.4]
6 28 ms 30 ms 28 ms 41.181.180.10
7 46 ms 53 ms 30 ms 196.44.0.72
8 29 ms 30 ms 28 ms 41-208-29-125.mtnns.net [41.208.29.125]
9 31 ms 30 ms 35 ms ar2-rba-gi0-0.ip.isnet.net [196.26.0.33]
10 31 ms 30 ms 30 ms mi-za-rba-gsrpe2-gi1-1-0.ip.isnet.net [168.209.86.250]
11 27 ms 29 ms 29 ms 197.96.154.191
12 64 ms 29 ms 28 ms webafrica.co.za [41.185.61.34]

Trace complete.


WA:
Tracing route to webafrica.co.za [41.185.61.34]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 192.168.1.1
2 * * * Request timed out.
3 7 ms 6 ms 7 ms cdsl2-ctn-vl2473-ipc.ip.isnet.net [196.38.72.194]
4 6 ms 6 ms 6 ms vlan2473.cdsl2-ctn.isdsl.net [196.38.72.193]
5 6 ms 7 ms 6 ms 196.35.115.136
6 6 ms 6 ms 6 ms 196.26.78.33
7 8 ms 6 ms 6 ms wa-is.isdsl.net [196.38.72.84]
8 6 ms 6 ms 7 ms wnls-hr2-vl-140.wa.co.za [41.185.1.1]
9 7 ms 7 ms 6 ms webafrica.co.za [41.185.61.34]

Trace complete.

We had a previous issue where clients on an MSAN we being routed strangely. We logged this several times with MTN but in the end it was an issue that had to be changed on Telkom's side, to do with their routing and IP allocations

 

[xrapidx]

Been two days too long.

The capacity issues shouldn't be a problem on the unshaped capped account I pay for. I have a 10MB line and can only get 10-20KB/s at the moment.

If it is not performing as paid for by end of business I'm cancelling the account. I need a 2GB file for work this weekend, and it looks like it'll only be done in 24 hours. I was hoping to start at 4pm today.

Lets give all the capacity to apple users, and screw the rest of the network?

EDIT: Great, 1KB/s - 258 hours to completion....

 

[AfriMan]

Been two days too long.

The capacity issues shouldn't be a problem on the unshaped capped account I pay for. I have a 10MB line and can only get 10-20KB/s at the moment.

If it is not performing as paid for by end of business I'm cancelling the account. I need a 2GB file for work this weekend, and it looks like it'll only be done in 24 hours. I was hoping to start at 4pm today.

Lets give all the capacity to apple users, and screw the rest of the network?

EDIT: Great, 1KB/s - 258 hours to completion....

It was a tough decision on how to handle the update. We knew it would have an impact, but we felt that blocking or throttling the update was not going to be a good solution, as this would lead to other general complaints. We do believe that we'll see the demand die down soon and services will return to normal.

 

[xrapidx]

It was a tough decision on how to handle the update. We knew it would have an impact, but we felt that blocking or throttling the update was not going to be a good solution, as this would lead to other general complaints. We do believe that we'll see the demand die down soon and services will return to normal.

Like I said, you basically decided the entire customer base should suffer for the benefit of Apple users.