Official Crystal Web ADSL performance feedback thread Part 2...

[DJ...]

There's a nationwide Telkom issue at the moment affecting general connectivity. FYI, HTTP is not shaped. Downloads can be shaped depending on the demand, but that's all...

 

[CataclysmZA]

Is it hitting you guys up in JHB as well? Or is it only the South?

 

[Crystal Web]

Is it hitting you guys up in JHB as well? Or is it only the South?

Seems to be hitting the Cape areas the hardest.

 

[CataclysmZA]

From other information on other services I am involved with, it looks like there is a problem with an IPNet router in Bellville.

We (Telkom Internet) don't have any IPC circuits, or any SAIX circuits used for ADSL customers on this router, we have circuits on the *other* router in the same location.

Fun times!

 

[CataclysmZA]

I spent an hour and a bit with General Support today. Internet is overall okay, but Youtube never uses more than 3Mb/s of bandwidth and my torrent testing reveals that I'm maxing out at about 50Kb/s for the most part, but there's occasionally zero throughput. Browsing is still snappy, which is what's really important for me right now.

 

[Crystal Web]

I spent an hour and a bit with General Support today. Internet is overall okay, but Youtube never uses more than 3Mb/s of bandwidth and my torrent testing reveals that I'm maxing out at about 50Kb/s for the most part, but there's occasionally zero throughput. Browsing is still snappy, which is what's really important for me right now.

What line speed do you have? YouTube is not shaped on our end, and will pull what it needs based on Google's player algorithms. The torrent issue concerns me a little. Is there a certain time of day that you see this on torrents?

 

[Manemarak]

There's a nationwide Telkom issue at the moment affecting general connectivity. FYI, HTTP is not shaped. Downloads can be shaped depending on the demand, but that's all...

Okay, your support agent explicitly told me HTTP is shaped which after I explicitly expressed my unhappiness. Perhaps just run this past them

 

[CataclysmZA]

What line speed do you have? YouTube is not shaped on our end, and will pull what it needs based on Google's player algorithms. The torrent issue concerns me a little. Is there a certain time of day that you see this on torrents?

4Mb and I've never seen any of this before last night when the IPNet issues popped up. Yesterday I could saturate my line with a Ubuntu 14.10 torrent, today I can't.

 

[Hectic]

@Crystal web

Can you please point to or post what are the latest and greatest regarding shaping policies on the new Network?

 

[Crystal Web]

Okay, your support agent explicitly told me HTTP is shaped which after I explicitly expressed my unhappiness. Perhaps just run this past them

HTTP downloads. Downloads are shaped, but not general HTTP traffic.

Let me explain a little better, and also give you guys some history about what we've done and why:

How the network works is pretty simple (although the setup is complex, the manner in which it operates is simple):

Latency sensitive protocols such as gaming and VOIP get highest priority
Then streaming as it is the most used service on the network and is often latency sensitive depending on the service you're streaming from (remember we used to only prioritise CDN streaming? mister will attest to the fact that we fixed this now too)
Then HTTP(s) downloads
Then NNTP and torrents

So downloads are shaped dynamically - no other traffic is. And HTTP(s) downloads will take preference over torrents and NNTP. So when everyone fires up their torrents and NNTP (services that open massive numbers of connections and try to demand full line speed at all times) they are dynamically shaped down to ensure that gamers, streamers, VOIP users, people updating Steam and other users are not affected.

It also means that you can burst to full line speed at any time on these protocols as the system automatically analyses all of the traffic and on the fly allocates the correct amount of bandwidth to each and every service based on what customers are using on the network. It also means that 500 customers switching on their torrents at 6pm won't end up destroying the gaming experience for other customers at all, and other services too. It means that only torrents for example at that point in time will be affected. And rather than then shaping for hours or days on end, it's truly dynamic and instant to release the shaping too.

Previously the network worked in the following manner:

Unshaped for a portion (but this resulted in numerous customers annihilating their lines to deplete this unshaped data putting massive strain on the network at pretty much all times. Our previous upstream provider was unable to manage this, as every time upgrades were made on the network even existing customers simply consumed this bandwidth as fast as they could). This traffic as it turns out was made up of torrent and nntp traffic predominantly
After unshaped, torrents and nntp were pretty much blocked except between midnight and 7am, and HTTP(s) downloads ran at under 10% of line speed, down to around 1% on occasion as well.
Streaming and gaming were left untouched, but this dichotomy of experiences also left many customers unhappy.

So rather than implement the shaping, we instead took the financial knock and ran customers on our capped bandwidth entirely with no shaping (for about 4 months BTW). But this too resulted in massive network strain, as the moment you upgrade the network, the bandwidth is simply consumed as quickly as it is added. There's simply far too much demand by (typically) torrenters and NNTP users for bandwidth at all times of day, and less of an inclination for very many customers to schedule these downloads for off peak periods.

This too resulted in us being unable to launch capped products, as uncapped customers were consuming the highest priority data on the network. There were lessons to be learnt there - if you by and large leave your network at the liberty and responsibility of end users to manage their downloads, you'll very quickly find your network under considerable and constant strain. And until the wholesale cost of access to Telkom's last mile infrastructure comes down considerably to ISPs, you're left with a very unsustainable situation.

So we worked hard on our own network deployment that we could control, knowing what our customers use the net for, and when. And have put in place what amounts to the easiest, and most transparent shaping policy around.

No Star rating system (competitor x). No 8.75% torrent and nntp bandwidth limit (competitor y). No throttling (competitor z). And no sharing of accounts or having to embark on saving-face marketing budgets and copying of other ISP's shaping systems (non-entity we laugh at occasionally when doing a competitor matrix). In terms of uncapped, it's the best you're going to get in our opinion in SA.

Let's look at how this now works:

Netflix, Hulu, YouTube - perfect
Twitch - perfect
Gaming - stellar
HTTP downloads - shaped to perfect
NNTP and Torrents - shaped to perfect (burstable even in peak hours)

For a consumer uncapped account, this is what we know 95% of customers really want. We've also put together new products that we'll launch soon, and some very cool new features too. In addition, we're the only ISP in SA now to offer 7Mbps and 15Mbps products (available now) to help customers make the transition to faster speeds or to help you get the most of a low sync 10Mbps profile on your ADSL line.

We also now offer two news servers - IS and our premium offering too (our premium news server we've also never capped, although the service itself should be limited to 100 gigs per month).

So what we offer is no throttling; no star ratings; no calendar month's notice on ADSL data accounts; an awesome network that "just works" on (in our opinion) the best tier 1 operator in the country; burstable speeds for things like torrents even during the day; new product speeds; new products coming soon; 24 hour support; 2 free news servers (one being a premium service); and we've fixed what was previously broken. Oh, and we scored the highest in this year's MyBroadband Best and Worst ISP survey (had to throw it in there ). We also deployed an entirely new network in record time (2 months - one of those months was a network freeze period by Telkom).

And we'll continue to fine tune the new network at all times to keep it running optimally for everyone. Last night's Telkom fault in the Cape is one we cannot control (a processor on a router in Belville failed and had to be replaced), but things under our control we will constantly monitor and with the impeccable help of the Internet Solutions teams and our guys, keep evolving to be better.

 

[CataclysmZA]

After seeing an average of 200KB/s for ten minutes, I'm back down to sub-10KB/s. From my conversation earlier this morning, General Support said that they can't see in real-time what the traffic shaper is doing. I'm at less than 5GB usage in total for this month according to my most recent usage report.

Edit: After reading DJ's post, it makes more sense now. General Support never told me that much in that detail.

 

[Crystal Web]

@Crystal web

Can you please point to or post what are the latest and greatest regarding shaping policies on the new Network?

See my last post. I tried to go into a lot of detail there and be as transparent as possible about the network and shaping.

I'm also busy updating the website at the moment (new website, in fact) but to be honest am starting to require the help of a developer with some of these forms which I will hopefully have right during the course of this week.

 

[Crystal Web]

After seeing an average of 200KB/s for ten minutes, I'm back down to sub-10KB/s. From my conversation earlier this morning, General Support said that they can't see in real-time what the traffic shaper is doing. I'm at less than 5GB usage in total for this month according to my most recent usage report.

It will burst up and down. Please don't be alarmed when you see it drop down occasionally - it will burst up again.

You won't be able to saturate a line at full speed on download protocols at any time of day. That would essentially be an unshaped account. I am however going to deploy some changes to the network make-up over the next two or three days to give some higher priorities to consumer uncapped accounts during the day.

We've always been transparent with you guys, so I'll explain a little why this is necessary. If you need to saturate your line at all times, we need to ensure IPC is provisioned for that. On a 4Mbps account, raw IPC provisioned uncontended to allow for this, would cost around R8,000 per month alone (that's without internet breakout). You pay R429. Theoretically that should mean that to break even one would have to run your account at a 20:1 contention (just to recover IPC costs), but in reality contention doesn't work like that. Because each segment of bandwidth is classified separately into priorities, average contention is actually much lower than that on our network.

So in essence, you're not going to be able to saturate an account at any time of the day on downloads - instead we burst that traffic to give you more value for your money. Other ISPs choose a fixed cost and fixed shaping and throttling policy to guarantee profits. We instead give you the additional capacity at any time. Moreover there are some changes we'll make in the coming days as mentioned to get some better throughput for some things during peak - we are constantly monitoring on our end and while we try to deploy changes rarely to avoid breaking things, we can and do make these changes more frequently now than we previously could. The guys at Internet Solutions are truly passionate about keeping their customers happy, and while we're their customer, they actually pro-actively monitor this and other CW threads just to be sure that everything is working as it should. So between us and IS, you guys are in good hands.

 

[Manemarak]

Thank you for the detailed explanation. In the meantime seeing you mentioned 15mbs, can we now please see my line upgraded to 10-20mbs as I've requested before the migration started.

Thanks!

HTTP downloads. Downloads are shaped, but not general HTTP traffic.

Let me explain a little better, and also give you guys some history about what we've done and why:

How the network works is pretty simple (although the setup is complex, the manner in which it operates is simple):

Latency sensitive protocols such as gaming and VOIP get highest priority
Then streaming as it is the most used service on the network and is often latency sensitive depending on the service you're streaming from (remember we used to only prioritise CDN streaming? mister will attest to the fact that we fixed this now too)
Then HTTP(s) downloads
Then NNTP and torrents

So downloads are shaped dynamically - no other traffic is. And HTTP(s) downloads will take preference over torrents and NNTP. So when everyone fires up their torrents and NNTP (services that open massive numbers of connections and try to demand full line speed at all times) they are dynamically shaped down to ensure that gamers, streamers, VOIP users, people updating Steam and other users are not affected.

It also means that you can burst to full line speed at any time on these protocols as the system automatically analyses all of the traffic and on the fly allocates the correct amount of bandwidth to each and every service based on what customers are using on the network. It also means that 500 customers switching on their torrents at 6pm won't end up destroying the gaming experience for other customers at all, and other services too. It means that only torrents for example at that point in time will be affected. And rather than then shaping for hours or days on end, it's truly dynamic and instant to release the shaping too.

Previously the network worked in the following manner:

Unshaped for a portion (but this resulted in numerous customers annihilating their lines to deplete this unshaped data putting massive strain on the network at pretty much all times. Our previous upstream provider was unable to manage this, as every time upgrades were made on the network even existing customers simply consumed this bandwidth as fast as they could). This traffic as it turns out was made up of torrent and nntp traffic predominantly
After unshaped, torrents and nntp were pretty much blocked except between midnight and 7am, and HTTP(s) downloads ran at under 10% of line speed, down to around 1% on occasion as well.
Streaming and gaming were left untouched, but this dichotomy of experiences also left many customers unhappy.

So rather than implement the shaping, we instead took the financial knock and ran customers on our capped bandwidth entirely with no shaping (for about 4 months BTW). But this too resulted in massive network strain, as the moment you upgrade the network, the bandwidth is simply consumed as quickly as it is added. There's simply far too much demand by (typically) torrenters and NNTP users for bandwidth at all times of day, and less of an inclination for very many customers to schedule these downloads for off peak periods.

This too resulted in us being unable to launch capped products, as uncapped customers were consuming the highest priority data on the network. There were lessons to be learnt there - if you by and large leave your network at the liberty and responsibility of end users to manage their downloads, you'll very quickly find your network under considerable and constant strain. And until the wholesale cost of access to Telkom's last mile infrastructure comes down considerably to ISPs, you're left with a very unsustainable situation.

So we worked hard on our own network deployment that we could control, knowing what our customers use the net for, and when. And have put in place what amounts to the easiest, and most transparent shaping policy around.

No Star rating system (competitor x). No 8.75% torrent and nntp bandwidth limit (competitor y). No throttling (competitor z). And no sharing of accounts or having to embark on saving-face marketing budgets and copying of other ISP's shaping systems (non-entity we laugh at occasionally when doing a competitor matrix). In terms of uncapped, it's the best you're going to get in our opinion in SA.

Let's look at how this now works:

Netflix, Hulu, YouTube - perfect
Twitch - perfect
Gaming - stellar
HTTP downloads - shaped to perfect
NNTP and Torrents - shaped to perfect (burstable even in peak hours)

For a consumer uncapped account, this is what we know 95% of customers really want. We've also put together new products that we'll launch soon, and some very cool new features too. In addition, we're the only ISP in SA now to offer 7Mbps and 15Mbps products (available now) to help customers make the transition to faster speeds or to help you get the most of a low sync 10Mbps profile on your ADSL line.

We also now offer two news servers - IS and our premium offering too (our premium news server we've also never capped, although the service itself should be limited to 100 gigs per month).

So what we offer is no throttling; no star ratings; no calendar month's notice on ADSL data accounts; an awesome network that "just works" on (in our opinion) the best tier 1 operator in the country; burstable speeds for things like torrents even during the day; new product speeds; new products coming soon; 24 hour support; 2 free news servers (one being a premium service); and we've fixed what was previously broken. Oh, and we scored the highest in this year's MyBroadband Best and Worst ISP survey (had to throw it in there ). We also deployed an entirely new network in record time (2 months - one of those months was a network freeze period by Telkom).

And we'll continue to fine tune the new network at all times to keep it running optimally for everyone. Last night's Telkom fault in the Cape is one we cannot control (a processor on a router in Belville failed and had to be replaced), but things under our control we will constantly monitor and with the impeccable help of the Internet Solutions teams and our guys, keep evolving to be better.

 

[CataclysmZA]

It will burst up and down. Please don't be alarmed when you see it drop down occasionally - it will burst up again.

Now see, if General Support had told me that when I first spoke to him/her this morning, I could have avoided an entire hour and a bit's worth of testing.

If someone had mentioned "bursting" earlier I wouldn't have been confused about why performance was apparently erratic.

 

[Soulcode]

CW, how does the shaping on the capped accounts work? Is it the same as uncapped or are they unshaped?

 

[Crystal Web]

Now see, if General Support had told me that when I first spoke to him/her this morning, I could have avoided an entire hour and a bit's worth of testing.

If someone had mentioned "bursting" earlier I wouldn't have been confused about why performance was apparently erratic.

My apologies about that. I'll have a meeting all support staff and ensure that they're properly trained up to explain this 100% in future. It's still new to the support staff too, so they're also still learning. But I do apologise about the wasted hour there.

 

[Crystal Web]

CW, how does the shaping on the capped accounts work? Is it the same as uncapped or are they unshaped?

Unshaped. But we have not released capped on the new network just yet (existing customers were moved though).

We may pose the question to you guys as to what you'd prefer. Optimally we'd prefer cheaper (slightly shaped) capped accounts with bonus night data and rollover. What do you think?

 

[Sinbad]

Can you guys see if my exchange supports a 20mb profile?

 

[Crystal Web]

Can you guys see if my exchange supports a 20mb profile?

Yours does. But you'll only get a sync of around 13/14Mbps in my opinion.