copacetic
King of the Hippies
I look forward to the mail.
South Africa’s biggest forum. Discuss, discover, and connect with thousands of members.
Hi all,
So apologies for the lack of responses in the thread since my last update. As previously mentioned there have been very sensitive legal and commercial discussions taking place on our end, which as anyone in business will know, is not something that can be openly discussed. I can however disclose the following now:
1) nearly all network performance issues have been fully resolved. Even Gauteng which had remnants of issues after the upgrades has had a stark performance increase over the last few days.
2) Http download traffic remains an issue that the upstream provider are busy resolving. It's a pretty complex problem they are having to deal with, but in our testing multi-threading the download if possible resolves most of the problems with http downloads.
3) other traffic is working very well in our testing: streaming, torrents, nntp and the like, although we are still monitoring this and sending requests for changes when we spot problems. Let us know what's not working optimally and we'll try to get them fixed up now.
4) the big announcement will be sent tomorrow to customers. We have done something pretty unprecedented in the history of ISPs to a) fix the performance issues; and b) mitigate the possibility of them ever returning. This is what has prevented us from commenting on this publicly - needless to say an email is being drafted to be sent to customers during the course of tomorrow to explain exactly what we've done to resolve any possible future performance issues on the network. I'm quite sure once the mail is read you will understand why this was kept so quiet and why it involved such a lengthy legal and commercial avenue to get right.
5) the issues with some customers not receiving updates from our support systems has been fixed.
6) the automated systems that involved customers moving to a blocked IP range has been fixed (read: won) by us.
In terms of the lengths we have gone to, to protect customers; to ensure performance issues don't crop up again; and to resolve actual issues will become abundantly clear in tomorrow's direct mail to all customers. It's been a rough patch the last month or so and there's no denying it. But unlike other ISPs we actually tried to contact every customer who may have been affected who tried to sign up to alert them to the fact, and even went as far as to ask customers to delay their activations with us - not only to preserve the network for existing customers but because it's the right thing to do. We've had to undergo some drastic, expensive, and serious changes to ensure that we mitigate as many risks of quality and performance issues cropping back up again. Naturally one cannot guarantee that no network issues will ever crop up again, but there are measures one can take to prevent them in future, and this is precisely what we've done. And it's a rather massive change that is now on its way, which ensures that we not only take direct control over the network, but that we employ redundancy even on the core network and backhaul network should a future issue ever crop up again. Details will become clear tomorrow when we will be able to disclose more information, and when we'll be able to explain what happened, and what's been done about it.
The bottom line is that the corner has been turned on network performance in a massive way; we've essentially built a completely redundant network around the existing infrastructure at a huge expense; and we will be ensuring that all customers don't have to walk into the new year with concerns about their ISP - as long as they're a Crystal Web customer. We do need your support to make this a reality, and while I know that it's been a rough ride the last month and a bit, we are incredibly grateful to everyone who has stuck with us through it. With the changes that are about to be made, it is about to become well worth it, so thank you.
I will be able to get back to the frequency of updates and openness you've come to expect of us from tomorrow onwards now, which is such a relief for all of us in management here. And I am now back in the office after all commercial and legal negotiations are done, which means I can help out on the forums again and we'll now be back to looking after you guys the way we have always committed to on the MyBroadband forums.
Do fix this:
Crystal Web Premium Business Uncapped 4Mbps
Current Montly Usage: 106.78 GB
Last IP Address: 105.xxx.x.xxx
Username: xxx@crystalpremium
View Access Details
..........
3) other traffic is working very well in our testing: streaming, torrents, nntp and the like, although we are still monitoring this and sending requests for changes when we spot problems. Let us know what's not working optimally and we'll try to get them fixed up now.
...........
CW, NNTP still not on par. I am getting 60KB/s down on CW news server and about 140KB/s down on test.xennanews.com.
OK thanks for letting me know. Will report it to upstream and try to get some feedback for you.
Our news server partners are in the process of installing new hardware at the moment which may or may not impact performance. It's a protocol I personally haven't been testing since I've been back in the office but I'll load it up on our test servers and see what I can pick up for you.
so here's the difference between a stable account with an alternate ISP and my CW account, using a 25MB test
alternate
View attachment 175543
CW
View attachment 175545
Crystal Web aren't going to get away from these issues, I was a cybersmart unshaped capped user for a year and a half before moving to crystal web and exactly the same issues regarding throughput. When I was with Cyber Smart they had good patches and bad patches, But nothing as bad as what Crystal Web have been experiencing the past month or so.
I personally believe it's just the Cybersmart network as a whole that has the throughput issues or it could be a misconfigured shaper, Although considering you still get bad throughput when you are still on your first unshaped block with Crystal Web, It leads me to believe that they are either - 1) Shaping the Unshaped Blocks which uncapped users get. OR 2) They just have really bad throughput issues for 2 years now.
So when is the big announcement email coming out today?
Just make sure you nail the dismount.... we have had to do some acrobatics on a business model.
Just make sure you nail the dismount.