You should join our great online community now - you can win great prizes
Register now
You should subscribe to our free MyBroadband newsletter


+ Reply to Thread
Page 1 of 12 12345 11 ... LastLast
Results 1 to 15 of 176

Thread: Vumatel - CW Fibre Outage ("partial")

  1. #1

    Default Vumatel - CW Fibre Outage ("partial")

    So since monday there has been a DDOS attack on CW.

    At times the fibre has performed absolutely flawlessly. But for the majority of time there is packet loss, timeouts, international only bursting to 1mbps on a 100mbps line.


    I am getting increasingly frustrated. I understand this is not CW's fault, but it certainly is annoying to not have internet working.

    Is my only option to switch back to DSL?

  2. #2

    Default

    Quote Originally Posted by requiem View Post
    So since monday there has been a DDOS attack on CW.

    At times the fibre has performed absolutely flawlessly. But for the majority of time there is packet loss, timeouts, international only bursting to 1mbps on a 100mbps line.


    I am getting increasingly frustrated. I understand this is not CW's fault, but it certainly is annoying to not have internet working.

    Is my only option to switch back to DSL?
    You could try another isp on vuma

  3. #3

    Default

    Quote Originally Posted by Sinbad View Post
    You could try another isp on vuma
    Fair enough. Thank you

  4. #4

    Default

    Quote Originally Posted by requiem View Post
    So since monday there has been a DDOS attack on CW.

    At times the fibre has performed absolutely flawlessly. But for the majority of time there is packet loss, timeouts, international only bursting to 1mbps on a 100mbps line.


    I am getting increasingly frustrated. I understand this is not CW's fault, but it certainly is annoying to not have internet working.

    Is my only option to switch back to DSL?
    I've had the same problems, but for 3-4 weeks on Vuma + CW. Starting before the first DDOS attack. International has been spectacularly bad, 3-6mbps peak speeds on a 50mb line. Tried support a couple of times with no changes made and no improvement.

  5. #5

    Default

    Quote Originally Posted by Rivin View Post
    I've had the same problems, but for 3-4 weeks on Vuma + CW. Starting before the first DDOS attack. International has been spectacularly bad, 3-6mbps peak speeds on a 50mb line. Tried support a couple of times with no changes made and no improvement.
    There have been no network faults for that period of time at all and many here can attest to that. To test actual throughput you can always load up a few torrents and see the speed. Quite certain you'll get full throughput if your Wi-Fi supports it. More often than not poor speeds are a result of poor Wi-Fi or tcp settings causing issues on the host devices. We operate so much spare capacity on these links that they operate wholly uncontended, let alone unshaped. The only time any network management has been implemented has been during the DDoS attacks.

  6. #6

    Default

    Quote Originally Posted by Crystal Web Management View Post
    There have been no network faults for that period of time at all and many here can attest to that. To test actual throughput you can always load up a few torrents and see the speed. Quite certain you'll get full throughput if your Wi-Fi supports it. More often than not poor speeds are a result of poor Wi-Fi or tcp settings causing issues on the host devices. We operate so much spare capacity on these links that they operate wholly uncontended, let alone unshaped. The only time any network management has been implemented has been during the DDoS attacks.

    Which are still ongoing? Telling your customers that their is some third party messing with the service you are providing, and there is nothing you can do about it, is not quite the right way to handle this.

    Can you not understand the frustration of paying for a product that does not work, and having zero options other than to wait till someone gets bored of attacking CW.

  7. #7

    Default

    We've most certainly been doing something about it. Repeatedly, for days on end. Publicly disclosing our methods in this respect, what we're doing, as well as the longer term strategy is naturally not a smart move and further comments about this cannot be made. Not right now at least.

    The last line of that post was certainly not intended to be read that there's nothing we can do, nor have done.

  8. #8

    Default

    Quote Originally Posted by Crystal Web Management View Post
    We've most certainly been doing something about it. Repeatedly, for days on end. Publicly disclosing our methods in this respect, what we're doing, as well as the longer term strategy is naturally not a smart move and further comments about this cannot be made. Not right now at least.

    The last line of that post was certainly not intended to be read that there's nothing we can do, nor have done.
    You should update your notice page, and inform support to say as much to your clients. I can quote verbatim "there's nothing More CW can do"

    It has come across that the ddos attack is a smokescreen for inadequate capacity.

  9. #9

    Default

    Quote Originally Posted by Crystal Web Management View Post
    There have been no network faults for that period of time at all and many here can attest to that. To test actual throughput you can always load up a few torrents and see the speed. Quite certain you'll get full throughput if your Wi-Fi supports it. More often than not poor speeds are a result of poor Wi-Fi or tcp settings causing issues on the host devices. We operate so much spare capacity on these links that they operate wholly uncontended, let alone unshaped. The only time any network management has been implemented has been during the DDoS attacks.
    Thanks for responding, and apologies if my post came across aggressive.

    As requiem said, it's just frustrating getting 10% of line speed on international connections. And the experience, according to my perceptions and my tests has been worsening over time.

    I'll clarify my symptoms:
    • My PC is the only device connected to the CPE
    • I get full line speed to local resources, including speed tests, locally cached content (like youtube) and local http downloads. I understand this to mean that my line is perfectly fine, and no issues with Vumatel
    • Any download or upload sent internationally is severely hamstrung compared to local experience. I undersand this to be due to breakout to international links from CW -> IS -> Peers (please correct me if I am wrong here). Things I have tested are below.
    • HTTP downloads from North America (i.e nVidia graphics drivers) - max 7mbps
    • Linux or raspberry pi distributions from various providers in EU (HTTP downloads), max 5-7mbps
    • Speedtests done to: New York, Chicago, or West Coast - 3-7mbps varying.
    • Speed tests done to London (Softlayer as support recommends) - <8mmbps
    • Speed tests done to France/Switzerland - 5mbps
    • Unable to stream Twitch.tv in high or source (this used to be possible on this exact configuration). Threaded connections using a tool called livestreamer makes this possible for some channels (Twitch partners).
    • Other parallel international downloads tend to work far better - including torrents, or HTTP using a download manager (how 2002?)


    I have sent this all to support, via emails and in discussions on live chat.

    Please check the two tests done below (today, 17h45), and indicate whether this is what CrystalWeb expects an end user to see from a 50/50 FTTH connection?

    to ISP handover:
    Name:  5641726426.png
Views: 659
Size:  31.6 KB
    London:
    Name:  5641719907.png
Views: 659
Size:  30.6 KB

    If the answer is yes, I'll shut up and accept that this is the product, but if there is something wrong, I am genuinely interested in any possible remedy.
    Last edited by Rivin; 18-09-2016 at 05:53 PM. Reason: Added speed tests

  10. #10

    Default

    Quote Originally Posted by Rivin View Post
    Thanks for responding, and apologies if my post came across aggressive.

    As requiem said, it's just frustrating getting 10% of line speed on international connections. And the experience, according to my perceptions and my tests has been worsening over time.

    I'll clarify my symptoms:
    • My PC is the only device connected to the CPE
    • I get full line speed to local resources, including speed tests, locally cached content (like youtube) and local http downloads. I understand this to mean that my line is perfectly fine, and no issues with Vumatel
    • Any download or upload sent internationally is severely hamstrung compared to local experience. I undersand this to be due to breakout to international links from CW -> IS -> Peers (please correct me if I am wrong here). Things I have tested are below.
    • HTTP downloads from North America (i.e nVidia graphics drivers) - max 7mbps
    • Linux or raspberry pi distributions from various providers in EU (HTTP downloads), max 5-7mbps
    • Speedtests done to: New York, Chicago, or West Coast - 3-7mbps varying.
    • Speed tests done to London (Softlayer as support recommends) - <8mmbps
    • Speed tests done to France/Switzerland - 5mbps
    • Unable to stream Twitch.tv in high or source (this used to be possible on this exact configuration). Threaded connections using a tool called livestreamer makes this possible for some channels (Twitch partners).
    • Other parallel international downloads tend to work far better - including torrents, or HTTP using a download manager (how 2002?)


    I have sent this all to support, via emails and in discussions on live chat.
    So just a quick note. If concurrent connections solves the issue it has nothing to do with capacity and is a TCP issue. If UDP is the same then it is more evidence of that. This is pretty much the nature of the internet (TCP) and there are some articles on this - http://blog.performancevision.com/en...nd-packet-loss and http://www.noction.com/blog/network_...t_loss_effects and shows why single threaded TCP connections will always struggle on long distance connections. It is how the internet is built and those articles explain it well.

    Twitch is one where it is up to the source, the quality of the server it delivers to from the source, and the quality of their connection from the servers to the IXP locations in London at which point ISPs take over. They don't use proper CDNs for distribution which is a pity and there is next to no local content.

    We have then also had issues with a DDoS attack for which there is no simple solution other than to employ mitigation efforts, which involve network management.

    But there is a misconception about how the internet works. The internet is dependent on your having sufficient resources to download something (bandwidth, RAM, sufficient I/O on the hardware to process the data at those rates, the protocol supporting those rates, the low latency to support the protocol, the server delivering the content to have sufficient of the exact same resources, and solid internal networking). The ISP takes responsibility for connecting you to one of those servers, at which point if we are the bottleneck you will pick it up easily by running a few simple checks.

    In short you can increase international performance using download accelerators which allow you to multi-thread connections to servers that traditionally serve a single-threaded connection. This isn't your ISP throttling or poor performance from the ISP. It's the nature of the internet and TCP. It has a buffer in the OS while it waits for packets to be acknowledged in the correct order. This means that throughput over TCP is very dependent on latency to those servers whereas things like UDP are not (but this is also why UDP is exploited to perform things like DDoS attacks). This is why when you test a torrent (UDP) it will be 100% while TCP instead requires ACK packets and so distance to the server now dictates speeds. On local resources this is not a problem as latency is fine there but the moment you have to get to USA it goes via Europe then USA (there are no direct undersea cables for commercial use between SA and USA yet). Even Europe is a sufficient distance away to cause trouble with TCP connections. We are exploring ways to optimise these for clients even further, but more often than not the bottleneck is not the ISP. So using the charts linked in the above articles you can see that a single thread over 250ms will likely only see single digit downloads, not because the ISP is shaping or throttling but because the nature of TCP requires ACK packets and therefore has a technological limit, and this is why WAN optimisation exists for businesses to deploy in their networks. There are two ways to resolve this or at least get better performance: look at things like tcp optimiser which will try to determine optimal tcp settings for your configuration; and use a multi-thread downloader where possible so you can initiate multiple TCP connections to the same server at the same time.

    There is no network management applied to our fibre accounts. Literally every protocol and every packet is treated identically. So UDP and TCP are not treated differently in terms of policy management through the network. But one single dropped packet on a tcp connection over 200ms or 250ms will massively hamper performance. More often than not the buffer fills too quickly as well, or is too small. More often than not the operating system is set for stability rather than high performance. High performance has its drawbacks too. And changing the tcp settings for high performance has its own risks dependent on your hardware and drivers and operating system. So this is not something your ISP can simply say "do x, y, and z and all will be fine, especially considering the thousands of drivers from hundreds of manufacturers out there, as well as the different tcp stacks in various operating systems and versions.

    As an ISP we take responsibility for transit from your connection's CPE to other servers. In much the same way as we cannot own and control those servers, we also cannot own, control, nor make changes to your internal network and the connected devices, and making changes there are things that you should consider by your own accord and at your discretion. We have some devices launching soon which will allow us to assist in this regard n an effort to help customers to make these changes with your permission, but they cannot get to your operating system which often plays the most important role. If the OS halves the receive window and doesn't restore it after one dropped packet, that download speed may be halved or worse for its entire duration, no matter how much bandwidth we have spare for you. The OS has already limited you, as it controls the transfer to your PC/devices. Then you should also bear in mind that very few servers can even deliver a full 100Mbps to all people connecting to it. If 3 people are downloading at once they require a dedicated or E-WAN link capable of 300Mbps, which is not cheap. You will never find "free movies and tv shows" using this sort of model. They use shared hosting which has the lowest priority on the host's outbound WAN links and splits traffic between all shared hosts accordingly. In fact most websites on the internet and most downloads reside on shared hosting, and the hosting provider shapes that traffic on a best effort basis to incoming and outgoing connections.

    There are lots of things that impact performance, and when it comes to fibre and high speeds there is an expectation that everything should come down at full line speed no matter what. Unfortunately that is simply not the reality of how the internet works, by and large, and we can say without a shadow of a doubt that we have so much spare capacity that it would be 100% pointless implementing traffic management in any way and "creating" these problems for ourselves, for zero benefit. To be frank, there is an expectation vs reality problem which creates most of the problems. It doesn't mean there are not things that we can do. We are busy investigating options for optimising certain things on customers' behalves to assist over and above just our network. And we're naturally taking a very hard line approach to any attacks on the network as well.

    I hope this helps to explain the technical side of why sometimes expectation vs reality issues are not as a result of the ISP, at least on fibre.
    Last edited by Crystal Web Management; 18-09-2016 at 06:22 PM.

  11. #11

    Default

    Quote Originally Posted by Rivin View Post
    to ISP handover:
    Name:  5641726426.png
Views: 659
Size:  31.6 KB
    London:
    Name:  5641719907.png
Views: 659
Size:  30.6 KB

    If the answer is yes, I'll shut up and accept that this is the product, but if there is something wrong, I am genuinely interested in any possible remedy.
    That's not testing to ISP handover by the way. The handover test is here http://speed1-sa.venturanext.se/

    The 4 and 14 difference immediately says there is a tcp issue at play. Fine on tcp up but not down. Two different connections made there. Is this wireless by the way? Remember if your OS says over the air 100Mbps wireless, expect 30 max on actual data throughput and then you still have to consider the post above. Have a read through my previous post and we can chat about it if need be. We're also in trials of some new ways to look at optimising some things on behalf of customers. Will see if they are ready to trial with you, perhaps.

  12. #12

    Default

    Thanks for the spectacular reply, I've learned a load from that! This may be worth a blog post - there is enough detail to make a mybb news article. I know it's not just me with these perceptions.

    I'm still a little baffled how some of the early posters on Vumatel or other fibre providers were showing speed tests to London and USE at 25-40mbps, how would they be achieve this? (I'm curious if other CW subscribers on 50 or 100mb can get better speedtest results (single thread TCP and all accounted for). I wonder if there is something specific to my PC, I'll need to do further international tests with another PC..

    To answer your questions:
    - This was done over ethernet directly from my pc. If I use a wifi router to the CPE (I know how to manage the mac addresses) and use wifi, results are the same.
    - Thanks for pointing out the the ISP handover test. For completeness, results are the same: http://netgauge.ookla.com/share/535956384.png
    - I would be more than happy to assist with trials
    - I'll have a look at TCP optimisers
    - Twitch are infamous with their poor peering, a quick search shows how widespread it is, but anything you guys could do to improve performance into them would be hugely appreciated.

    Edit: Check this thread for examples of guys hitting 50+mbps to London (100mbps accounts, but by your explanation this shouldn't matter?): http://mybroadband.co.za/vb/showthre...ps-users-help-!
    Last edited by Rivin; 18-09-2016 at 07:12 PM.

  13. #13

    Default

    Quote Originally Posted by Rivin View Post
    Thanks for the spectacular reply, I've learned a load from that! This may be worth a blog post - there is enough detail to make a mybb news article. I know it's not just me with these perceptions.

    I'm still a little baffled how some of the early posters on Vumatel or other fibre providers were showing speed tests to London and USE at 25-40mbps, how would they be achieve this? (I'm curious if other CW subscribers on 50 or 100mb can get better speedtest results (single thread TCP and all accounted for). I wonder if there is something specific to my PC, I'll need to do further international tests with another PC..

    To answer your questions:
    - This was done over ethernet directly from my pc. If I use a wifi router to the CPE (I know how to manage the mac addresses) and use wifi, results are the same.
    - Thanks for pointing out the the ISP handover test. For completeness, results are the same: http://netgauge.ookla.com/share/535956384.png
    - I would be more than happy to assist with trials
    - I'll have a look at TCP optimisers
    - Twitch are infamous with their poor peering, a quick search shows how widespread it is, but anything you guys could do to improve performance into them would be hugely appreciated.
    Oh on speedtest servers we're busy looking into why it appears slower. It's not in reality but there is something odd at play resulting in speedtests (which we simply don't use in support, we opt for real world tests) seeming slower than reality. Actually have a meeting in the coming week to address this as we've never actually cared too much about them and we don't prioritise them on our DSL network either.

  14. #14

    Default

    As for a blog post, our CEO is planning to blog about this and other subjects moving forward. There should be some really good info that comes from those blogs.

  15. #15

    Default

    Thought I would just pop in regarding twitch. On a 50/50 connection in Fibrehoods with Cool Ideas and High is as far as I can go, and that was due to some fiddling on their side. Prior to the fiddle High would buffer every 2 or 3 seconds. Source (typically 1080p) still buffers every 3-5 seconds. There is ZERO buffering on a friends 20/2 package on Vumatel with the same ISP - luck?

+ Reply to Thread
Page 1 of 12 12345 11 ... LastLast

Similar Threads

  1. Replies: 6
    Last Post: 09-07-2013, 06:25 PM
  2. Low-cost "shaped fibre" from Dark Fibre Africa
    By jes in forum Broadband and IT News
    Replies: 6
    Last Post: 11-10-2012, 09:11 PM
  3. FTTH Council slates "no more fibre deployments" moratorium
    By jes in forum Broadband and IT News
    Replies: 40
    Last Post: 23-03-2012, 10:07 AM
  4. BlackBerry on outage: "no comment"
    By QuintonB in forum Cellular: Mobile Phones, Call Contracts, Content, MMS and SMS
    Replies: 4
    Last Post: 11-10-2011, 08:15 PM
  5. DSTV loss of signal due to "solar outage"
    By bekdik in forum Off Topic
    Replies: 5
    Last Post: 31-03-2007, 11:02 AM

Bookmarks

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •