Markd

Senior Member
Joined
Oct 8, 2009
Messages
659
also struggling with download speeds/streaming/buffering issues to certain sites in the early evenings. The big ones like Youtube and Netflix seem stable, but everything else seems totally poor until later in the evening. Also on Octotel, in CT. Fibre my ass. Wishing for my ADSL line back!
 

ArtyLoop

Expert Member
Joined
Dec 18, 2017
Messages
4,579
also struggling with download speeds/streaming/buffering issues to certain sites in the early evenings. The big ones like Youtube and Netflix seem stable, but everything else seems totally poor until later in the evening. Also on Octotel, in CT. Fibre my ass. Wishing for my ADSL line back!
I just had loss of connectivity in trying to read this thread.. that's how bad its become... FTTH, what a joke..
 

TheRoDent

Expert Member
Joined
Aug 6, 2003
Messages
3,120
Good Lord, this internet sucks... can't even read a UK forum.. I am off to bed... before I have nightmares of Telkom R7 infinitcall and my old 33.6k modem... Jesus Christ...
Arty, which area on the Octotel network. Would you be willing to install a probe device on your network ?
 

TheRoDent

Expert Member
Joined
Aug 6, 2003
Messages
3,120
I don't have the energy to take it that far.. just easier to get a new ADSL line and go back to that. Or LTE, depending on how much of Telkom's shyte I have to eat in the process.
Just FYI, I have an apartment in Bantry Bay, on Octotel, and I don't see anything as terrible as you are reporting.
As per previous post, would you be willing to let me ship a device to you, which we'll put in bridge mode, so that I can give Octotel raw iperf results?
 

TheRoDent

Expert Member
Joined
Aug 6, 2003
Messages
3,120
At what point in time will I actually be allow to use the new angola (SACS) routes?
We are currently routing some traffic via Angola cables/SACS but it's a very coarsely defined route, and we have to deal with them all the time to optimize. The datacentre in Fortaleza is due to go live in April, or sooner, and then we will have better control over how routing works, once we slap down a router there, and peer directly.
 

joeline

Well-Known Member
Joined
Oct 30, 2013
Messages
144
CISP CrowdSource request: Please help us solve FNO issues by providing information.

As an ISP, we have a handover connection from Octotel, and most other FNO (Fibre Network Operators) in Teraco in Rondebosch, or JHB where our POP's are located. A "handover" is simply a fibre cross-connect from the FNO's infrastructure to us where we supply your internet connectivity from. This is similar to Telkom's old "IPC" model.

Thus far we are seeing reported issues in specific areas, which indicates that certain areas on the Octotel network might be under pressure during peak times. Back in the day of ADSL, congestion happened at exchanges on Telkom's network. It's not really that different in the Fibre space.

This is not something that Cool Ideas can control. This applies to other networks too, not just Octotel. I understand that you guys are paying CISP, and that that is your only channel...

We need solid information to build our case(s).

Why international is the most affected.

A local speedtest might seem fine, but this is simply due to the way TCP interacts with packetloss and latency. On a low-latency speedtest server, or iperf server, packetloss causes a retransmission that is so quick that the TCP window doesn't have to change. On higher latency destinations, such as to the UK, retransmissions have a huge effect on a TCP connection's performance.

Retransmissions are caused by packet loss. Packet loss is caused by congestion on a network, or a line fault. This congestion from our experience is largely due to lack of backhaul capacity by the FNO from it's urban POP to our interconnect with them in our datacentres. Occasionally there is a bad optical signal from a client to the closest aggregation point.

I can assure you, that we have more than enough international capacity. In fact our business model is so simple that it's cheaper for us to buy more international capacity, than to attempt to shape, or manage it.

We don't shape. All packets are treated exactly the same. It's the easiest and cheapest thing for us to do. Unfortunately we cannot control the peak backbone capacity of our FNO partners.

We monitor our national, and international backbone, and cross-data-centre connections vigorously, and upgrade well in advance of reaching 80% utilization on any link that's part of our infrastructure.

80% link utilization in this industry equates to congestion (as an aside).

In the case of OpenServe, they are just damn slow to process our upgrade requests, taking weeks, instead of days simply to change the rate limit on an Ethernet interface, so we pre-empt that now by just ordering another 2 Gig every month.

In the case of other FNO's that don't have a silly capacity fee (Such as Octotel and Vuma) we simply add 10Gigabit connections as we need, way in advance, to ensure that we don't have capacity problems.


How can you help?

I would like your assistance to collate, and track things a bit by filling in the spreadsheet, so that we can build cases with the particular FNO. I'm asking you guys to assist with this, because digging through thousands of tickets is a challenging task for me and PBCool, and if we can have accurate information collated by an intelligent crowd it will save us many hours...

I'm working on a "1 shot" iperf/powershell script that will output some useful data that we'll add into a final column. In the meantime please just populate as you experience issues on the spreadsheet.

On Octotel we have varying experiences. Seapoint and Bantry Bay is fine, for example, but other areas are struggling.

Here is the link to the google docs sheet :
https://docs.google.com/spreadsheets/d/16rzVo64z-rP2qMAEil5LwqTgqEGGPxBEgp-0XUew1Pk/edit?usp=sharing

The spreadsheet is a work in progress. More columns might be added requesting further diagnostic info.

Please feel free to log multiple rows per day, this will help us to track things a bit better.

I'm not adding columns for b*s like Cabled, or Wifi, because I assume that you folk, being of the somewhat technical persuasion will use cabled tests for everything.

What you can expect..

Our relationship with FNO's like Telkom ADSL back in the day, is the same as any other provider. We can report slow speeds on an individual line basis to them through their support channels, as that is typically all they allow in their interactions with them.

Realistically they don't pick up trends. We have to typically supply a body of evidence before it gets escalated to senior engineers.

PBCool and I have good relationships with the CTO's of most of the FNO's but sometimes they are overwhelmed too, with the noise, and they also have complex internal structures to deal with.....

We work very hard to try and give hard, accurate information to them to resolve issues, but it's not a case of "log a ticket, and sort out the line" when it comes to congestion and capacity issues.

I've spent nearly a year of my life dealing with TTConnect to no resolve, which is why we decided to drop them entirely.

I understand the frustration of logging a ticket with CISP, and getting canned responses, or "it has been escalated to senior engineers". Take it from me, when it gets escalated both PBCool and myself are aware of it. It's just not that simple to fix when you are dealing with a "black box" network, and you have to argue for weeks to get things resolved.

We also cannot reply to each of you personally, as that would just take time away from doing more important things like resolving the issue. Here, and there, we will personally deal with an issue, this forum being one...

There are two people in this thread, dealing with you. It's me and Paul. We are both Directors at Cool Ideas. We are both the "senior network engineers".

This thread is the best technical support you can ever get. But we cannot answer every single post. If you want that kind of "feel-good" then paste on facebook or log a ticket.

I am a network engineer. A shitty network makes me sad. I take pride in building good networks. I have a connection on Octotel in Cape Town, and Vuma in Joburg, and when it sucks, I try and figure out why, because I, just like you, want a good internet connection.

Unfortunately, our network doesn't always translate into a good experience when having to deal with third party networks that have problems.

I'm sure you've seen in this thread that there are people entirely happy with the service they are receiving versus people having issues. Believe me, we do not discriminate. We want all our clients happy. It's just that it's not always that simple...

The information you provide will help us to build a case with the FNO, and escalate things beyond single tickets in a ticketing system. From experience, this has helped us to get FNO's to admit that they had capacity issues, and it was addressed within days.



We appreciate your assistance.

I'm contemplating a reward system, for forumites that assist us in resolving the situation. Suggestions are welcome.
What about the people that are not reading this on Mbb? Can you not send out an email to all your Octotel customers?
 

ArtyLoop

Expert Member
Joined
Dec 18, 2017
Messages
4,579
Just FYI, I have an apartment in Bantry Bay, on Octotel, and I don't see anything as terrible as you are reporting.
As per previous post, would you be willing to let me ship a device to you, which we'll put in bridge mode, so that I can give Octotel raw iperf results?
Yip. Willing to assist.
I am not exaggerating when I say I am having issues, and they also magically disappear after 10pm
I am also very careful when I say something is wrong i.e. I make sure there's no problem in my home with a router etc..
 

ArtyLoop

Expert Member
Joined
Dec 18, 2017
Messages
4,579
Finally a decent response. And at an ungodly hour.
After reading all of this.. my respect level for CISP just went all the way to the top.
No other ISP in South Africa will go the extra mile like this for you. Take note people... CISP are the good guys.
It took time but I suspect that an internal investigation had to be completed first, and I guess feedback about that could have been a bit more forthcoming but yes, I will give Paul and Roelf the benefit of the doubt here.
 

Praeses

Expert Member
Joined
Oct 29, 2005
Messages
4,665
CISP CrowdSource request: Please help us solve FNO issues by providing information.

As an ISP, we have a handover connection from Octotel, and most other FNO (Fibre Network Operators) in Teraco in Rondebosch, or JHB where our POP's are located. A "handover" is simply a fibre cross-connect from the FNO's infrastructure to us where we supply your internet connectivity from. This is similar to Telkom's old "IPC" model.

Thus far we are seeing reported issues in specific areas, which indicates that certain areas on the Octotel network might be under pressure during peak times. Back in the day of ADSL, congestion happened at exchanges on Telkom's network. It's not really that different in the Fibre space.

This is not something that Cool Ideas can control. This applies to other networks too, not just Octotel. I understand that you guys are paying CISP, and that that is your only channel...

We need solid information to build our case(s).

Why international is the most affected.

A local speedtest might seem fine, but this is simply due to the way TCP interacts with packetloss and latency. On a low-latency speedtest server, or iperf server, packetloss causes a retransmission that is so quick that the TCP window doesn't have to change. On higher latency destinations, such as to the UK, retransmissions have a huge effect on a TCP connection's performance.

Retransmissions are caused by packet loss. Packet loss is caused by congestion on a network, or a line fault. This congestion from our experience is largely due to lack of backhaul capacity by the FNO from it's urban POP to our interconnect with them in our datacentres. Occasionally there is a bad optical signal from a client to the closest aggregation point.

I can assure you, that we have more than enough international capacity. In fact our business model is so simple that it's cheaper for us to buy more international capacity, than to attempt to shape, or manage it.

We don't shape. All packets are treated exactly the same. It's the easiest and cheapest thing for us to do. Unfortunately we cannot control the peak backbone capacity of our FNO partners.

We monitor our national, and international backbone, and cross-data-centre connections vigorously, and upgrade well in advance of reaching 80% utilization on any link that's part of our infrastructure.

80% link utilization in this industry equates to congestion (as an aside).

In the case of OpenServe, they are just damn slow to process our upgrade requests, taking weeks, instead of days simply to change the rate limit on an Ethernet interface, so we pre-empt that now by just ordering another 2 Gig every month.

In the case of other FNO's that don't have a silly capacity fee (Such as Octotel and Vuma) we simply add 10Gigabit connections as we need, way in advance, to ensure that we don't have capacity problems.


How can you help?

I would like your assistance to collate, and track things a bit by filling in the spreadsheet, so that we can build cases with the particular FNO. I'm asking you guys to assist with this, because digging through thousands of tickets is a challenging task for me and PBCool, and if we can have accurate information collated by an intelligent crowd it will save us many hours...

I'm working on a "1 shot" iperf/powershell script that will output some useful data that we'll add into a final column. In the meantime please just populate as you experience issues on the spreadsheet.

On Octotel we have varying experiences. Seapoint and Bantry Bay is fine, for example, but other areas are struggling.

Here is the link to the google docs sheet :
https://docs.google.com/spreadsheets/d/16rzVo64z-rP2qMAEil5LwqTgqEGGPxBEgp-0XUew1Pk/edit?usp=sharing

The spreadsheet is a work in progress. More columns might be added requesting further diagnostic info.

Please feel free to log multiple rows per day, this will help us to track things a bit better.

I'm not adding columns for b*s like Cabled, or Wifi, because I assume that you folk, being of the somewhat technical persuasion will use cabled tests for everything.

What you can expect..

Our relationship with FNO's like Telkom ADSL back in the day, is the same as any other provider. We can report slow speeds on an individual line basis to them through their support channels, as that is typically all they allow in their interactions with them.

Realistically they don't pick up trends. We have to typically supply a body of evidence before it gets escalated to senior engineers.

PBCool and I have good relationships with the CTO's of most of the FNO's but sometimes they are overwhelmed too, with the noise, and they also have complex internal structures to deal with.....

We work very hard to try and give hard, accurate information to them to resolve issues, but it's not a case of "log a ticket, and sort out the line" when it comes to congestion and capacity issues.

I've spent nearly a year of my life dealing with TTConnect to no resolve, which is why we decided to drop them entirely.

I understand the frustration of logging a ticket with CISP, and getting canned responses, or "it has been escalated to senior engineers". Take it from me, when it gets escalated both PBCool and myself are aware of it. It's just not that simple to fix when you are dealing with a "black box" network, and you have to argue for weeks to get things resolved.

We also cannot reply to each of you personally, as that would just take time away from doing more important things like resolving the issue. Here, and there, we will personally deal with an issue, this forum being one...

There are two people in this thread, dealing with you. It's me and Paul. We are both Directors at Cool Ideas. We are both the "senior network engineers".

This thread is the best technical support you can ever get. But we cannot answer every single post. If you want that kind of "feel-good" then paste on facebook or log a ticket.

I am a network engineer. A shitty network makes me sad. I take pride in building good networks. I have a connection on Octotel in Cape Town, and Vuma in Joburg, and when it sucks, I try and figure out why, because I, just like you, want a good internet connection.

Unfortunately, our network doesn't always translate into a good experience when having to deal with third party networks that have problems.

I'm sure you've seen in this thread that there are people entirely happy with the service they are receiving versus people having issues. Believe me, we do not discriminate. We want all our clients happy. It's just that it's not always that simple...

The information you provide will help us to build a case with the FNO, and escalate things beyond single tickets in a ticketing system. From experience, this has helped us to get FNO's to admit that they had capacity issues, and it was addressed within days.



We appreciate your assistance.

I'm contemplating a reward system, for forumites that assist us in resolving the situation. Suggestions are welcome.
Thanks for this. I've created two lines with my entry into the document. One for general international performance and one for single threaded performance issues. Hope that's OK. The issue is probably related, but just in case it isn't :)
 

ryanrich

Well-Known Member
Joined
Aug 4, 2011
Messages
233
After reading all of this.. my respect level for CISP just went all the way to the top.
No other ISP in South Africa will go the extra mile like this for you. Take note people... CISP are the good guys.
It took time but I suspect that an internal investigation had to be completed first, and I guess feedback about that could have been a bit more forthcoming but yes, I will give Paul and Roelf the benefit of the doubt here.
Agree completely. People get angry as they're paying for a service so it's extremely frustrating when it doesn't work as it should, but I honestly believe CISP is doing all they can do resolve these issues with the fibre providers.
 
Top