CISP CrowdSource request: Please help us solve FNO issues by providing information

TheRoDent

Expert Member
Joined
Aug 6, 2003
Messages
3,394
#1
CISP CrowdSource request: Please help us solve FNO issues by providing information.

As an ISP, we have a handover connection from Octotel, and most other FNO (Fibre Network Operators) in Teraco in Rondebosch, or JHB where our POP's are located. A "handover" is simply a fibre cross-connect from the FNO's infrastructure to us where we supply your internet connectivity from. This is similar to Telkom's old "IPC" model.

Thus far we are seeing reported issues in specific areas, which indicates that certain areas on the Octotel network might be under pressure during peak times. Back in the day of ADSL, congestion happened at exchanges on Telkom's network. It's not really that different in the Fibre space.

This is not something that Cool Ideas can control. This applies to other networks too, not just Octotel. I understand that you guys are paying CISP, and that that is your only channel...

We need solid information to build our case(s).

Why international is the most affected.

A local speedtest might seem fine, but this is simply due to the way TCP interacts with packetloss and latency. On a low-latency speedtest server, or iperf server, packetloss causes a retransmission that is so quick that the TCP window doesn't have to change. On higher latency destinations, such as to the UK, retransmissions have a huge effect on a TCP connection's performance.

Retransmissions are caused by packet loss. Packet loss is caused by congestion on a network, or a line fault. This congestion from our experience is largely due to lack of backhaul capacity by the FNO from it's urban POP to our interconnect with them in our datacentres. Occasionally there is a bad optical signal from a client to the closest aggregation point.

I can assure you, that we have more than enough international capacity. In fact our business model is so simple that it's cheaper for us to buy more international capacity, than to attempt to shape, or manage it.

We don't shape. All packets are treated exactly the same. It's the easiest and cheapest thing for us to do. Unfortunately we cannot control the peak backbone capacity of our FNO partners.

We monitor our national, and international backbone, and cross-data-centre connections vigorously, and upgrade well in advance of reaching 80% utilization on any link that's part of our infrastructure.

80% link utilization in this industry equates to congestion (as an aside).

In the case of OpenServe, they are just damn slow to process our upgrade requests, taking weeks, instead of days simply to change the rate limit on an Ethernet interface, so we pre-empt that now by just ordering another 2 Gig every month.

In the case of other FNO's that don't have a silly capacity fee (Such as Octotel and Vuma) we simply add 10Gigabit connections as we need, way in advance, to ensure that we don't have capacity problems.


How can you help?

I would like your assistance to collate, and track things a bit by filling in the spreadsheet, so that we can build cases with the particular FNO. I'm asking you guys to assist with this, because digging through thousands of tickets is a challenging task for me and PBCool, and if we can have accurate information collated by an intelligent crowd it will save us many hours...

I'm working on a "1 shot" iperf/powershell script that will output some useful data that we'll add into a final column. In the meantime please just populate as you experience issues on the spreadsheet.

On Octotel we have varying experiences. Seapoint and Bantry Bay is fine, for example, but other areas are struggling.

Here is the link to the google docs sheet :
https://docs.google.com/spreadsheets/d/16rzVo64z-rP2qMAEil5LwqTgqEGGPxBEgp-0XUew1Pk/edit?usp=sharing

The spreadsheet is a work in progress. More columns might be added requesting further diagnostic info.

Please feel free to log multiple rows per day, this will help us to track things a bit better.

I'm not adding columns for b*s like Cabled, or Wifi, because I assume that you folk, being of the somewhat technical persuasion will use cabled tests for everything.

What you can expect..

Our relationship with FNO's like Telkom ADSL back in the day, is the same as any other provider. We can report slow speeds on an individual line basis to them through their support channels, as that is typically all they allow in their interactions with them.

Realistically they don't pick up trends. We have to typically supply a body of evidence before it gets escalated to senior engineers.

PBCool and I have good relationships with the CTO's of most of the FNO's but sometimes they are overwhelmed too, with the noise, and they also have complex internal structures to deal with.....

We work very hard to try and give hard, accurate information to them to resolve issues, but it's not a case of "log a ticket, and sort out the line" when it comes to congestion and capacity issues.

I've spent nearly a year of my life dealing with TTConnect to no resolve, which is why we decided to drop them entirely.

I understand the frustration of logging a ticket with CISP, and getting canned responses, or "it has been escalated to senior engineers". Take it from me, when it gets escalated both PBCool and myself are aware of it. It's just not that simple to fix when you are dealing with a "black box" network, and you have to argue for weeks to get things resolved.

We also cannot reply to each of you personally, as that would just take time away from doing more important things like resolving the issue. Here, and there, we will personally deal with an issue, this forum being one...

There are two people in this thread, dealing with you. It's me and Paul. We are both Directors at Cool Ideas. We are both the "senior network engineers".

This thread is the best technical support you can ever get. But we cannot answer every single post. If you want that kind of "feel-good" then paste on facebook or log a ticket.

I am a network engineer. A shitty network makes me sad. I take pride in building good networks. I have a connection on Octotel in Cape Town, and Vuma in Joburg, and when it sucks, I try and figure out why, because I, just like you, want a good internet connection.

Unfortunately, our network doesn't always translate into a good experience when having to deal with third party networks that have problems.

I'm sure you've seen in this thread that there are people entirely happy with the service they are receiving versus people having issues. Believe me, we do not discriminate. We want all our clients happy. It's just that it's not always that simple...

The information you provide will help us to build a case with the FNO, and escalate things beyond single tickets in a ticketing system. From experience, this has helped us to get FNO's to admit that they had capacity issues, and it was addressed within days.



We appreciate your assistance.

I'm contemplating a reward system, for forumites that assist us in resolving the situation. Suggestions are welcome.
 
Last edited:

Meester

Active Member
Joined
Feb 28, 2007
Messages
52
#2
Why not get and deploy some RIPE Atlas probes to your customers in the areas that is of concern, and then you can use that for other graphings etc. too!
 
Top