Afrihost "legacy" capped fibre issues

Deckert

Well-Known Member
Joined
Jan 13, 2004
Messages
425
Reaction score
38
Location
Centurion, South Africa
Hi,

Been a while since Afrihost introduced "pure fibre", but I thought I'd write about the issues we've been having as capped fibre users ever since the introduction of "pure fibre:

Summary:
- had several 100GB capped accounts for over two years (20Mbit line, Openserve in my case)
- worked perfectly
- since the introduction of "pure fibre", been having all sorts of issues with various protocols (except http/https)
- issues are only from around 6pm to 10pm, sometimes later ... problem goes away for all other times
- note that http/https access and ICMP continues to perform well during these times

So I believe that the Afrihost classifier is having trouble classifying certain protocols (or just not giving these protocols correct priority). For me, it is affecting specifically the following:
- standard pop3
- VPN access (Cisco IPSec client)
- upload streaming (i.e. broadcasting high quality audio for church groups)
- IP in IP tunnels (e.g. IP-ENCAP, protocol no. 4)

It's important to understand that none of the above had any issues before the introduction of "pure fibre". It should also be understood that a test account from Telkom Internet (yes, can you believe) does not have any of the issues we are experiencing on Afrihost during the 6pm-10m time period.

Why do I think it's the Afrihost classifier?

Simple experiment:

- set up a web server (local to ZA), make it listen on port 110
- do several wget requests from my AH line at 9pm, performs at full line speed
- move back to pop3 (plain text) and download a large batch of emails: barely 75kbits/sec
- move to http over port 110 again: full line speed

Further proof:

Since I have access to the server on which the web server and pop3 server is running, a simple netstat shows the following on the outbound tcp queue (server IP anonymised):

Code:
nnnnn@server:~$ netstat -atn | grep :110
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State     
tcp        0      0 0.0.0.0:110             0.0.0.0:*               LISTEN     
tcp        0 237744 1.8.28.32:110           169.0.209.234:53587     ESTABLISHED


Note the outbound queue forming. This is due to packets not being ACKd by the remote side. However, a trace on the remote side (where a client app is running behind the AH link) shows that ACK packets are indeed being sent. The only explanation is that ACK packets are being dropped by the Afrihost classifier/policies because of some rule.

The same happens with the VPN access as well as IP-ENCAP access. It used to also happen with SSH (port 22) connections that stayed open, but that seems to have been solved recently.

Anybody at AH willing to give a technically proficient answer before I start migrating all my clients away from AH?

--deckert

edit: corrected formatting of plaintext pasted text
 
Last edited:
Hi,

Been a while since Afrihost introduced "pure fibre", but I thought I'd write about the issues we've been having as capped fibre users ever since the introduction of "pure fibre:

Summary:
- had several 100GB capped accounts for over two years (20Mbit line, Openserve in my case)
- worked perfectly
- since the introduction of "pure fibre", been having all sorts of issues with various protocols (except http/https)
- issues are only from around 6pm to 10pm, sometimes later ... problem goes away for all other times
- note that http/https access and ICMP continues to perform well during these times

So I believe that the Afrihost classifier is having trouble classifying certain protocols (or just not giving these protocols correct priority). For me, it is affecting specifically the following:
- standard pop3
- VPN access (Cisco IPSec client)
- upload streaming (i.e. broadcasting high quality audio for church groups)
- IP in IP tunnels (e.g. IP-ENCAP, protocol no. 4)

It's important to understand that none of the above had any issues before the introduction of "pure fibre". It should also be understood that a test account from Telkom Internet (yes, can you believe) does not have any of the issues we are experiencing on Afrihost during the 6pm-10m time period.

Why do I think it's the Afrihost classifier?

Simple experiment:

- set up a web server (local to ZA), make it listen on port 110
- do several wget requests from my AH line at 9pm, performs at full line speed
- move back to pop3 (plain text) and download a large batch of emails: barely 75kbits/sec
- move to http over port 110 again: full line speed

Further proof:

Since I have access to the server on which the web server and pop3 server is running, a simple netstat shows the following on the outbound tcp queue (server IP anonymised):

Code:
nnnnn@server:~$ netstat -atn | grep :110
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State    
tcp        0      0 0.0.0.0:110             0.0.0.0:*               LISTEN    
tcp        0 237744 1.8.28.32:110           169.0.209.234:53587     ESTABLISHED


Note the outbound queue forming. This is due to packets not being ACKd by the remote side. However, a trace on the remote side (where a client app is running behind the AH link) shows that ACK packets are indeed being sent. The only explanation is that ACK packets are being dropped by the Afrihost classifier/policies because of some rule.

The same happens with the VPN access as well as IP-ENCAP access. It used to also happen with SSH (port 22) connections that stayed open, but that seems to have been solved recently.

Anybody at AH willing to give a technically proficient answer before I start migrating all my clients away from AH?

--deckert

edit: corrected formatting of plaintext pasted text

Hi Deckert

Sounds like you really know the subject matter, and could probably put my knowledge of networking to shame.

I am pretty confident that we haven't done anything that would affect our legacy profiles on our network management tools, so essentially those products should continue to run as they always did.

However, I am happy to put you in touch with one of our network engineers so we can assure you that everything is as it should.
 
Right, will send PM shortly. I just needed to gather some more empirical and quantitative data on the issue, which I now have. And it's definitely real and somehow related to peak usage.

Green line=good.
Purple line=bad (50% packet-loss or more).

"Cloudy" parts indicate more latency variance. All tests done on an unloaded 20Mbit/s fibre line over a VPN connection. Note that the non-VPN connection tests out fine 100% of the time.

Here is a plot for the past week, up to Saturday, of packet-loss over an IPENCAP VPN tunnel:

724192

Let's zoom in on one evening (Thursday):

724194

Those are the exact times that all the other issues also appear (i.e. SSH, IPSec VPN issues, audio-streaming issues [upload-broadcast], pop3, etc).

It's amazing how well-defined it is.

I suspect (but cannot emphasise how much it is only a suspicion) that the Afrihost network detects increased use during streaming and deprioritises the rest of the traffic so much, that it is unusable, regardless of whether there is real congestion or not.

On Saturday and Sunday the issues start earlier - clearly people start streaming stuff earlier? :-)

724202

I have no explanation for the green part that is practically in the middle of what I consider to be peak time.

--deckert
 
I've asked the technical team who are going to be in touch with you to go through your posts here as well :)
 
Top
Sign up to the MyBroadband newsletter
X