My line speed for at least the 2nd time now is half of what it should be

[Zuldritch]

I've just noticed that on speedtest.net I'm getting 50Mbps instead of 100Mbps. I'm connected by Gigabit/s ethernet cable to my router, so it's not anything like WiFi interference.

I had this happen late last year (which I noticed about a week or so after that Gauteng wide Cool Ideas/OpenServe outage). I thought that it may be related to the outage, so I powered down both the router and the OpenServe unit, powered them back up and it was back to full speed again.

The reason I did a speed test and noticed the problem at the time is because I was having issues with Teams call quality and web pages loading when the line was fully utilized, like for example due to a big HTTP download or torrenting. I had never had slow web pages or bad quality Teams before when the line was fully utilized (but operating at full instead of half speed).

This is pure conjecture, but perhaps because some upstream equipment from me thinks it should be able to send twice as much data compared to what's actually getting through, it's using on oversized buffer causing packets to drop unpredictably.

And I only happened to notice the issue now as I was having a call quality issue with a colleague and checked my line speed just in case. Turns out it was an issue with their WiFi on their side, but I noticed I'm at half line speed again.

I suspect if I reset my equipment again then everything will be fixed again, but I am worried now that this is a recurring issue and don't want to find myself getting affected by it during a Teams presentation to a client, so have left it as is for the moment in case there is anything you can check from your side before I reset it.

In case you need details on my account, I've also just created ticket #COOL-20250107-2286691.

 

[CoolEscalator]

I suspect if I reset my equipment again then everything will be fixed again, but I am worried now that this is a recurring issue and don't want to find myself getting affected by it during a Teams presentation to a client, so have left it as is for the moment in case there is anything you can check from your side before I reset it.

In case you need details on my account, I've also just created ticket #COOL-20250107-2286691.

HI , thanks for reaching out
Are you able to enable ICPM / Respond to ping on your router to allow me to run some tests to your IP?
If you could also advise what kind of router you are using? Or is it the same one we supplied in 2018?

 

[Zuldritch]

It's a PC running PFSense.

ICMP traffic is now getting through, tested by tethering from my phone.

 

[CoolEscalator]

It's a PC running PFSense.

ICMP traffic is now getting through, tested by tethering from my phone.

Thanks, busy running tests now.

So it could be just electronics being electronics, needing a reboot every now and then.
If a reboot of the router resolves the issue, then this is the likely scenario

Have you tried implimenting a timed restart in the early hours of the morning? So that you never have to worry about a rebooting the equipment.

You could also isolate it further, by bypassing the Pfsense router, and connecting via Dial up with your PC, to see if the same symptoms show.

In my tests, I did see some latency spikes and jitter, but this could also have been a speed tests on your end congesting the link.
But no packet loss observed. But the latency spikes are more frequent than I would like though.



I also did some pathping tests , and seems clean as well, nothing standing out.
Are you able to do the same test from your end?

pathping 1.1.1.1 -n

pathping 155.XX.XX.XX -n


Tracing route to 155.XX.XX.XX.cust.cisp.co.za [155.XX.XX.XX]
over a maximum of 30 hops:
  0  102.XX.XX.XX
  1  102.XX.XX.XX
  2  100.99.22.32
  3  100.99.1.246
  4  155.XX.XX.XX


Computing statistics for 100 seconds...
            Source to Here   This Node/Link
Hop  RTT    Lost/Sent = Pct  Lost/Sent = Pct  Address
  0                                           102.XX.XX.XX
                                0/ 100 =  0%   |
  1    0ms     0/ 100 =  0%     0/ 100 =  0%  102.XX.XX.XX
                                0/ 100 =  0%   |
  2    0ms     0/ 100 =  0%     0/ 100 =  0%  100.99.22.32
                                0/ 100 =  0%   |
  3    0ms     0/ 100 =  0%     0/ 100 =  0%  100.99.1.246
                                0/ 100 =  0%   |
  4    2ms     0/ 100 =  0%     0/ 100 =  0%  155.XX.XX.XX


Trace complete.

 

[Zuldritch]

It's been running perfectly with this exact same equipment for over a year, so it's very weird there is this intermittent issue now.

I restarted just the firewall and it's back to full speed again.

Before: https://www.speedtest.net/result/17221055514
Now: https://www.speedtest.net/result/17221144767

I'll try test it daily to see if it happens again and next time I'll try to restart the ONT and see if that's at all different. I've noticed that the PPPoE connection restarts itself after a number of days regardless (7 days uptime is a lot in my experience), I'd be interested to know if the issue is after one of these restarts or happens in between them.

 

[CoolEscalator]

It's been running perfectly with this exact same equipment for over a year, so it's very weird there is this intermittent issue now.

I restarted just the firewall and it's back to full speed again.

Before: https://www.speedtest.net/result/17221055514
Now: https://www.speedtest.net/result/17221144767

I'll try test it daily to see if it happens again and next time I'll try to restart the ONT and see if that's at all different. I've noticed that the PPPoE connection restarts itself after a number of days regardless (7 days uptime is a lot in my experience), I'd be interested to know if the issue is after one of these restarts or happens in between them.

Thanks for confirming its back to normal.
Yeah issues such as these are difficult and time consuming to troubleshoot, especially if you have to wait for something to happen again.
But perhaps setting up a timed reboot every day , say 1 or 2 am, would make this something you never really have to worry about.
Could also be Weekly if you want?

I've noticed that the PPPoE connection restarts itself after a number of days regardless

Yes we have timed restarts on our AR's as well , which is likely the cause

 

[CoolEscalator]

I'd be interested to know if the issue is after one of these restarts or happens in between them.

Should be unlikely for "after", as I see your session was up for about 12.5 days before you rebooted now.( So not the case for this instance that is.)

 

[TheRoDent]

It's been running perfectly with this exact same equipment for over a year, so it's very weird there is this intermittent issue now.

I restarted just the firewall and it's back to full speed again.

Before: https://www.speedtest.net/result/17221055514
Now: https://www.speedtest.net/result/17221144767

I'll try test it daily to see if it happens again and next time I'll try to restart the ONT and see if that's at all different. I've noticed that the PPPoE connection restarts itself after a number of days regardless (7 days uptime is a lot in my experience), I'd be interested to know if the issue is after one of these restarts or happens in between them.

Hmm, it doesn't make sense that restarting the firewall would suddenly make more bandwidth available, especially if the ONT wasn't cycled.

Does reconnecting the PPPoE on pfsense achieve the same? Normal speeds? Or is it specifically a reboot of the device ?

When this "halving" happens can you check that there is no other unexpected traffic on the pfsense PPPoE interface? Perhaps some background traffic ?

 

[Zuldritch]

Should be unlikely for "after", as I see your session was up for about 12.5 days before you rebooted now.( So not the case for this instance that is.)

It's possible that it's been a problem for 12 days, but I haven't been using the line heavily enough recently that I would necessarily notice that it's not at full speed. 50Mbps is still pretty fast for a lot of things.

But perhaps setting up a timed reboot every day , say 1 or 2 am, would make this something you never really have to worry about.
Could also be Weekly if you want?

Feels like a cludge, which I'll resort to it if we can't get to the bottom of what's happening, but I prefer the longest uptimes possible (without having to pay extra for a static IP or something).

It would have to be a nightly restart if I want to minimize the chance or it affecting my work calls.

Does reconnecting the PPPoE on pfsense achieve the same? Normal speeds? Or is it specifically a reboot of the device ?

This is only the second time I've noticed this happening, will do different tests if I notice again.

When this "halving" happens can you check that there is no other unexpected traffic on the pfsense PPPoE interface? Perhaps some background traffic ?

One of the things I like about pfSense is being able to see live interface traffic graphs, so I am absolutely sure there was no other significant traffic at the time.

 

[Zuldritch]

Hmm, it doesn't make sense that restarting the firewall would suddenly make more bandwidth available, especially if the ONT wasn't cycled.

This is exactly why I posted this here, it's weird, especially that it's to a round number of essentially 50Mbps which is the size of an OpenServe package. My suspicion is it's something on OpenServe's side, the ONT, or (or possibly yours).

 

[supersunbird]

Who owns the router? It might be that it's not lekker anymore. I had one that would get hot and them it's performance would degrade.

 

[TheRoDent]

This is exactly why I posted this here, it's weird, especially that it's to a round number of essentially 50Mbps which is the size of an OpenServe package. My suspicion is it's something on OpenServe's side, the ONT, or (or possibly yours).

We don't dynamically change the queue on our BRAS. The queue profile is assigned during first log-in of the PPPoE session, and if upgrades/downgrades are done, the session is disconnected and the new profile applies again on first login.

The ONT provisioning is also unlikely to change unless we use OpenServe's portal to reprovision or upgrade/downgrade the line.

It doesn't make sense that a restart of your firewall changes it, hence me asking to check if just a restart of the PPPoE session on the pfsense has the same effect. Let us know what your testing shows the next time it happens.

If possible, check which of our BRAS's you are connected to, this would be the "AC" or "Access Concentrator". I'm not sure if pfsense will show that, it might show up in the logs.

 

[Zuldritch]

If possible, check which of our BRAS's you are connected to, this would be the "AC" or "Access Concentrator". I'm not sure if pfsense will show that, it might show up in the logs.

pfSense logs this!

All the entries since the 13th has the same name, here is the previous "slow" and today's "fast" session:

Dec 26 03:37:56	PPPoE: rec'd ACNAME "rrba-ip-bng-2"
Jan 7 16:59:04	PPPoE: rec'd ACNAME "rrba-ip-bng-2"

It also occurred to me I can use the traffic graphs to see historic traffic, and it seems that it can go slow mid-session:

It doesn't make sense that a restart of your firewall changes it, hence me asking to check if just a restart of the PPPoE session on the pfsense has the same effect. Let us know what your testing shows the next time it happens.

Will do.

Who owns the router? It might be that it's not lekker anymore. I had one that would get hot and them it's performance would degrade.

It's a very small form factor PC running pfSense, I'm very doubtful the hardware is an issue, but you never know:


(That temperature is normal as it's a passively cooled CPU)

 

[TheRoDent]

pfSense logs this!

All the entries since the 13th has the same name, here is the previous "slow" and today's "fast" session:

Dec 26 03:37:56	PPPoE: rec'd ACNAME "rrba-ip-bng-2"
Jan 7 16:59:04	PPPoE: rec'd ACNAME "rrba-ip-bng-2"

It also occurred to me I can use the traffic graphs to see historic traffic, and it seems that it can go slow mid-session:

View attachment 1786005


Will do.

It's a very small form factor PC running pfSense, I'm very doubtful the hardware is an issue, but you never know:

View attachment 1786016
(That temperature is normal as it's a passively cooled CPU)

Ah yes, I forgot that on openserve you will only see the OpenServe BRAS, which is then forwarded to our BRAS side.

If you want to test our new L2TP based infrastructure, rather than the old IPC infrastructure just change the domain in your PPPoE username to "coolideas.co.za" rather than "coolideas.co" and see if the same occurs? That is a more direct way of connecting to our infrastructure, and you should also be able to get an IPv6 address if you're interested at all.

 

[Zuldritch]

Thanks, I changed the username and it seems IPv6 is working.

Will the IPv6 range assigned always be static for my account?

For anyone else wondering about the settings, after trying a few different things, eventually got to the following which worked:

 

[TheRoDent]

Yes, requesting a delegated IPV6 range via DHCP6 prefix delegation is the right way. At this point it's not guaranteed to be static. We're working on that.

 

[CoolEscalator]

Thanks, I changed the username and it seems IPv6 is working.

Will the IPv6 range assigned always be static for my account?

For anyone else wondering about the settings, after trying a few different things, eventually got to the following which worked:
View attachment 1786054

The DP size here would be /56 or 56. as we assign this size now. I believe a while back it was /60

 

[Zuldritch]

This seems to have happened again, some time between about 7pm last night and now it dropped to ~50Mbps.

Simply restarting the PPPoE session on my pfSense router was enough to get it back to full speed:

 

[TheRoDent]

Doesn't make any sense, we wouldn't vary the speeds on your PPPoE session dynamically, but if you could PM me a Coolzone account, or e-mail address, I will try and see what profile our BRAS sent you at the time. You don't maybe have windows updates, or some downloads scheduled?

 

[Zuldritch]

Sent.