Charting Fibre speed over time.

[jacof]

Remember the nic port, if not greater than 100mb, will suffer from slower downspeeds as the upspeed (ack requests) will take up the rest. Had that happen on my mikrotik when I upgraded to 100mb line, the nic was also 100, and I only got max 93mbps because of the upload overhead

I am running a gig router https://mikrotik.com/product/RB951G-2HnD
I am also running the code on a Intel NUC that nobody uses at this time. So that is not it.

 

[fragtion]

Following this thread closely. Love what you're doing @jacof. Most people just deal with the crappy speeds because they aren't clued up or have the time/patience to do their own investigation. In principle, we shouldn't have to, but in reality, if nobody does, then the ISP's will shaft us. Please keep us updated and yes if you could share some of those scripts for the rest of us on github or something, that would be great! Looking forward to seeing some more graphs. I really like the idea of logging scheduled speedtest results in order to get a graphical representation of speed over time and will be doing the same on my line with various ISP's to see which of them are more aggressive with shaping

 

[jacof]

are you willing to share the code? I sure it will be a great starting point for others in the same situation (and are able to compile)

Yes no problem. But this code does not have any real error handling etc. I wrote it very fast just to see what is going on.
I save the data to a DB. Send it to a EMON server on my network to plot the charts and also publish it to MQTT.
But you dont have to do that.

The source ip is set as that is my gig port on the NUC as it has wifi as well.

import speedtest
import socket
import paho.mqtt.client as mqtt
import time
import os
import sys
import traceback
from datetime import datetime
import httplib
import psycopg2



#Declare the db vars
conn = psycopg2.connect("dbname='Add your DB' user='pi' host='add your host' password='password")
cur = conn.cursor()



download = 0.0
upload = 0.0
ping = 0.0
source_ip =" "
source_isp = " "
server_name = " "

# On connect Callback
def on_connect(client, userdata, flags, rc):
    print("rc: " + str(rc))


mqttc = mqtt.Client()

# Assign event callbacks
mqttc.on_connect = on_connect

# Connect
mqttc.connect("10.64.9.34", 1883,50)

#Start the mqtt loop
mqttc.loop_start()


# Write apikey of emoncms account
apikey2 = "add your key"

run = True

servers = []
source = "10.64.9.42"
speedtest.SOURCE = source
socket.socket = speedtest.bound_socket

while run:

    try:
               
        os.system('cls')       
      
        print("Set source ip : " + source)
        s = speedtest.Speedtest()
        source_ip = s.config['client'].get('ip')
        source_isp = s.config['client'].get('isp')

        print("Source ISP ip : " + source_ip)
        print("Source ISP : " + source_isp )

        print("Get the server list")
        s.get_servers(servers)
        print("Find the best server")
        s.get_best_server()
        print("Start download test")
        s.download()
        print("Start upload test")
        s.upload()

        download = s.results.download / 1000.0 / 1000.0
        upload = s.results.upload / 1000.0 / 1000.0
        server_name = s.results.server['sponsor']
        ping = s.results.ping

        print("Server : " + server_name)
        print("Ping: {0:.2f}".format(ping))
        print("Download: {0:.2f}".format(download) + " Mbit/s")
        print("Upload: {0:.2f}".format(upload) + " Mbit/s")

        #Write the values to the DB
        print("Write vales to the DB")
	cur.execute("insert into speedtest(record_timestamp , source_isp , source_ip , server_name , ping , download , upload) values (now(), %s , %s , %s , %s  , %s , %s  )" , (source_isp, source_ip, server_name , ping , download , upload ))
        conn.commit()
        

        print("Send Values via MQTT")
        mqttc.publish("Internetspeed/Download", str(round(download , 2)))
        mqttc.publish("Internetspeed/Upload", str(round(upload , 2)))


        print("Send Values via EMONCMS")
        domain = "10.64.9.34"
        emon_conn = httplib.HTTPConnection(domain,timeout = 1)
        emon_conn.request("GET", "/emoncms/input/post.json?node=internetspeed&json={ping:" +  str(round(s.results.ping , 2)) + ",download:" +   str(round(download , 2))  + ",upload:" +   str(round(upload , 2)) + "}&apikey="+apikey2)
        emon_response = emon_conn.getresponse()

        print emon_response.read()

        print("Waiting for 10 minutes")
                

        time.sleep(600)


    except KeyboardInterrupt:
        print "End"
        run = False

    except:
        traceback.print_exc()
        time.sleep(10)
        pass

 

[jacof]

You have to use the speetest-cli library.

https://github.com/sivel/speedtest-cli

Now I know they say you can use the it to find the real speed. But what I did was to monitor the Router while the test is running and I get very close to what I see on the router. So as a consistency tool it will be fine for proving there is an issue.

I have sent ICASA an email as they seem to be in control of the complaints on internet speed.

https://www.icasa.org.za/ConsumerProtection/tabid/66/Default.aspx

I just got this email from WebAfrica.

"There were changes made on your data account please perform the following .

Our network operations team has requested you to please connect directly to the ONT and create the PPPoE account using both the Web Africa account and the Telkom account , test the account and please provide us with speed test results. "

I will switch to the WebAfrica account later and see if it is any better. But the fact that they changed something tells me they know they are doing something they should not.

 

[jacof]

Following this thread closely. Love what you're doing @jacof. Most people just deal with the crappy speeds because they aren't clued up or have the time/patience to do their own investigation. In principle, we shouldn't have to, but in reality, if nobody does, then the ISP's will shaft us. Please keep us updated and yes if you could share some of those scripts for the rest of us on github or something, that would be great! Looking forward to seeing some more graphs. I really like the idea of logging scheduled speedtest results in order to get a graphical representation of speed over time and will be doing the same on my line with various ISP's to see which of them are more aggressive with shaping

I can understand that I wont get 100Mbps every time but my avg is 86 Mbps. If they drop the price with the price of a 14 Mbps line then that will be fine. I am paying for something I am not getting. I found this that basically states the same.

http://www.researchictafrica.net/do...suring_broadband_performance_South_Africa.pdf

So where does it leave us ?

 

[eddief1]

And nobody mentions IPC in this thread....it's not GPON that is the issue here

 

[cavedog]

And nobody mentions IPC in this thread....it's not GPON that is the issue here

It could be IPC no doubt but with all the drops Openserve gave do you still think it's IPC? If it is IPC then specific networks would suffer that does not have sufficient IPC.

GPON is the most likely culprit since people are going for the higher speeds like the 100Mbps and 40Mbps options especially because ADSL is painful for most people so the R200 saving on the analogue could be spent on a higher package. This is just my opinion though.

Realistically if you look at possible bandwidth bottleneck you the OLT has a relatively low max bandwidth split to so many users so one would think it's there. IPC is much much bigger and if an ISP see their links getting close to capacity would they really just leave and let people suffer.

Also when an OLT get's congested would it cause latency to spike vs congested IPC links?

 

[eddief1]

It could be IPC no doubt but with all the drops Openserve gave do you still think it's IPC? If it is IPC then specific networks would suffer that does not have sufficient IPC

.

Admittedly I have not thoroughly read this thread but...

If it was GPON OLT congestion then the OP would still be having the same issue with the Telkom guest account which he does not.

GPON is the most likely culprit since people are going for the higher speeds like the 100Mbps and 40Mbps options especially because ADSL is painful for most people so the R200 saving on the analogue could be spent on a higher package. This is just my opinion though.

As said, openserv uses a 32:1 split (or so we told), so theoretically if ALL 32 subs were on 100Mbps AND saturating their line at the same time then yes maybe there could be OLT saturation...realistically with subs spread between 10mbps and 100Mbps the chances are slim.

Realistically if you look at possible bandwidth bottleneck you the OLT has a relatively low max bandwidth split to so many users so one would think it's there. IPC is much much bigger and if an ISP see their links getting close to capacity would they really just leave and let people suffer.

Considering that openserv uses 32:1 GPON split, that would mean 32 people have to 1) have a 100Mbps line (unlikely) & 2) all be using it at the same time, which is not realistic.

follow the money...IPC costs about R10K++ for 100Mbps...so where do to ISP's contend to make $

Also when an OLT get's congested would it cause latency to spike vs congested IPC links?

Either would cause higher latency

 

[jacof]

I registered for a trail Crystal Web account yesterday. I got the logon details this morning. I was running the VOX account until 8:49 this morning. I decided to test with the Crystal Web account during the day as I only have it for 3 days. But as soon as I ran a speed test I saw they are also on the IS backbone. Aaa dammmm. So I let the test run anyway. Nobody was at home today. My wife was at school with the kids.

As you can clearly see the speed was all over the place

When I got home I decided to switch back to the VOX account at 4:28. The kids got home from school and started to watch Netflix and as you can see even with that the VOX account stays above 90 Mbps. So I now have 3 IS accounts from different ISP s (MWEB, WebAfrica and CrystalWeb) that all show the same thing. And the 2 non IS account (VOX and Telkom) stay above 90 Mbps and the pings also stay fast.

So I think this proves that IS is buggering around with my speed and it is not the Fibre or PON. At this stage it looks like I should get my Data from VOX.

When I switched over to Crystal Web

When I switched back to VOX

[video=youtube;Q5or1MovReY]https://www.youtube.com/watch?v=Q5or1MovReY[/video]

 

[jacof]

So today I did not switch to an IS account. I left it on VOX. As you can see the speed stayed up. When I get home I will switch to the WebAfrica account and see what I get. So I am happy it is not the fiber as this test shows it is not. It is all on the ISP side.

The highlighted area was when I was watching Netflix so I know the download would be lower as my PC was using that bandwidth.

 

[jacof]

I switched back to WebAfrica at 3:24. I rest my case.

 

[mister]

I switched back to WebAfrica at 3:24. I rest my case.

I see the same thing with Webafrica...

 

[jacof]

Its not just WebAfrica. It will be all ISP s that sit on the IS backbone. I just need to clear something up. I used an Openserve test account not Telkom when I tested the first time. I have been chatting with the Technical specialist from OpenServe. The problem with IS is where I circled the 3rd party area.

 

[cavedog]

So you best bet is to move ISP to a non IS network as I doubt it would get sorted. Most times it's a blame game between Openserve and ISP.

So here is some ISP's that don't use IS as far as I know.

Axxess, Afrihost, Vox, Cybersmart, Cool Ideas, Mweb, Telkom, RSAWeb, Home-Connect

 

[jacof]

MWEB is also on IS. I moved from MWEB to WebAfrica. I was with MWEB for 17 years. My Fibre was all OK until IS took them over. Then my problems started.

https://www.is.co.za/about-us/press-releases/internet-solutions-finalises-acquisition-of-mweb/

 

[rorz0r]

Interesting thread... I assumed it was my cheapy router that's been serving me well for a few years (not in the mood to set up the free one from scratch). Here's some of my results with WA on 100/50 openserve.

I guess I should be getting some test accounts and shopping around.

 

[jacof]

OK so you are seeing the same thing. If others test they should see the same. Any ISP on the IS backbone will do these drops of speed.

I made this video for IS hope to hear from them.

[video=youtube_share;tC4XLPNOc6o]https://youtu.be/tC4XLPNOc6o[/video]

 

[rorz0r]

Yeah there's only 1 month shown above but I think the max I've even seen was 86mb. It's always looked like the graph above though.

 

[infscrtyrisk]

OK so you are seeing the same thing. If others test they should see the same. Any ISP on the IS backbone will do these drops of speed.

I made this video for IS hope to hear from them.

[video=youtube_share;tC4XLPNOc6o]https://youtu.be/tC4XLPNOc6o[/video]

Good on you jacof. IS should pay you for this.

 

[Orihalcon]

Great thread! Thank you for the effort in doing all these tests. This also runs in line with my own experience using an IS account (CW) and a non IS back up (VoX).

As soon as the afternoon comes around the ping goes to absolute hell for gaming on EU servers. Switch to VoX and all good all the time. I've cancelled my CW account and will be going VoX full time going forward.