Internet providers caught inflating speed test results

Electron1

Expert Member
Joined
Jan 29, 2009
Messages
4,229
Reaction score
199
Location
Joburg
http://www.myce.com/review/internet-providers-caught-inflating-speed-test-results-78458/
A work colleague in the UK called me one day to say that his ISP Pulse8 Broadband installed their Fibreoptic service at his house. He went for their 38Mbps package, but suspected wasn’t getting the full speed and asked for my advice on testing his connection. Like most ISPs, Pulse8 recommended checking his speed with its speed test on the Pulse8 website. This speed test is powered by Ookla, the company behind the well-known speedtest.net website.

Personally I prefer the TestMy speed test service for a number of reasons. It performs its test by measuring the time it takes for one or more blocks to download or upload, just like timing a download or upload with a stopwatch to work out its transfer speed. It also runs its test over a single connection to the server, just like streaming or downloading a large file.

So I asked him to run a few download and upload tests through testmy.net using their UK server, to see how its test results compare. He came back to me saying he is only getting about 4.5Mbps on TestMy’s download test compared to around 20Mbps with both Pulse8’s website speed test and speedtest.net. I asked him to this time try a multithread test on TestMy, but that only delivered 7.4Mbps. While I was on a VoIP call to him, that multithread test severely degraded the VoIP call quality which did not happen when running a test on speedtest.net, so TestMy was clearly maxing out his downlink bandwidth despite its lower test result. This got me thinking – Just what is speedtest.net doing differently with his connection to achieve around 20Mbps and without even affecting the VoIP call quality?

As explained in the first post of this thread, Ookla’s Speedtest.net uses a rather complex testing process, so in theory, the ISP could fiddle with the traffic flow to trick the speed test into delivering an inflated figure. One important point mentioned is that speedtest.net makes multiple connections (also called threads) to the server. Straight away, this has the potential to deliver a much higher result than let’s say streaming video where most video on demand services only make a single connection to their content server. As I mentioned already, he got 7.4Mbps down in the TestMy multithread test, which is significantly higher than the 4.5Mbps result in the standard TestMy result, but still well short of what speedtest.net reported.
....

He ran the regular HTTP multithread test a handful of times and not one test result exceeded 8Mbps. So it is obvious that either his ISP or the Talktalk network it resells has given port 8080 special treatment. As Ookla’s speed test service operates over port 8080, it is pretty clear how it was able to deliver test results that were not possible to achieve with real-world web usage such as downloading large files. In fact, this inflated 38Mbps figure turned out to be the ISP’s advertised speed, which as we can see can only be achieved under specific test conditions, just like the VW emissions test scandal.

After discussing this with the Myce staff, our founder Jan Willem discovered something similar with his ISP, Ziggo, the largest cable operator in the Netherlands. The following is the ordinary HTTP test on the left and port 8080 on the right. Unlike my UK work colleague, this was not even a multithreaded test:
....
In fact, the only way to take advantage of that throughput is to find a way of getting it to come in over port 8080, which obviously requires the server to accept connections on port 8080. As I mentioned in the introduction, very little real world traffic is carried over port 8080, making this additional throughput practically useless to the user other than to inflate their speed test results.

Running your own tests
Let’s start by running TestMy with a single connection to its test server:

Go to http://testmy.net/mirror and click [Set Default] for the test server nearest to you. The yellow star indicates the chosen test server.
Click the “Test My Internet+” button towards the top right and click ‘Combined’.
Now let’s repeat the test over port 8080:

Go to http://testmy.net:8080/mirror and click [Set Default] for nearest test server to you. The yellow star indicates the chosen test server.
Click the “Test My Internet+” button towards the top right and click ‘Combined’.
If the port 8080 download or upload result figures are considerably higher, then it’s likely your ISP is giving port 8080 special treatment. In this case, repeat the above tests another time, to rule out a fluke.

If neither test result is close to what you typically get on speedtest.net, then perform the following multithread tests:

Go to http://testmy.net/multithread and click the ‘Enable Multithreading’ button.
Tick the nearest test server to you. Ensure the rest of the check boxes are clear.
Click the ‘Test My Internet’ button below the list to run the multithread test.
At the time of publishing, TestMy does not support multithread upload tests, so the test result figure is for the downlink only.

Now let’s repeat the multithread test over port 8080:

Go to http://testmy.net:8080/multithread. If it does not show the server list, then click the “Enable Multithreading” button again.
Tick the nearest test server to you. Ensure the rest of the check boxes are clear.
Click the ‘Test My Internet’ button below the list to run the multithread test over 8080.
Note: If you use any web caching or acceleration plug-in such as “Data Saver” in the Chrome browser, this must be disabled before running any tests.

Interpreting the test results
If the ISP is behaving properly, the two single thread tests should be very similar and the two multithread test results should also be very alike. If they vary by much, then repeat the two multithread tests again. If the port 8080 multithread test result is consistently quicker than the regular multithread test, then the ISP likely has port 8080 configured to only deliver additional bandwidth with multiple connections running over it, like what I’ve seen with Pulse8 broadband (which resells Talktalk).

Finally, if the multithread test results are considerably quicker than the non-multithread test results, even for the standard HTTP port, then this is a good indication of high contention on the network. For example, high contention on a fixed wireless service typically results in significant speed variations between running a standard test and a multithread test, even without any significant difference from the equivalent tests run over port 8080.
 
I got 1.43Mbps on Speedtest and 1.5Mbps on TestMy. Consistent results for an awful line speed...
 
Top
Sign up to the MyBroadband newsletter
X