Averages are a handy statistical tool, but the context of the average is important to understand what it measures.
Cybersmart CTO Laurie Fialkov told MyBroadband this is especially true for the indices that rank Internet service providers by their average download speeds.
“The problem with averages is that it is difficult to claim something is the fastest because it has the highest average speed,” said Fialkov.
Referring to Netflix’s ISP Speed Index, Fialkov said it would be more useful if the streaming service gave separate speed indices for 360p, 720p, HD, and UHD content.
Netflix Speed Index
Netflix produces a report every month which shows the top 10 ISPs in a region, and compares several of the regions where it operates to one another.
Currently South Africa, is ranked 50 out of the 59 territories measured.
The index is a measure of prime-time Netflix performance on particular ISPs, and not a measure of overall performance of the networks.
Netflix said it calculates the average bitrate of Netflix content streamed by its members per ISP.
“We measure the speed via all available end-user devices. For a small number of devices, we cannot calculate the exact bit rates, and streaming via cellular networks is exempted from our measurements,” said Netflix.
Netflix recommends a 5Mbps connection for full HD streaming, and its top profile for 1080p content encoding uses a bitrate of 4,300kbps.
It should therefore not be surprising that the average speeds for South African ISPs in its index are all below the 5Mbps mark.
What the averages hide
“When you start lumping services together – 360p, 720p, HD, and UHD – and work out an average, what is the relevance of the index and how does it assist the consumer in making a decision?” said Fialkov.
“I am not sure what it actually shows or what benefit it has.”
A fibre-only ISP which offers uncapped packages, for example, would have an advantage over ISPs like Cybersmart – which has a subscriber base on slower DSL services.
Netflix does indicate what kind of services the ISPs in its index offer, but it does not segment the average performance by the encoded bitrate of the content being streamed, as Fialkov suggested.
It is interesting to note that Vox, however, which offers a variety of services – including fibre, DSL, wireless, and satellite – has been consistently near the top of the index.
This may mean that Vox has fewer DSL subscribers on connections slower than 4Mbps, that its network is well optimised for streaming, or that its investment in fibre is paying off and subscribers are migrating to faster FTTH packages.
It could also be that most of its Netflix-viewing subscribers are on faster services.
This is Fialkov’s point: It is difficult to draw conclusions about the performance of ISPs based on average speeds alone.
Fialkov said the same issues apply to comparisons that rely on averages across multiple services on a network.
Averaging speed tests, for example, would result in completely different ranking if you exclude all services below 2Mbps, or above 10Mbps.
He said ISPs will start seeing this phenomenon as fibre becomes more widely available.
Early adopters of the technology are likely to base their purchasing decision on speed and nothing more, but as fibre becomes available to the masses, this will change, said Fialkov.
“Now that fibre is being more widely adopted, purchasing decisions are a function of reliability, speed, and price.”
This will lead to lower-speed fibre packages coming online, which will in turn decrease the average speed across the board.
Average speed could then, in certain instances, be regarded as an indication of wider fibre adoption, said Fialkov.