- Jan 24, 2007
And when you done, summarize it in English please
Sorry guys. OK, let me do a somewhat TL;DR:
NPS is bullschit, and cannot answer the question supposedly being answered here. There are major flaws why, from a logical, mathematical, statistical, economic, and just plain common sense perspective, it cannot achieve what it is being tasked with to achieve here, if you know NPS. Furthermore, the sample sizes of each ISP are not sufficiently large enough to make a definitive statement as is being made in the article. It's simply not statistically possible unless you publish your margins of error as well. In order to publish margins of error, you MUST display these on the chart. Because you have no way to know whether these statistical errors exist on the up or downside of the mean, the only statement that you can make is that an ISP scored somewhere between the 2 numbers - it is as likely that they scored the lower number and a competitor scored their higher number. Because that is what you have calculated - you have not calculated anything else.
Think about it like this (simple way to see this just on the stats side, forgetting the lengthy arguments about the logic): if your result variance difference on average is 1% in the top 6, your margin of error CANNOT be more than 1% in your source data, especially when compounded by variable confidence levels. You can think of confidence interval as a kind of statistical margin of error, while confidence levels is (put simply for everyone reading) in this context the number of times that the confidence interval will likely be true. So for 95% confidence level with 99% accuracy (I can say with near certainty that 95 times out 100, the correct value we are trying to calculate will be within a 1% variance of a specific number). These are standard statistical models and algorithms. So if I know my number of customers, and the number of users who responded to a survey, and I want to work out how accurate the results of that survey are as actual actionable data, it's incredibly easy to do, and it works in all directions. If I know someone else's customer-base size to a relatively accurate degree, and I need to be as close to 100% accurate with my calculations, I can simply work out how many people I need to ask that specific set of questions to, in order to obtain an incredibly high level of accuracy across the entire customer base. So if I ask 1 customer of competitor X who has 1000 customers, my data will not worth a thing. If however I ask 943 of their customers the same thing, I can say that with 95% confidence level, and 99% confidence interval, the median data will translate to everyone.
Often there is no need for such accuracy, but in this case the results variance contains as low as 0.4% variance between provider results and the top 6 is separated by about 1% variance on average, in results (which are not single number answers/results AT ALL because statistically you do not have a high enough level of probability of having calculated a narrow confidence-interval-number). So for example in this case with the stupidity of adopting an NSP methodology for underlying data (which doesn't even equal "the best"), I can calculate that MyBB would have required 68214 responses to the survey from a wide and diverse enough type of internet user that replicates an ISP's userbase properly, in order to have come to these results without the margin of error resulting in a fluctuation that could result in any ISP being in the incorrect position. Seeing as we know that MyBB only received 2732 responses, we can work out what that margin of error is per ISP, and identify what the lower and upper margin of error score is for each published result.
The only accurate statement that you can make is that ISP X scored somewhere between the following two numbers the number of responses means that some ISPs have quite a heavier margin of error than others. You cannot make a statement as to what that number may or may not be, as all numbers between upper and lower limits are equally likely, as this is what you have calculated. Calculating a mean from NPS is great, but it doesn't mean that it is accurate. You have to determine the margins of error of that number, and in this case doing so is easy - it unfortunately results in a variance so high that no single ISP is necessarily in the position that they belong, and it is impossible to actually rate any ISP using these numbers.
So let's use Telkom Internet for example:
I can tell you that given the number of responses for TI, and knowing their (nearly exact) market share number, they could have scored as low as 5.38, but they could also have scored as high as 6.359, 95 times out of 100 given the current data. To say they definitively came last is just not accurate. Likewise, to say that Axxess came first is also not accurate, as they could have scored as low as 7.824 or as high as 8.901 - there is simply insufficient data to statistically tighten the range of the two variables and increase the confidence level closer to 99/100 times, which is needed in this case considering how adoption of this stupid single-data-point nonsense NPS has resulted in such a tight spread of results between most ISPs. Once again, the only statement that can be made is that all numbers between the upper and lower limit for every ISP are equally likely to be their actual score.
By publishing median data only, MyBB do not take into account their margin of error. We know they must have market data information as we were excluded from being published in Q1 2015 on this very basis. So they have all of the tools to have accounted for this but they didn't do so, but somehow saw fit to publish standard deviation metrics, which in this case due to the nature of NPS, is all but useless other than to correlate with responses to market share ratio to a small extent...