Mobile network quality: ICASA versus Operators
The Independent Communications Authority of South Africa (ICASA) recently published their ECS and ECNS compliance reports for South Africa’s largest telecoms operators.
As part of the compliance reports ICASA assessed the network availability statistics provided to them by the cellular operators.
Vodacom submitted that their network availability was 99.6% over the measured period, while MTN said that they have always achieved a network service availability of over 99% in all areas of coverage and Cell C showed an average network availability of 99.33%.
These figures mean that the three cellular operators comfortably comply with the regulations which require operators to “…achieve an average 95% network service availability over a period of six (6) months”.
The statistics provided by the networks were however significantly higher than figures released based on ICASA’s own tests.
ICASA conducted a quality of service survey of the cellular operators in Gauteng, and published a document in July 2011 which showed that Cell C, Vodacom and MTN failed to meet the minimum requirements for Call Set Up Success Rate (CSSR) and Dropped Call Rate (DCR).
Overall Performance
ICASA’s testing methodology questioned
Many industry players raised questions about ICASA’s testing methodology and the statistical significance of ICASA’s testing sample.
MTN said that they have identified and informed ICASA of a number of fundamental flaws pertaining to the methodology utilized by the regulator in the Quality of Service report.
“MTN believes that this test was done in isolation, sampled only a statistically irrelevant number of calls and does not reflect the overall quality of MTN’s network. In fact, the report does not even follow the criteria set down in ICASA’s own regulations,” said Robert Madzonga, MTN SA Chief Corporate Services Officer.
“By ICASA’s own admission their report is but a snapshot of the quality of the network over a very short period of time. No real conclusions can therefore be drawn from ICASA’s essentially defective report as the methodology is not only scientifically unsound, but the imprecise results thereof is not an accurate reflection of the true quality of the MTN network.”
MTN says that they continually assess the quality of the whole network and believes that it meets not only world class standards but also the standards set by ICASA in their regulations.
“In fact the compliance report as published by ICASA clearly indicates in the supplementary report (that is attached to the main report) that MTN is in compliance with the required regulations. It is important to note that MTN had provided evidence of compliance to ICASA even prior to the date of the first ICASA compliance report,” said Madzonga.
Mothibi Ramusi
Mothibi Ramusi, Cell C’s Executive Head of Regulatory Affairs, points out that ICASA only conducted the tests for two months and also in limited areas – specifically in Gauteng and KZN.
“We have requested a meeting with ICASA to discuss methodology and procedures. It is only fair to divulge the methodology once such a meeting has taken place,” said Ramusi.
Ramusi said that there are no issues with regards to tracking of performance, however, it must be an activity that is based on agreed principles and methods in order to avoid any misinterpretation and related clashes.
“It is always advisable to agree upfront on key performance indicators and targets before embarking on such an activity. Activities must then be conducted against set targets,” said Ramusi.
Andries Delport
Vodacom CTO Andries Delport says that the number of test samples is very low and is unlikely to be statistically representative of network quality.
“The number of areas trialed is too few to be representative of the Gauteng province overall, and the network quality measured in these areas alone is unlikely to be representative of the entire province,” said Delport.
“To have more representative network quality results for an area, one needs to trial the area repeatedly, and cannot rely on the trialing of one day only.”
Delport points out that Vodacom measures the call setup success rate and call drop rate based on every single call in the network. “Based on these statistics, the call drop rate in the network is far better than stated in ICASA’s report,” said Delport.
Delport added that it is only the operators that can provide the statistics for every single call on the network. “If one also wants to include drive testing results, it is recommended that ICASA uses an independent third party that has a proven track record in doing such measurements,” said Delport.
ICASA responds
ICASA explains that they have done “snapshot” testing in a limited area which explains the difference in results.
“ICASA is testing results are based on a short period of time hence the Operators results are done over six month period. ICASA’s testing is aimed at replicating the subscriber’s experience when making calls in a certain area at a given point in time,” said ICASA spokesperson Paseka Maleka.
“It is a small sample that is not representative of the entire network, rather the network only at a given point and time. The network operators are reporting on a much larger sample of their entire network over an extended period of time.”
ICASA Gauteng Cellular Network Testing
Maleka added that ICASA has no reason to believe that the results provided by the operators are not correct, and that they would expect ICASA’s aggregated results to be similar to the operators’ results as their sample size increases.
Maleka however defended their testing, saying that ICASA’s testing procedure is valid, as it is a snapshot of the mobile service provider’s network performance, based on the specified routes during the time of day when the measurements were carried out and using a particular type of handset.
“The Authority is confident of the methodology and the equipment used-as it is the same or similar equipment as used by the network operators. The testing is statistically valid as the numbers of samples are acceptable for a snapshot result,” said Maleka.
“The Authority expects that the confidence level will increase with the number of samples taken, giving us a more accurate reflection of overall network quality. These snapshot results cannot be viewed as indicative of the overall network quality.”
“We are continuously doing testing and would be publishing results quarterly of our testing, increasing our sample size and hence a reflection of the overall network quality.”
Maleka added that the data from their tests and the data provided by the operators themselves complement each other. “The operators should still declare their performance and ICASA would still conduct QoS testing for verification,” said Maleka.
Loading ...