Broadband Testing procedure: Your Feedback Needed

[rpm]

Hi folks

As you know we are using TCPIQ, web based speed tests and some ping tests for the ‘volunteer testers’. This will already give us a good idea regarding the performance of the various services, but the basic ‘lab based’ testing is more in-depth. We have a basic testing routine, which includes the following:

TCPIQ Testing
BBST (Another speed test application testing Download, Upload, Ping)
Web Based Speed Tests (local and international)
Ping (using NetPing & other software)
Ping (Using online gaming apps on various ports)
P2P (Using various networks)
Download (Using download manager)
VOIP (web based tests)
VOIP (using Skype & Google Talk: Practical Test & human feedback)
Mobility Testing in Gauteng (Being planned: most likely involving a Minibus, a few volunteers & lots of beer )

Other data will involve:

User feedback and questionnaires
Network data from providers
Network Footprint
Support Centre/Helpdesk Assessment
Forum or Online Support

Please let us know if you think we should include other specific testing. Your feedback about this testing procedure and how we can improve on it will be greatly appreciated. Any advice will be great.

Regards,

RPM

 

[Vio]

Ping (Using online gaming apps on various ports)

I asume this includes international games like Guildwars and World of Warcraft?

 

[TheRoDent]

Whilst TCP/IQ is great for consistency monitoring, I can recommend http://www.myspeed.com/index.html as a generic "once-off" test of connection quality. This is the same test that Slimothy punted earlier at internetfrog.com when testing iBurst, and is one of the better "one shot" tests out there.

The service is also comercially licenseable, and gives a pretty good "generic" indication of connection quality.

In my view, P2P tests need to be very explicitly controlled, and perhaps an MyADSL international test torrent site would be in order. A hacked up tracker that only returns a single peer (the myADSL seed server) would be a great test. Other P2P protocols could be tested but any such tests would be very subjective, in fact I wouldn't even bother going into things like Gnutella, or Limewire that much.

VOIP quality will once again be indicated by the Quality of the connection, and RTT and latency measurements. I think a generic test such as myspeed.com is good enough to indicate service quality in this field.

Testing in itsself is a huge artform and requires WAY more discussion and thought than I can currently spend on this post. I've sent you a PM with some ideas RPM, but I do think we need to define some specific, empirical tests, in order to avoid any "whining" by the service providers.

Ideally, these tests should be developed as a client application, by the MyADSL community as a series of test "modules" by our talented .za programmers. We have enough skills here, to do it. My original Sentech "wardriving" application still exists and provided many insights into the TCP behaviour on Sentech's network.

Whilst the timeline for such a project might not meet the demands for a September report, I do believe that it is a project worth spending some time on.

 

[8321]

I see the problem being the 'volunteer' bit. There could be maniacs doing test every 15 minutes for 8 days and then lurkers doing only one test in 8 days. How does one ensure that all products get equal exposure to the tests (whether the basic tests or RPMs extended list), unless users are identified beforehand and asked to do the tests at the same time?

Then, with anonymous volunteers, there is a possibility of connivance by one or more supplier to bias their results.

 

[rpm]

I see the problem being the 'volunteer' bit. There could be maniacs doing test every 15 minutes for 8 days and then lurkers doing only one test in 8 days. How does one ensure that all products get equal exposure to the tests (whether the basic tests or RPMs extended list), unless users are identified beforehand and asked to do the tests at the same time?

Then, with anonymous volunteers, there is a possibility of connivance by one or more supplier to bias their results.

Hi 8321

I understand these concerns, but we will assess all of the stats before including them in our report. We also have a system in place that will eliminate most poor results (weighed) and eliminate one set of result that might significantly influincing the outcome.

Regards,

RPM