Broadband Report - What do you think?

[slimothy]

Talk is easy - rpm is actually DOING SOMETHING as opposed to your bitching and whining about how you see the whole exercise as flawed.

I'm not whining, i'm trying to contribute thoughts as to how the test may be improved in the future, what? am i supposed to go open another website and have another test? whats the point, why not just help make this test better.

also we are not bitching because it scored the lowest, look adsl384 gives out a measly 30k or so, any server in the world at any time can give you 30k on a single thread, so when you look at promised speed versus speed recieved on a single thread they are close to the same, so homeadsl384 scores nice and high. But if i had a 10mbit, i won't get 10mbit from another server off a single thread will i? i can fill my pipe with 10mbit, but not off one single request, same applies to iburst

 

[rpm]

I must be off now...will answer your next posts later

 

[JayT]

And for the linux guys.... install WINE and see if the tests run. RPM, you might want to get some official tests done so that there arent any complaints in the future

Linux: http://sourceforge.net/projects/wine/
Mac: http://sourceforge.net/projects/darwine/

 

[Vio]

I think you need a larger sample size, the larger the better. Maybe publicise the testing in the individual forums. It was a long time after arriving here before I ventuired onto other forums. I also think your reports should cover the variability of results (standard deviation). I also think your tests should cover local access and multithreading, after all that is also what users actually experience. And I think it is very important that each user is weighted equally.

Some constructive points there, but I’d like to stress that international through put is what counts as most content is hosted offshore.

 

[Daveogg]

Sorry jumping in here late.
RPM if you really only had eight testers on iburst representing a user base of over 2500 your results are completely meaningless. Anyone doing stats 101 will confirm that for you. You are now using useless and baseless information to influence potential broadband adopters. To maintain your and this forums credibility you have to can it now.

 

[Gatecrasher]

The weighing depended on a few factors, including the frequency of the tests and the number of days on which tests were conducted. A results from a person conducting only one test using one application must obviously carry a lower weight than a person that have done numerous tests using all three methods.

This reiterates my point. You already have a small sample size of 8 users. But in arriving at 120kbps what is the largest weighting given to a single user. What if, say, 50% weighting is given to just one user with a bad connection who submitted gazillions of horrible test results. Will that not skew your results such a degree that they are completely worthless?

If I get 1Mbps and 7 others get 0kbps, then, with equal weighting, the average is comes out to the result you are publishing

If I get 0kbps and 7 others get 1Mbps, but because I submitted a gazillion tests, I'm get 88% of the weighting, the average also comes out to the result you are publishing.

But the two results are not equal.

 

[rpm]

Hi Dave

Thanks for the feedback.

I agree that a sample size of 8 is rather small, but if we consider the population of 50000 of ADSL it means that we need a sample size of 400 to be within academic research limits. This means that we will be unable to produce such a report…ever! I think this report has great value and give accurate results. If we can find conclusive proof that this is not the case we will gladly can it. Here I am not talking about ‘I get great speeds’, but rather empirical results from a group of users over an extended period (including the same results from users from other services).

Regards,

RPM

 

[JayT]

Gatecrasher, there is nothing stopping the others from doing multiple tests. And dont forget, soon the iBurst cap will be in place so I doubt there are going to be people wanting to just waste all their bandwidth for a test. Because then the results will be accurate as they sit on a 64kbit connection for the rest of the month. Accept the results and make sure everyone you know participates in the next test.

Also note that results such as tcpIQ are not totally accurate. If I download an update or busy downloading mail, the results drop below 100kbps. As my tests are scheduled to run every 30min on a different machine, I get a lot of inaccurate data. But its accurate in the sense that my speed is xyz during average use. Obviously when Im a sleep and not using my connection the results 'should' increase.

 

[rpm]

Hi folks

This thread is going nowhere. I think the best way forward is if each member gives his/her view on:

1. How you think we can improve the testing procedure
2. How we can ensure a larger sample size.

Regards,

RPM

 

[bb_matt]

1. More people, allow for drop-down boxes to rate price, speed, reliability, support (see point 2)
2. Easier entirely web based tests (no need to download anything or install a plugin)

Where I'm going with this is an easy to use single web page - I'm no programmer, but I do know that the current testing method will put a lot of people off.

The drop downs allow testers to use thier subjective thoughts on the service as a whole, the speed part will obviously be weighted against the actual speed results.

Name :
Service <select from dropdown>
Price : 1 to 10
Speed : 1 to 10
Reliability : 1 to 10
Support : 1 to 10

[Click to test your connection]

 

[slimothy]

i think thats what i suggested way back on the first few pages, yes bb_matt, thats what we need exactly

 

[Gatecrasher]

1. More people, allow for drop-down boxes to rate price, speed, reliability, support (see point 2)
2. Easier entirely web based tests (no need to download anything or install a plugin)

bb_matt, I agree completely. But how do we ensure integrity? That people actually have the service being rated...

 

[slimothy]

as i said before you just make sure they are on the right IP block, if a user says they're on iBurst and they have the IP of 196.46.*.* you can be sure they are on iBurst.

 

[Gatecrasher]

as i said before you just make sure they are on the right IP block, if a user says they're on iBurst and they have the IP of 196.46.*.* you can be sure they are on iBurst.

Good for Iburst, but maybe not so straightforward for other services. Presumably, the same methodology must apply to all.

 

[Gimli]

Hi folks

This thread is going nowhere. I think the best way forward is if each member gives his/her view on:

1. How you think we can improve the testing procedure
2. How we can ensure a larger sample size.

Regards,

RPM

Correct, you obviously have something to defend (your paper). There are plenty positive suggestions on this thread that you can work into your methods to ensure a better test if you do not just pass it over as 'not scientific' or 'subjective comment'.

Lets take another tack on this, please publish your complete testing method, formulas and algorithms that are so 'complex' as per your report. Also all testing data used. Then we get the chance to scrutinize it and let us then see if it can stand up on its own two feet. Like proper scientists do.

 

[qDot]

Here's my contribution

How about doing a protocol by protocol or application by application test such as HTTP | FTP | P2P | etc.

We measure that against each 'broadband' provider.

That way when asked "which is better?"

I can reply "What do you want to use it for ? "

and be able to say well iBurst is the best for browsing ... etc.

I think the tower info and even tower sector can be useful that way you should be able to say well if you are in Sunninhill then browsing on mywireless is better from Sunninghill than browsing from Paulshof (same tower different sectors correct ?)


So a laborious test excerice such as the one i suggest can make it all the more difficult for a technical person to answer the question from a non technical person who wants to know

"WHICH PRODUCT GIVES ME FAST INTERNET?"

 

[jmn]

Agreed Gimli

I actually get better results from the TCP/IQ test when my connection is under heavy load than when it sits idle Contradictory at least...

 

[Daveogg]

This whole process is just flawed. If you have an axe to grind with Iburst, just take a drive to an area on the very limit of reception. Do all the speed tests post a few pings and complain about the "horrendous" packet loss and inconsistent pings. To accurately asses Iburst you would have to have some way of verifying signal strenght frame rate etc of the tester, otherwise you may just be assesing geography.

Put it another way. When i am at work, Iburst is amazing, last weekend i was watching live Belgium Classic cycling (OT for any cycling fans try www.cycling.tv) streaming in at 350kbps and NO buffering stops.
At home i am lucky to get 10kbps, but i am on the edge of a very pale green zone. I am a realist and never expected it to work here. Why do others sign up in dodgy areas then moan about the speed and pings??

IC dont take this personally i understand that you were rendered a poor signal due to antannae alterations AFTER you signed up. Thats a balls up by iburst and you should be compensated.

 

[slimothy]

Good for Iburst, but maybe not so straightforward for other services. Presumably, the same methodology must apply to all.

they all have specific IP blocks

 

[Vio]

I believe bandwidth testing is the way to go. "Asking" someone to fill in a form is not a good idea cause Sammy that just checks his mail thinks the service is astounding.

I do think that there should be categories of tests to perform as business / downloaders / gamers all have different views of how a service should perform.

Regarding bandwidth / latency / stability tests, why re-invent the wheel the existing benchmark applications are good enough. Other services use them why shouldn't we?

The only problem is the amount of people performing the benchmark is not necessarily a fair reflection. To resolve this problem why not let MyADSL members specify the broadband connection(s) they use as part of their registration information and send out request emails to participate in the tests when they happen. This will increase the size of the test sample dramatically.

That’s just my 2 Zim cents