In a Bug Battle, sponsored by uTest, three top-browsers (Internet Explorer 8, Firefox 3.1 beta and Google’s Chrome) were tortured by 1,331 uTesters in order to find as many bugs as possible, with $1000 as a reward. The testers found a total of 672 bugs, divided over the three browsers, and 15% of these bugs were classified to be in need of immediate attention.
The big question of course is which browser performed best? Looking at the absolute numbers, Internet Explorer 8 was the winner with “only” 168 bugs found, followed by 207 bugs found in Firefox. The testers found the most bugs in Chrome, namely 297. At first sight this is not very strange, since Chrome was released very recently.
But the bigger amount of bugs that were found in Firefox was a little surprising me, since it is open source, and therefore one would suspect that the many eyes looking at the source code would result in less errors in the code and therefore result in less bugs (http://www.wired.com/software/coolapps/news/2004/12/66022).
However if you take the number of testers per browser into account, this is exactly what we see: There were 514 testers for Firefox, 356 testers for Internet explorer and 461 testers for Chrome. So there were far more testers that were torturing Firefox than there were testers that were checking the other two browsers. If we divide the bugs with the number of testers, the classification is as follows: Firefox: 0.40 bugs/tester; Internet explorer: 0.47 bugs/tester ; Chrome: 0.64 bugs/tester. Firefox however did have the highest percentage of bugs that were classified as needing immediate attentions, namely 24% of the found bugs, in comparison, 9 and 12% of the bugs found in Internet Explorer and Chrome were classified in that way (I don’t know on what basis they were classified).
Another interesting result from the test came from the usability of the browser according the testers. When asked about the usability of Internet explorer, 87% defined it as fair to good, but nobody defined it as excellent. The usability of Chrome was defined as fair to good by 74% of the testers and another 10% defined it excellent. Also in this “test” Firefox is the absolute winner, with 81% defining the browser usability as fair to good and 16% defined it as excellent. That is 97% of the testers that are satisfied with the browser.
If you want more details on the test you can find the uTest past following this link: http://www.utest.com/news_bug_battle_browsers_Q408.htm
uTest will hold bugbattles on a quarterly basis and if you want your software to be tested, to contact uTest might be a very interesting idea, since you will not have to hire testers yourself and will not be dependent on the feedback of your regular customers.