Crowdtesting
Crowdtesting is part of the croudsourcing concept. Its purpose is to use the crowd as tester for applications, softwares, etc. instsed of a test team. "The information technology world is shifting from one driven by corporations to one driven by consumers, who demand more flexibility as the carry their devices through a myriad of environments and ecosystems." [Dieter Speidel, Mayank Mittal, Mithun Sridharan, Testing exprience, Crowd Testing, march 2014]
Contents
Definition
Crowdtesting is a manner of managing a test. The crowd is used to test a software, a product, an application, etc. It has appeared as an opposing option to the well-known test team usually trained to that purpose. Crowdtesting achieves results that are generally much less focused that a test team. Always depending on the project, such an approach can be extremely useful. A crowd can cover much more development scenari and patterns of use.
People involved
- All this process is realized to satisfy the client who will in the end use the product.
- In order to solve every possible problems that the product might have, the producer hires someone who organizes a test (organizer).
- After that the crowd has to play its role by using the product and giving feedbacks. Depending on the project, the crowd may or may not be selected precisely. This is maid thanks previous questionnaries that of course contain a certain margin of error, since the crowd answers these questions at home without any supervision.
- In the end, the results have to be analyzed and the product corrected in order to be as optimal as possible (specialists).
Process
- The first step consists of the definition of the project, and its analysis, so that it is clear that crowdtesting is better adapted than a classical test team.
- After that, it is necessary to define the questions that will lead to the discovery of the possible bugs of the product.
Now that the test is ready, the crowd enters the process by using the product. People have to document their use, so that when a problem is found, it is possible to know exactly how to reproduce it and solve it. The documentation (texts, images, screanshots, etc.) is extremely important since the testers are not taking the test under the surveillance of the organizer, but at home or in the train or anywhere else.
- The next step concerns the results that have to be collected and analyzed.
- The last step is to solve all the problems found.
Each project implies a different way of managing each step, since the product can have variable purposes.
Once the product is on the market, it is almost impossible to correct the problems and still be successful which means that the testing process is extremely important !
Money
There are different option in managing the money :
- “Pay-per-bug” is the most used option and the person taking the test only gets money if founding a significant problem and proving that it exists.
- Another option is to be paid a certain amount of money for taking the test and in addition to that be paid again if finding a significant problem.
- The last option is to be paid per hour spent taking the test and in addition to that be paid again if finding a significant problem.
Motivations
For the producer, "crowd-sourced software testing helps achieve a high probability of consumer acceptance of software modules by ensuring that the software is defect-free to a very high degree." [Dieter Speidel, Mayank Mittal, Mithun Sridharan, Testing exprience, Crowd Testing, march 2014] The main motivation in participating in a crowdtest is the money. There are other influent factors, but they are far less mentioned than money : competence, altruism, fun, challenge oneself, etc.
Benefits
The benefits of crowdtesting are for instance coverage, reduction of costs, availability, speed, impartiality, diversity, real conditions.
- Coverage is a very important factor since an extremely large amount of devices is used as well as system configurations, whereas a test team could not cover such a large diversity of devices and therefore notice all the possible problems.
- Reduction of costs also plays an important role for the enterprises that organize a test and are looking for results. As descibed above, there are three different ways to manage the money but the one that is most used is "pay-per-bug" which is often applied regardless of the importance of the bug.
- Availability concerns the time as well as the location. Indeed, a test taken by the crowd runs at any time of the days because it involves people from all over the world.
- Speed is a result of the availybility but also of the fact that the amount of people is much larger than the one of a test team.
- Impartiability results of the fact that a crowd consists of people whose education, interests, knowledges, etc. are indefenitely variable.
- Diversity concerns as well the people as the devices. The crowd may help the developping process of a product by enlightening a more intuitive manner of using it.
- Real conditions refers to the fact that these products are created for the people, so using the people to test it is as close as possible to the real situation in which it is going to be after its release.
Risks
The risks or challenges in crowdtesting are for example the lack of ownership, the lack of control, the lack of measurement, the reward concept, the communication with the crowd, the security, the continuous involvement, etc.
- Security concerns the confidentiality the testers want to have in order to take the test, so that they can give any personal information that will be kept confidential. This criteria is generally well organized by the crowdtesting platforms.
Examples
Ebay, Amazon, Microsoft, Google, Facebook, etc.
- For instance, nine million people tested the beta-version of Microsoft Office 2010, with wich they gathered two millions significant comments. This allowed them to improve their product.
Sources
- Dieter Speidel, Mayank Mittal, Mithun Sridharan, Testing exprience, Crowd Testing, march 2014
