The modern business model of app distribution and the proliferation of smart-phones has created a potential gold-mine for app developers, but it has also created some hazards. It is no longer possible to assume that your end-user is going to be in the same country, or even the same part of the world, as the developers. Remote usability testing is a way to get a truly international cross-section of the potential user-base at relatively little cost.
In addition to the flexibility and scope of remote usability testing, it also allows the testers to use their own devices in a natural context and setting. In the case of asynchronous remote usability testing, you can even recruit people as they land on your page, so that you have a testing group that is already interested in the product, rather than people who may be professional QA testers, or otherwise not invested in your product.
The flexibility inherent to remote usability testing allows for the study to be designed in a way that is convenient for both the person testing the app and the people who want to have it tested. There are a couple of different approaches that have both pros and cons, but the nature of remote user testing is that you can arrange the test in the way that best suits your needs.
Unmoderated Remote Usability Testing
It is possible to run a usability test as a completely asynchronous affair. Neither the users nor the people running the test need to be involved at any particular time. This affords the greatest flexibility to both parties, but it means that you don’t have as much control over what features the user pays attention to. This might even be its own advantage, since that way the user won’t pick up any biases from the moderator.
There are two different ways of doing unmoderated testing, either with direct user participation or through statistical reporting. Pure statistical reporting provides the greatest flexibility, the most data, and data of the highest quality, since gathering statistical data about users without asking them to fill out a response form eliminates the possibility of non-response bias. On the other hand, you cannot ascertain a user’s subjective impression of the usability of your software, only their behavior.
For that reason, having the user participate and respond directly to specific questions about the user experience after they have used your software is another way of doing asynchronious, unmoderated usability testing. You will get fewer responses this way, since many people just won’t bother, and this also creates non-response bias, but the benefit of having specific subjective impressions may be worth the disadvantages.
Moderated Usability Testing
The other option is to coordinate between the users testing the software and moderators who guide them in their testing. This offers less flexibility than asynchronous testing, since moderators and users have to be online at the same time. This means even fewer users giving you even less data.
The advantage to moderated usability testing is that it lets the moderators guide the users and get immediate feedback from them. If there are specific aspects of an app that need particular attention, this kind of direct feedback can be useful, but it introduces the risk of the moderator biasing the user in what they notice and assume.
Because of this lessened flexibility and greater risk of bias, in most cases unmoderated usability testing is the better way to find out how people naturally interact with your app, and identify possible problem areas, or places where the user experience can be smoothed out.
Jonathan Martin is the President and Founder of CoolBlueWeb, a full-service web design, development, testing and marketing agency.