We have a wide array of UX research methods that we use at Centralis, and each one has its own advantages and limitations. Which method will work best in your situation will depend on your learning objectives. Broadly speaking, there are three types of usability tests: un-moderated quantitative studies, un-moderated qualitative studies, and moderated qualitative studies.
Un-Moderated Quantitative Tests
Quantitative tests, which are usually un-moderated, use large participant samples to allow projectible measurements of how effectively customers can use your site or application. Participants access your site or app through a platform that records key performance metrics, where they complete a series of simple closed-ended tasks. These tasks are often followed by a brief survey exploring users’ subjective impressions of the experience.
This type of testing is helpful in measuring error rates for common tasks and seeing how quickly and consistently users can complete them. We can also see if users have a significant preference for one design over another. A key limitation of this method is that it does not help you diagnose usability issues or understand designs from the user’s perspective.
Strengths
Speed: These tests can be quick to conduct for general target audiences, since hundreds of participants can take them at the same time.
Low cost: Since users are only expected to devote about 15 minutes to each session, it is relatively cheap per participant. This means that it can be an inexpensive test to run depending on the platform you choose.
Projectable statistics: Quantitative tests allow you to collect data from a large sample of people, and to receive statistically reliable results. These results can be used to estimate how your design might perform in the real world.
Limitations
Lack of diagnostic value: Metrics such as completion rate, time on task and abandon points can help you detect usability issues, but they don’t shed light on the nature of those issues. Qualitative research is the key to understanding and resolving usability flaws.
Low user engagement: Participants in these studies generally take many tests for very small cash payments, often less than $5 per study. As a result, they have little incentive to put in significant effort. Moreover, taking the tests at home or in coffee shops means they face many distractions of everyday life.
Small Scope: Low user engagement means that these tests need to be limited to 15 minutes to encourage complete responses. Such short session lengths make it impractical to explore complex or comprehensive usability issues in this format.
Un-Moderated Qualitative Tests
Another option is un-moderated qualitative testing. This is when users record themselves performing pre-planned tasks with a product and talking about their experience out loud. This option is relatively quick and light, and users can complete the test on their own time. Once enough users complete the tasks, you can analyze the videos and see how easy it was for participants to use a feature of your product. Since users are on their own to complete the tasks, it’s best to limit the study length to 15 minutes or less.
Strengths
Good for small questions: These tests can quickly answer targeted questions that you have about your product – if you want to know how easily people can find or use a specific feature on your site, this is an effective method.
Quick: With a general population, these tests can be fast because multiple participants can test your product at the same time. Once a study is launched, there are usually enough participant completes within a few days to begin analysis.
Low cost: Since users can complete these tests at their home in a span of 15 minutes, un-moderated qualitative is a less expensive option for small tests. Note: watching and analyzing all these user videos can still cost you many hours!
Limitations
Low user engagement: Low incentives and an uncontrolled setting mean that these tests face the same challenges with low user engagement as unmoderated quantitative studies.
No big questions: As with unmoderated quantitative tests, tasks in these studies must be short so that users don’t leave the test unfinished.
Inflexible: Without a moderator to ensure that users understand what they are expected to do, participants can easily misinterpret a task or veer off course. There is also no ability to follow up on why a user did something within the session. This means that there is lost meaning, which leads to less comprehensive findings.
Moderated Qualitative Tests
Moderated tests are when a researcher facilitates live one-on-one testing of your product. These involve giving the participant a series of tasks, having them think aloud, watching what happens, and probing selectively to clarify their experience. Moderated tests focus on finding the ‘why’ behind site performance and behaviors, which leads directly to potential design enhancements.
We choose this method to find out how well users can understand and use a design, gauge how well it meets their needs, and identify flaws and areas of strength in the user experience.
Strengths
High Engagement: It is easier for users to focus on testing the product with a facilitator, because moderators can keep participants on-task and ask follow-up questions.
Good for big questions: Moderated tests allow for more complex or open-ended tasks than other methods, both because sessions are longer (typically 60 minutes) and because there is a facilitator to keep users on-track.
Comprehensive: Moderated sessions produce more complete data than other methods. Facilitators can note where users become stuck, but then help them move on to test the rest of the experience. Facilitators are also able to follow-up on unexpected issues in real time, which can lead to findings that would be missed in pre-scripted, automated studies.
Selective Recruiting: These tests usually involve fewer users, which allows you to focus more precisely on who should be recruited as participants.
Limitations
Speed: Moderated testing is more time-consuming than un-moderated testing because participants must be sourced and pre-scheduled, and the sessions are conducted one at a time.
Cost: The amount of time and attention paid to these tests tends to increase the budget they require.
Statistics: By definition, qualitative testing does not attempt to measure the the prevalence of a given usability issue. However, after observing several testers running into the same problem over the course of a two-day study, it becomes clear which issues will most severely affect future users.
If you’d like to know more about usability testing options, email us or call us anytime at 847-864-7713.