User Experience: When to Use Which Research Method (Part 2)

AnastasiaUser Testing


User Experience: When to Use Which Research Method (Part 2)

In our last blog, we looked at the various user experience research methods most commonly used. Some of these methods, such as lab-based usability testing, are more commonly applied than others. Some of them have also been developed very recently, such as unmoderated online UX assessments.

Of course, using every method possible isn’t possible for all products. However, that does not change the fact that using a combination of methods is the best way to go. The key is in knowing when to apply which research method.

Sadly, many developers prefer sticking to a method or two at most, which are usually chosen based on familiarity. In order to determine the best methods, we must consider applying them by referring to a 3 dimensional frame work.

Dimension I: Attitudinal vs. Behavioral

Here, we apply certain methods by contrasting what users state and what they actually do. In UX research, it has been often observed that the two vary tremendously.

An attitudinal approach is used to gain insight on the beliefs that users state. Though usability studies tend to work better with behavioral models, self-reported information methods can be effective as well. For instance, card sorting is one such method that offers information concerning a user’s mental perception of the information space. This can help developers determine which information architecture would work best for their product; be it a website or an application.

Similarly, even surveys can offer the same benefits as they can be used to measure attitudes and categorize the. Surveys can also be used to acquire self-reported information that might prove to be very helpful when it comes to tracking or identifying issues that need immediate attention.

On the other hand, the focus group method does not work for figuring out usability for a wide range of reasons. Even so, this method does offer an instant picture of what users, as a group, perceive a brand or product to be.

At the other end of the attitudinal vs. behavioral dimension, there are methods that focus primarily on behavior. They help us understand what users actually do with the product. The A/B testing method can be used as an example here. For instance, changes can be made to a website’s design and previewed to a random set of users. This will help developers observe the effects that such design changes have on a user’s behavior. Similarly, eye-tracking can be used to determine how users interact with the new design on a visual level.

The methods mentioned above lie at opposite extremes. However, at the center of the attitudinal vs. behavioral dimension, we have two very commonly used UX research methods – field studies and usability studies. These studies rely on a blend of behavioral information and self-reported information. They can also be applied on either ends of the dimension. But, it is best to stick to the behavioral end.

Dimension II: Qualitative vs. Quantitative

Distinction is of the utmost importance in this dimension and it is presents a much broader view of the whole qualitative being “open-ended” concept. In this dimension, qualitative studies provide information on behaviors through direct observation. For quantitative studies, it is through an indirect approach that behavioral data is gathered. For instance, an analytical tool or survey may be used.

In the case of qualitative studies, we can take the example of usability or field studies. In such methods, researchers gather data by directly observing how users use a product to meet needs. Researchers have the benefit of probing behavior, asking questions and even modifying the protocol to achieve their objectives. Mathematical analysis of data is not relied upon.

It is the exact opposite with quantitative studies. Mathematical analysis is a standard feature, considering the fact that large volumes are obtained via instruments such as web-server logs or survey tools.

To put it in perspective, qualitative studies are ideal for answering queries related to how and why a problem needs to be fixed. Quantitative studies, on the other hand, give us answers relating to how much and how many. Numbers provide developers with the ability to prioritize resources by directing their attention to matters that have the most significant impact.

Dimension III: The Context of Product Use

In this dimension, methods are chosen on the basis of whether participants are actually using the product in question. This is further divided into 4 categories.

The first is natural or near-natural use of the product. Here, the study is designed to be non-interfering so as to observe the user’s behavior or attitude within a highly realistic context. The advantage here is that the results have higher validity. However, there is very little room for control. Ethnographic field studies serve as a qualitative example of this approach, whereas data mining and intercept surveys serve as quantitative examples.

The second is called a scripted study of product usage. Here, researchers observe participants through specific usage parameters. How far the study is scripted is something that can vary according to goals. For instance, benchmarking studies are known to be scripted heavily and quantitative in nature. This is done to obtain usability metrics that are accurate.

The third study avoids product use completely to gain insights on broader issues such as brand perception or cultural behavior.

Finally, we have hybrid studies, where a unique idea of product usage is developed to achieve study goals. Participatory design methods serve as an example of hybrid studies as users are allowed to make changes to a product’s user experience design in order to see how their ideas meet their objectives and also to understand why such changes were made.

Similarly, concept-testing methods rely on presenting users with a prototype of the product that focuses on its core purpose. This is done to understand the need for the product in question.

Most UX research methods can be applied across all 3 dimensions and sometimes, even within the same study. This is usually done in order to achieve multiple goals.