Investigation or Interrogation? Don't let mirrors spoil your results!

Usability testing is a great way to connect with the actual users of your website and to understand not only what issues and problems users have when trying to use your site, but also what they like and what is working well. Can your users find their way around your website, select products and successfully complete a journey to the shopping basket without getting lost or frustrated? Or is there a key piece of information your users cannot find? It is here that usability testing comes in. Usability testing closes the gap between thinking you know how your users behave and knowing why they behave they way they do. Closing this gap will result in dramatic increases in customer satisfaction, conversion rates and profitability. If you’re considering commissioning some usability testing, you will probably have thought about what aspects of your site to evaluate, which user journeys are important to test and which audience segments you want to recruit as participants for the testing. However, you may not have given too much thought to exactly how that usability testing will be conducted – one-way mirrors in a formal usability lab setting are the industry standard after all, aren’t they? At Nomensa, we disagree and have taken the time to develop a usability testing set-up that, we believe, not only provides better results, but also gives our clients a richer experience. By taking the one-way mirror out of usability testing and minimising the equipment required, we ensure that test participants behave in a less biased manner. As a result, the findings that we deliver to our clients are higher quality and more trustworthy.

The potentially biasing effects of one-way mirrors

One-way mirrors are commonly used in usability labs to allow a large number of people to observe a usability testing session without being seen by the participant. However, most people’s experience of one-way mirrors is from seeing them in police shows where they’re used to intimidate suspects – a very different scenario to usability testing! In usability testing, one of the primary objectives is to understand how users really interact with your website and to discover likes, dislikes and problem areas. At Nomensa, we believe that the standard usability lab set-up may prevent you from achieving those objectives. Evidence shows that the presence of one-way mirrors and intrusive equipment can affect the way in which people behave. User Interface Engineering (an international usability research organisation – www.uie.com) report that “in a lab, we’ve found that users sometimes worry about who’s behind that one-way mirror and what those folks are doing”. To add to this, psychological studies suggest that monitoring equipment increases the anxiety levels of the person being observed. An anxious, worried participant is unlikely to be behaving naturally and, as a consequence, the results of your usability test could be biased. There are several ways in which participants’ behaviour could be affected, including:

  • Trying harder – some participants may try harder and persevere at tasks for longer, simply because they’re worried about who’s watching them. This could lead to lower failure rates (and higher success rates) in the results than are actually the case in reality. For example, an artificially high number of participants might persevere and complete a difficult check-out process simply because of the anxiety of being observed, thus masking the problem from you;
  • Making more errors – nervousness and worry about being observed may cause some participants to make more errors than they usually would. This could result in a minor issue being given greater prominence and priority simply because a large number of participants make the same simple error under pressure;
  • Reacting more positively – some participants may respond more favourably to your website as a result of social desirability bias. (This bias causes people to try and reply in a manner that will be viewed positively by those unknown observers.) This can mask problems with your site and cause issues to be overlooked.

In summary, if your usability test takes place in a lab with a one-way mirror, you may need to take great care in interpreting the results and recommendations from that testing. Some issues could be masked, whereas others could be falsely emphasised.

Nomensa’s solution to one-way mirrors

Our solution at Nomensa is to remove the one-way mirror and use an unobtrusive camera (typically a small webcam, or even the built-in camera on a laptop) to record the participant’s actions and relay them to an observation room (see figure 1). A Nomensa facilitator sits next to the participant in order to guide him or her through the tasks and elicit feedback. An additional Nomensa usability expert sits in the observation room to record the data. This set-up minimizes the extent of the overt observation and the intrusiveness of the equipment. In our experience, participants are happy with the presence of the camera to record their actions and seldom show any signs of anxiety or worry about unknown observers. In fact, after being told at an early stage that they will be recorded many participants then appear to forget about the camera. As a result, all effects related to anxiety about unknown observers, or trying to please unknown observers, are minimised.  

User testing diagram

Figure 1: Nomensa's usability testing set-up

We believe that our set-up therefore minimises the behavioural biases outlined above and leads to more reliable data that clients can trust to guide their website redevelopment.

More advantages

Worth mentioning are a couple of other benefits for our clients that arise from our solution. The mobility of our simplified usability testing set-up means that it is flexible, reduces cost and provides better access to users. Our set-up is fully portable and can be set up and used to test people anywhere. This means that we can conduct testing in any geographical location, as well as in homes, workplaces or particular locations where representative users are likely to be. This can considerably reduce recruitment costs and time, and provide access to groups who would perhaps be difficult to recruit in the traditional way. As a client you will also see the benefits of our set-up in terms of your experience observing the test sessions. Because our clients can observe from a dedicated room, they do not have to huddle in small darkened space on the other side of a one-way mirror. Our clients can also benefit from a Nomensa facilitator who can comment on and interpret the testing as it occurs – having more than a mirror between you and the test participants means that you do not have to keep quiet or be worried about being overheard. Figures 2 and 3 show the observation room and the view clients have of the platform and testing participant.

Clients observing testing session with Nomensa facilitator

Figure 2: Clients observing testing session with Nomensa facilitator

 

Client observation team’s view of testing sessions

Figure 3: Client observation team’s view of testing sessions

Other biases affecting usability testing

At this point, it is worth mentioning the numerous other factors that can also bias and affect your usability test participants. The ecological validity of a usability test is determined by the extent to which the context of the usability study matches the context of actual use. Some of the key factors which may lead to lowered ecological validity are:

  • Levels of motivation. Depending on circumstances, participants in a usability testing session may have higher or lower levels of motivation than actual users of your website. Factors such as participants’ incentive payments may lead to higher motivation whereas the lack of real-world drivers may lead to lowered motivation. For example, compare pretending to search for a new outfit to buy with actually searching for an outfit to wear at a wedding that is coming up shortly;
  • Users’ goals. The tasks that participants are asked to complete in a usability test may not be the same as the goals that actual users have when coming to your site. For example, participants may be asked to find the cheapest insurance quote, whereas users may actually tend to use other criteria to make a selection;
  • Environmental factors. A whole host of factors can impact a person’s behaviour when using a website, ranging from the amount of time available to complete a task to whether a person is accessing the web in the office or at home.

And Nomensa’s solution to these

Many of these factors are difficult to control, but at Nomensa we strive for the highest levels of ecological validity through techniques such as:

  • Recruiting the right participants. We work hard with our clients to determine the key characteristics of their audience so that we can then recruit those people as participants for usability testing. Ensuring that participants are as close as possible to actual users helps us to ensure that motivation levels are realistic amongst other things;
  • Matching the users’ goals. We often use a technique in our usability testing in which we let the participants determine the tasks. By asking each person what they would use the website for, we can then explore these tasks and ensure that we are testing and evaluating aspects of the site that are important to the users;
  • Testing in any location. As mentioned above, our testing set-up is fully portable and can be used in any setting (such as a workplace or youth club) in order to conduct the usability testing in the actual setting in which your users would typically be going online. This helps us to minimise the effects of many of the environmental factors.

To close…

Usability testing is your best window onto your users. At Nomensa, we try our hardest to make sure that this window does not distort your view and will give you a fantastic insight into how your users interact with your site.