Diary studies for multi-device project

As our personal collections of web-enabled devices grow, so too do the contexts in which we might use them. A single mobile device can be used across a seemingly infinite range of conditions and environments, but add a second, third or fourth device and the number of ways in which they might be used and used together expands exponentially. Understanding contexts of use is essential in order to foster great user experiences, but this has become more difficult in our multi-device, post-PC era. Many of the user research methods we often rely on as UX professionals just aren’t capable of revealing all the weird and wonderful ways in which people use their collections of devices.

Single-context research

User research methods often tend to collect insight from within a single context of use. For example, in a typical usability test, researchers will capture information about:

• one participant;
• undertaking one task at a time;
• in one place;
• on one device;
• at one time of day;

…you get the idea. What they end up with is a deep but narrow slice of insight that’s constrained by the context they created when designing the test session. While this will reveal a great deal about the usability of an interface, it can’t possibly show us how the product or service will perform ‘in the wild’. It might have been easy for a participant to book a holiday online using the computer in the lab, but how would they fare on a tablet? Or when standing on a busy train? Or at 3am after a few too many drinks? (Warning! That last link leads to a video with some colourful language.) To begin to understand some of the wonderfully obscure contexts of use made possible by the proliferation of web-enabled mobile devices (WMDs) we need an exploratory research method that can offer high ecological validity and is capable of transcending any single context of use.

Enter the diary study

In diary studies, participants are asked to keep a diary (no surprises there!) of their day-to-day experiences that are relevant to the purpose of the study. For example, if we wanted to better understand how people enjoy music we might ask them to make a note (no pun intended!) of each time they listened to music during the study period. Diary studies are exploratory and longitudinal – made up of behavioural ‘snapshots’ captured over extended periods of time. This puts them in a great position to reveal the less tangible factors that can affect a user’s experience. They’re also well-placed to uncover some of those rare, novel edge cases that would never come to light during an interview with a participant. With the right data capture methodology (more on this below), diary studies should allow their participants to record experiences quickly and easily, when they happen and with minimal disruption to their daily routine. This (should) result in an ecologically valid study as the data capture method interferes minimally with participants’ natural behaviour.

A capture method for multi-device research

In their 2005 paper ‘When participants do the capturing: the role of media in diary studies’, Carter and Mankoff outlined a process for capturing data from participants during diary studies. In their methodology, participants took photographs that documented their experiences. To mitigate the risk of forgetting their reasons for taking a photo, participants were asked to briefly annotate each one with a voice memo. These artefacts were submitted to researchers, who used them as discussion prompts during an end-of-study interview. We recently conducted a diary study with over 20 participants to learn how they access online content across a range of devices. Using Carter and Mankoff’s methodology as the basis of our data-capture workflow, our study yielded great results because participants were able to record detailed aspects of their lives with the click of a button. Their mobile devices often served as the capture tools (we received hundreds of photographs that were taken with smartphones and tablet devices). A picture is worth a thousand words, and the images we received often revealed insights that would be difficult if not impossible to express in words.

Conclusion

To begin to understand our digital lifestyles in a multi-device, post-PC era we need research methods capable of capturing user behaviour when it happens and with minimal disruption to that person’s routine. Diary studies where participants can use their own devices as the capture tools are almost perfect for this purpose. The next time you need to understand how a user’s experience of your site changes across devices, consider running a diary study. I guarantee you’ll be surprised by some of the things you’ll learn.