Image credit: CREATISTA

Remote UX Observation

Getting Out of the Room to Enhance Team Participation In UX

4 ½ minute read

One of the major tenets we adhere to as a team is transparency: All members of the team should have access to information about the project we’re working on. Designers, Developers, QA Analysts, Stakeholders: They all should have a complete picture of what’s going on and access to any details of the project that might help them gain product perspective and user insights.

This is especially true of any user research or usability testing that we do. Everyone on the team should have access to the results of a study, and the most effective way for them to gain these insights and perspectives is to see the users in action. Reading reports and summaries is nice, but experience shows that nothing works better than having someone see for themselves what the user is doing.

TiVo for the UX age

Traditionally, we’ve handled this by running studies on a computer in our dedicated usability lab (nothing fancy, just an empty office with a desk, a PC, a web cam, and a couple of chairs) and recorded the sessions using Camtasia. We’d then post the resulting video files to a local network drive where everyone could view the session after the fact.

While this worked in theory, the grim reality was that teamers — with their busy schedules, objectives and deadlines — would ultimately put off watching the sessions. Videos were always there to watch, so the sense of urgency wasn’t present. Before a person knew it, they’d have five or six 30-plus minute session stacked up waiting for them. It just wasn’t workable; they’d punt and just wait for the summaries to come out, thus missing the immediacy of watching users.

Greatest hits

The idea of creating “greatest hits” style summary videos — a collage of the good stuff compiled from all the sessions — seemed like the solution at first. But this greatly increased overhead for the research team. We don’t have enough staff to dedicate someone to video editing, and it was really bringing the velocity of our research efforts down.

‘Silent’ observers (in a crowded room)

So we tried pulling a few extra chairs and some of the team into sessions to silently observe. This was a fail from the get-go, for two reasons:

First, study subjects were markedly more uncomfortable in this setting, with a bunch of people sitting around watching them while they tried to use products or participate in an interview. It was like the Spanish Inquisition, and as any good student of history (and comedy) knows: No one expects the Spanish Inquisition!

Secondly, even the most well-meaning Dev, Designer or Stakeholder found it nearly impossible not to interject with questions of their own, or to react in ways that might give subtle cues to the participants. We’d kindly tell these folks, “Your job is to keep your eyes and ears open and your mouths shut” during the session, but it’s human nature. Our team is a collection of curious individuals who want to do good work. And as such, they were brimming with questions and ideas and emotional responses (“C’mon man… that button is RIGHT there next to your mouse. CLEEK EET!”). This is all good stuff. But the session isn’t the place or time for it.

Going high(er) tech

One of the benefits of working for an AV manufacturer is that I’ve got access to some pretty sweet gear, and our team was able to leverage this to great results. We were able to get some hardware that allows us to stream sessions live to multiple locations. With a fancier camera, a Blue Yeti mic, and a streaming media processor, we can now have the candidate and the moderator in the lab, going through the session activities, while the team watches from a conference room down the hall. For posterity purposes, the box also records the session and drops it on the desired network share (no more laborious converting of Camtasia files to MP4), which allows those who missed the session to catch up, if needed. Neat and tidy.

Benefits

The benefits of this new system are numerous:

• By scheduling a 45 minute session for the team (like a meeting), they’re more likely (or more required) to attend and less likely to skip the session. Their presence — and lack thereof — is noted by the team.
• Study participants don’t feel like they’ve got an audience breathing down their neck while they work with the moderator on tasks or answer interview questions.
• Team members in the remote viewing location can talk to each other and discuss what they’re seeing in real time.
• Participants also don’t have others present to provide cues while they’re doing whatever we need them to do.
• Teammates in remote locations can tune in live from their desks and watch the proceedings as they happen.

After the session is over, we invite the participant to pop into the conference room to meet the team, which also gives everyone a chance to ask followup questions.

Drawbacks

So far it seems to work. There are a few drawbacks, however, that are worth pointing out.

• Pulling everyone into a bunch of meetings can be fatiguing and put a dent in productivity. We try to give everyone the option of attending at least a couple of sessions for each study, if they can’t commit to the full schedule.
• Having a big stack of fancy equipment on the desk can be daunting to participants. It’s almost as if they walk in and ask, “are you going to hook me up to that thing?" We’ll likely move the gear to a more discrete location in the near future to alleviate this.

So far, so good

All of this has worked out very well so far. I realize that many teams aren’t going to have access to, or the budget for, this kind of fancy rig. On the flip-side, large organizations will view this as an interim step to a full-scale, “legitimate” usability lab. But I wanted to put it out there as one way that technology can aid in user research. For my team, it has — so far — dramatically improved team participation in user research and transparency for the entire team.

Until next time: Cheers.

If you’re interested in the gear list for this project (and associated cost), drop me a line and I’ll give you the details.