Research

In-person observation testing

Put a potential user in front of a product or prototype and get them to use it for realistic tasks. Watch what they do and listen as they 'think aloud'. In many ways, this is the deluxe version of remote testing.


Good for:

  • open-ended testing, getting people's general impression or reaction to a product
  • testing prototypes with very limited interactivity: you can redirect and/or explain if they try to interact with a non-functional part of the prototype, which isn't possible with remote testing
  • testing with exactly your target audience: you can recruit with whatever requirements you like, unlike in remote testing.
  • Combining with card sorts, interviews and other activities.

Bad for:

  • speed: you usually need to book your users a week in advance, or more if you're trying to get a demographic/behavioural niche
  • time: sessions can run from 30mins to 1 hour or more.
  • negative feedback: people tend to give you more positive feedback because they don't want to upset you or make things awkward.

How to set up

Download these bullets as a check-list

  • One week before: decide your requirements for participants. Requirements should get you the people you want, but not be unnecessarily prescriptive. For instance: does it really matter what age people are if you're profiling them by their behaviour? Think about what to rule out as well as rule in – if you want online shoppers, would someone who only shops online once a year count? What about someone who only buys digital goods like music and film? Good recruiters will be able to advise on this.

  • One week before: get a recruiter working on it. Names and contact details are in HighRise. Expect to pay £50-£100 fees per participant or more for stricter requirements. You'll also need to provide cash incentives for participants on the day, of between £20 and £60 - see next bullet. 5 people is usually enough to get the majority of the problems/insight. If you're targeting more than one type of person, try for 5 of each type.
  • 3-4 days before: fill in an incentive request form and give it to Rebecca. 
  • 3-4 days before: Book a room. The corner room, project room and board room are all pretty noisy due to the traffic below. If a really top-quality recording is important, consider booking space in a serviced office.
  • 3-4 days before: Arrange for recording & testing equipment, including 'reserving' any shared equipment you'll need. Consider having two recording methods in case one fails. See the section below called 'recording'. Even if you don't intend to make a highlight video, having a recording is useful if you need to go back and check something and essential if you want to record time taken for a task. You also tend to get deeper insight from rewatching the videos.
  • 1-2 days before: Plan the testing sessions and produce a 'script' for testers to follow. Consider including rough time limits for each section, and verbatim questions to ask in areas where you want to be sure you're getting the same question answered by every participant. Where-ever possible, make your results measurable, so consider asking them how easy something was to use on a scale of 1 to 5, or measuring the time it took the user to complete that task.
  • 1-2 days before: Decide how you'll record results and prepare for this. Divide a whiteboard into columns for each participant and rows for each task/area; stick your insights/results up on post-its in the relevant column/row. Or do the same in a shared Google Spreadsheet.
  • 1-2 days before: Prepare any other stimulus you'll need for the sessions, eg newspaper articles to prompt discussion, or a survey for participants to fill in at the beginning. Surveys can be a useful way to measure changes in opinion over the course of a session, with the same questions asked at the start and end of each session. Here's an example.
  • 1-2 days before: If possible, do a dummy run of your test, to identify any snags and make moderators more familiar with the process.
  • 1-2 days before: Print out an incentive receipt form for each participant.
  • 1-2 day before: Decide who will be present from the team. Ideally, you'd have one person leading the test and one person taking notes. If you need to share with more people, consider using a Webex to broadcast the screen and webcam to people sitting in another room.
  • 1 day before: Make sure the equipment is charged and has enough free memory to capture the recordings. Have back-up batteries/chargers and memory cards, as appropriate.

How to run

Download these bullets as a check-list

  • Prepare the room: clear anything that's on the whiteboards or walls, arrange the chairs and tables, decide who's sitting where, and set out water and glasses. Make sure the room is a comfortable temperature.
  • Set up your recording equipment and make a test recording to check positioning of cameras and quality of sound recording.
  • Prepare the stimulus you'll need, close at hand, and concealed from the participant until it's needed.
  • Fill in the top part of the incentive confirmation form with participant's name, amount given, etc
  • When the participants arrive, greet them, offer them a drink and/or visit to the loo. Give them a survey if you're using these.
  • Double-check they're happy to be recorded. The recruiter will have checked this, but it's good to double-check. If they ask, explain who the film will be shared with and why.
  • Give the participant the incentive: paying people at the start of the session can lead to more honest results, as they don't think the money is contingent on them giving the 'right' feedback.
  • Get the participant to sign an incentive confirmation form
  • Start recording
  • Introduce the session to the participant
    • Length of session, and when there will be a break.
    • Tell the participant that you're testing the prototype/site/app and not them. Reassure them that they can't do anything wrong in this test, and when things are confusing or tricky, it's the fault of the prototype/site/app and it's really helpful to know where it can be improved.
    • Tell the participant that you're only interested in their honest opinions: so, no need to try to say what you think most people would do, and giving negative feedback is really, really useful.
    • Consider whether you want to reveal the real goal of your research: it can be better to keep this vague so you don't prime your participant.
  • Ask the participant to do the first task on the prototype/product, and encourage them to speak their thoughts aloud.
    • As much as possible, keep silent during these parts of the session; you can ask questions afterwards.
    • If the participant asks if they're doing it right, or how to do something, reflect the question back: "what do you think?", "what would you try if I wasn't in the room?"
  • Ask questions about the task after the participant has finished. Use open questions as often as possible.
  • At the end of the session, thank the participant and see them out.
  • Stop the recording equipment and save/export recordings.
  • The moderator and note-taker should debrief, and note down each of their top 3 or 5 insights from that session. This can be really helpful when you review all sessions afterwards.
  • Reset the recording equipment, including recharging/swapping batteries and memory cards.
  • Label all of the stimulus material that the participant wrote on with the participant's name, and put somewhere safe.
  • Sign the incentive receipt form and put somewhere safe.
  • Tidy the room, including removing used water glasses and resetting the position of furniture.

Timeframe

How long does this take to do – both in elapsed time and active time.

Outputs

Describe what you get at the end – both the format and the content. EG: dozens of ideas for new products, on post-its grouped into categories.

What you need

  • £100-£200 per participant
  • a research recruiter – see HighRise for details of people we've used before
  • a quiet room
  • recording equipment (see 'recording equipment' below)

References

Advisors

Scott, Kat, Katy


Notes

  • A place to put other observations/experiences of what works well and works badly when you use this tool
  • This will probably start off quite free-form and we can learn what other sections the page template will need based on this content.