The Donut Strategy: Guerilla Ops at Soho Works

There are many ways to approach usability testing, but most share the same basic challenge - finding suitable test subjects. Your options are generally (a) dig into the database of a specialist testing platform (like Maze or Userlytics) or (b) go out and find your own, either online or IRL.

Guerrilla testing is an example of the latter that works particularly well when your test audience is centred around a particular kind of physical location. As the name would suggest, it involves camping out somewhere with a lot of footfall, and attempting to turn passers by into testers.

We were recently asked to undertake some early-stage prototype testing for a client (currently in stealth-mode). As it happened, our co-working space in Soho Works was the perfect melting pot for the kind of users we were interested in speaking to - specifically around lunchtime.

The trouble is that people tend to be pretty busy in Soho Works, and lunchtime isn’t the greatest time to catch people. (Competing against hunger is rarely a winning strategy…). 

Our solution? Crosstown:‍‍

We brought along a selection of mouth-watering Crosstown donuts, positioned ourselves near the entrance, and sat in wait…. The deal was that anyone willing to perform a simple task-based test using the prototype would be rewarded with a delicious donut of their choice. The task was only semi-guided, so we were able to observe their reactions without much interruption, and gather detailed exit feedback. 

Novelty aside, the donuts served as an excellent ice breaker, creating a relaxed environment, and a platform for open, honest reflection. We set out to test 15 subjects at a minimum, but such was our popularity we ended up speaking to closer to 30. 

Sadly, the test programme was curtailed when we ran out of donuts, however by that point we had gathered more than enough high-quality insight to work with. Aside from identifying several key issues and areas for improvement, we were also able to speak to a wide cross-section of potential users, validating some of our assumptions about likely early adopters.

Getting started with Guerrilla Testing

If you think guerrilla testing is a viable option for you, here are some tips to get you started:

1. Think through your task(s), location and incentive

• Task: Don’t tell the user what you’re testing; just give them a task to perform. The task should be related to the core functionality you’re testing. Prepare a script that outlines the key information, but is flexible enough to allow spontaneous and natural conversations. It’s often a good idea to run a few pilot tests to fine-tune your approach.
• Location: Choose a location where your target users are likely to be found. For us this was a co-working space, but cafes, shopping centres and transport hubs are all viable options. 
• Incentive: Ensure the incentive is appealing to your target demographic. Donuts can work well, but probably not if you’re working on a health and fitness app. Amazon gift cards are usually a winner, but try to zero in your target user’s specific needs and desires, as you’re likely to attract a better sample that way.

2. Avoid "Leading the Witness"

It’s hard to go into these sessions without any preconceived notions of what you’re likely to see, but be aware that those preconceptions can skew your results (see observer bias). If you can’t do a proper double blind test, try to bear the following in mind:

• Neutral Language: Use neutral and unbiased language to avoid leading the participant to a specific answer.
• Open-Ended Questions: Encourage participants to think freely and give honest feedback by asking open-ended questions.
• Non-Verbal Cues: Be conscious of your non-verbal cues, maintain a neutral expression to avoid influencing the participants' responses.
• Prompting: Keep your prompts light touch, and save them for when they really get stuck. Watching a subject try and figure something out can be instructive, but you don’t want everyone to get stuck on screen 1, or you’ll have no idea what issues emerge later in the journey.

3. Master the Art of Observation

• Prepare: Brief the participants about the process and what is expected from them clearly before starting the session. As above, keep this objective; simply outline the context and task in neural terms.
• Take Notes: Vigilantly take notes of the participants' actions, expressions, and comments.
• Video/Audio Recording: If possible, and with the consent of the participant, record the session for a more detailed analysis later.
• Empathy: It sounds a bit woolly, but it’s vital you approach your users with empathy, understanding, and patience, fostering a comfortable environment for them to share honest feedback.
• Watch for Unexpected Behaviour: Pay attention to unexpected behaviours and spontaneous reactions. Things that surprise you are often the starting point for valuable insights.

4. Structure Your Data

• Organisation: After the session, organise your data meticulously, segregating them into different categories like positive vs negative feedback, suggestions, etc.
• Patterns: Analyse the data to identify patterns and recurring themes in the feedback. (Advanced tip: ChatGPT can be a useful tool for identifying and summarising these kind of patterns).
• Stay Objective: Don’t rush to conclusions about what you’re seeing, and be careful not to narrow the field of possible solutions too early. There are probably several ways to interpret the data, and a plethora of possible remedies to any emerging problem. This exercise is about gathering data - solid recommendations require further reflection and exploration.
• ​​Reporting: Prepare a detailed report highlighting the key findings, along with actionable insights. Create a feedback loop where the insights and suggestions from the guerrilla user testing are integrated directly into product planning.

5. Understand the Usefulness (and the Limitations) of Qualitative Data

• Perils of Qual: Guerrilla testing can provide rich insights into user behaviour and preferences, however these should be treated more like clues than conclusive proof. This isn’t the kind of exercise that typically generates statistically significant results, however it does allow you to go into greater depth than any analytics platforms allows, and really get under the skin of your users. A mixture of qual and quant is generally desirable for planning.
• Artificial Environment: Guerrilla testing is limited by the quick and informal nature of the testing. Be cognisant that task-based testing may not not replicate real-world scenarios precisely, and participants might not always behave naturally.