6 tips to sidestep “we already knew it” feedback on a research report
Every UX researcher wonders how they can create amazing insights that go beyond expectations and elicit a ‘wow!’ reaction from their audience. Despite conducting numerous rounds of interviews, sometimes our research reports can seem too plain and bland if they don’t reveal anything surprising and unexpected. It can be discouraging to receive feedback like ‘I already knew all of that.’
Here are 6 tips to improve your UX research and avoid falling into this trap.
1. First thing first, start with stakeholder interviews
By conducting stakeholder interviews prior to other research, we can clarify what the project stakeholders already know and what they want to find out. It’s important to run them first because they provide an understanding of the current situation and set the direction for the rest of the research activities.
If your research plan involves conducting stakeholder interviews and user interviews simultaneously, you won’t be able to apply what you learned from the stakeholder interviews to the planning of user interviews. This means you miss the opportunity to iterate on your findings before jumping into user research.
As the saying goes, ‘haste makes waste.’ Trying to do too much at once in a hurry can compromise the quality of your results.
In fact, you can learn an important lesson about user research from a dishwasher. If you cram a lot in, nothing gets very clean.
2. Go multidimensional and in-depth
Just conducting many interviews won’t necessarily lead to deep insights. After talking to around five or six people, you often hear the same things repeated over and over.
Triangulation is a great way to build multidimensional insights. It involves combining diverse methods to collect data, such as qualitative and quantitative research methodologies, or behavioral and attitudinal data. By connecting the dots gathered from different approaches, we can see something unforeseen that might have been missed by a one-dimensional approach.
It’s important to delve into users’ thoughts by asking follow-up questions during interview sessions. If you’re working on a website redesign, simply asking questions like ‘What kind of problems do you have with the current website?’ may only bring superficial answers. You need to understand why people use the website in the first place, their underlying goals, and motivations. That’s why UX researchers often conduct ‘contextual inquiry’ because the context can provide richer stories behind user behavior.
Asking ‘Why?’ multiple times is also a good way to uncover the hidden thought process. For example, if you ask people at a supermarket what they bought, they may answer ‘milk.’ The result is that ‘People buy milk at the supermarket,’ which is too obvious and predictable. But if you ask ‘Why?,’ you might find out that some people buy it for making pancakes, carbonara, or even for washing their face. Asking ‘Why?’ can reveal their lifestyle, mindset and root-causes, leading to great insights.
3. Include extreme cases
Often the lovers or haters of a product can be the canary in the coal mine.
When conducting user interviews, UX researchers often include extreme users in addition to the typical target audience group.
For example, if we want to learn about milk consumers, we can talk to the majority who may drink it or use it for cooking. Learning about them can help us validate what we already know. However, extreme cases like someone who makes lotion out of milk, buys milk to collect the cartons, or is a ‘milk critic’ on YouTube, can lead to unexpected discoveries that we never thought of before.
4. Take time
Designers are not vending machines. Unlike AI, creators need time to collect and digest data and allow their subconscious mind to generate ideas. If the timeline doesn’t allow them to fully analyze the materials, look at the facts from different angles, and experiment with fitting ideas, then the quality of the output will inevitably drop.
This is a simple concept of trade-offs. You have to choose between giving less time and getting ‘obvious results,’ or giving enough time to get ‘great insights.’ Additionally, if designers’ calendars are filled up with endless meetings, their productivity will suffer. To bring out ‘amazing insights,’ it’s important to understand how the creative process works.
5. Tell stories
Even if we discover a lot of new findings during the research process, the results can still appear dull due to poor presentation. Don’t fill your 200 pages of slides with numbers and graphs just to show off that you have done a lot of work. It’s unlikely anyone will find that attractive.
Structure your report with powerful stories, and use supportive data that are enticing and relevant — such as videos of people actually using your product or service, memorable quotes from interviews, and counter-intuitive statistics.
Also, present clear action plans so your audience can understand how they can move forward with it. Don’t create any report that makes people think ‘So what?’
6. Rethink if it is really fair to hear ‘I knew it’
We tend to speak and think as if we already knew what happened in the past. This is called ‘hindsight bias.’
The hindsight bias is our tendency to look back at an unpredictable event and think it was easily predictable. It is also called the ‘knew-it-all-along’ effect.
— Why do we see unpredictable events as predictable after they occur?
It’s often easy to say ‘I knew it’ after something has happened. However, if you feel the urge to say it, you need to stop right there and rethink if you truly did know that. Do a reality check and don’t fall into the trap of overconfidence.
If someone gives you feedback of ‘I knew it’ on your research report, it might be that they feel like they knew it in hindsight because they already read your report. This might not be a fair comment.
As mentioned in point 1, it’s important to track records of what we already know and what we don’t know before conducting any user research. This way, we can set a research plan that aims to uncover the facts we don’t know. Additionally, this helps us protect ourselves from being belittled by people who claim they already knew everything.
Summary
- Start with stakeholder interviews to clarify both the facts we already know and the expectations for the research. Proceed to the next step only after achieving sufficient clarity.
- Combine different types of research methodologies. Keep in mind that a mix of 5 interviews and a survey is often more effective than conducting countless interviews alone. Go in-depth by continually asking “why.”
- Include extreme users when recruiting.
- Allow ample time for connecting the dots and generating “Aha!” moments. Remember the concept of trade-offs: less time often results in poor quality.
- Present your findings in a compelling narrative, fortified by enticing supportive data.
- Reconsider if any ‘I knew it’ feedback is really fair.
Articles written around this topic
- How exactly do you find insights from qualitative user research?
- Creating a User Insights Knowledge Base to keep track of research findings
- Hindsight Bias: Why You Make Terrible Life Choices
- To produce actionable UX research insights, ask well-formed questions
How you can create non-obvious UX research insights was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.