Why listening to users is less important than observing their actions.
In UX research, we constantly face obstacles that can easily skew the outcome of our work. One of the most challenging issues you can face is dishonesty of interviewed users. I often meet designers who strive to incorporate all feedback they collect in user interviews and surveys, even if it comes from one person. While gathering qualitative data is a decent method of learning more about users, many insights are often too logical to be valid in everyday life and true for every user. Why? Because what people say they do is not always in line with what they really do.
In one of my recent interviews, the respondent was shown a redesigned version of our website. Our main goal was to check if we’d missed any scenarios and if all the information customers saw across various pages would suffice their browsing needs. Here’s a sample quote from that interview:
– Imagine that as the next step, you land on this page. What do you think you’d do here next?
– I’d probably apply some filters to narrow down the results.
– And which filter would you apply first?
– I’d start with adjusting the price range.
In the given scenario, was the respondent’s answer logical? Absolutely. Is that what that user would really do in that situation? The correct answer is: we don’t know.
By including the short quote above as an example, I’m trying to illustrate that there is a striking difference between asking users what they would do and watching what they actually do. Just like Henry Ford, the founder of Ford Motor Company, once said:
“If I had asked people what they wanted, they would have said faster horses.” — Henry Ford
Although his words were more of an assumption about customer needs, this quote shows that Ford was skeptical about their input into his work.
Many decisions we make daily are subconscious and driven by emotions rather than logic. For instance, let’s think about how you make a purchase decision when you look for a laundry detergent in a supermarket. How do you decide which product to buy when you stand in front of an endless shopping aisle crammed with similar products? Would you look for the one you usually buy? What would happen if that product was sold out? Would you try to come back to the store a day later? Would you try to save some time and buy a different product that’s in stock with the most appealing packaging? Would you test out a laundry detergent with a more convincing slogan promising the product to work 10 times better than all others combined?
According to Gallup, an American analytics and advisory company that has conducted extensive research in behavioral economics, 70% of decisions we make are based on emotion, and only 30% are based on rational factors. Looking at this data, can we say that designers and researchers should entirely trust users in what they say without scrutinizing or challenging their actions in real life?
As you conduct user interviews, you meet people of various cultures, having diverse life experiences and personalities. It’s crucial to remember that among your participants, there will be people less likely to openly point out all issues they see in the tested solution. For example, depending on certain personality traits, some people may be more afraid of being judged than others, and they will strive to avoid openly sharing ideas that would portray them in a bad light. For instance, some users can feel ashamed to point out that they aren’t able to read text because of a small font size. They may try to come up with other explanations, such as low vision, wrong glasses, too little screen size, while essentially it’s just bad design.
To verify who would be more likely to either share or not share subjective opinions with other people, psychologists introduced a widely used test in the late 1960s named FNE, which measures the fear of negative evaluation on a 30-point scale. The higher your score, the more likely you are to keep your views away from anybody else. Respondents worried about what others may think of them will be more likely to rationalize their decisions and share only the feedback you’d like to hear. However, in the context of user research, any false responses can skew all of your qualitative data. That’s why it’s crucial to seek honest, subjective opinion and make sure that the tested user feels comfortable enough during the interview to share both positive and negative feedback.
Cultural background is another aspect that plays a crucial role in your respondents’ openness and can impact the outcome of user research. Just like researcher Wei Cai explains in her publication titled: Differences in Willingness to Express One’s Opinion in US and Chinese Online Consumer Interactions, in the US and many European countries, the society and school teachers put a strong emphasis on individualism and teaching how to openly express personal opinions from a very young age.
On the contrary, in many Asian countries, particularly those greatly influenced by Confucianism, such as China and South Korea, there is a stronger push towards living in harmony as a society. As a result, communities play a key role in “managing” individual views if they oppose a generally accepted idea by the majority. Moreover, societies strive to guide individuals toward what’s widely perceived as right while limiting unconventional thinking. Thus, your interviewees who grew up in those beliefs are likely to use more indirect and avoiding communication style in unfamiliar situations to refrain from conflicts.
On the flip side, many Asian societies emphasize the importance of paying respect to older people — I’m not referring to just seniors but generally people older than you. Thus, when you speak to younger individuals, they may feel too uncomfortable to openly and fully share their thoughts as that’s what they were taught to do by their parents and society.
However, the aim of conducting user research is exactly the opposite. As UX researchers, we intend to collect the truth about our target users, what they do, and what makes them struggle when they use our solutions. Only if the responses are valid and relevant—our insights can bring real value.
Another important aspect we should think about when analyzing qualitative data is response bias. According to research conducted by psychologist Adrian Furnham, people have tendencies to falsely or inaccurately respond in surveys and interviews (source). In other words: they lie, and any untrue facts lead to inaccuracies in the collected data.
People lie for various reasons — they may wish to sound reasonable purely to themselves, boost their confidence levels, or attempt to reassure the interviewer they are the ideal respondent for the interview. For instance, when you ask someone how frequently they’d used your service in the past, they may provide an incorrect answer to reassure you that you’re interviewing the right person. Your respondents, just like you, want the interviews to go as planned because they are often paid for their input. On the other hand, they may also feel excited to help you improve a widely recognized product. But in the end, the biggest obstacle hindering your relationship with your respondent is trust. Whatever you hear, you’re always forced to trust your respondents, and any seemingly innocent lies added on the spot can impact the end solution.
According to research revealed by BBC,
“A third of people in the UK will not give truthful answers about themselves when asked questions by pollsters.”
This rather disappointing data illustrates the importance of verifying participants’ responses and analyzing any factors that could skew the results.
Subconscious Decision-Making
If you ask your users why they selected a green button instead of a red, they will always strive to justify their decisions logically, as that’s what you intentionally prompt them to do. However, many decisions people make daily are made unconsciously, and naturally, we often don’t know the reason behind selecting one option and not the other. By asking for reasons, you force the user to provide a justification, even when there’s none. Consequently, you can unintentionally add irrelevant insights to your research.
1. Should You Always Listen to Your Interviewees?
Dale Carnegie, an American writer famous for his articles on psychology and self-improvement, once wrote this summary:
“When dealing with people, remember you are not dealing with creatures of logic, but with creatures of emotion, creatures bristling with prejudice and motivated by pride and vanity.” — Dale Carnegie
In his pieces of work, he always tried to emphasize that people’s actions were not often logical.
2. Consider Your User Research Goals and Expectations
It’s crucial to remember that you shouldn’t expect solely logical answers from your respondents. Instead, seek emotional and less rational explanations too. A simple “I don’t know” can be more helpful and motivating for further analysis than a forced and untrue feedback that’s misleading.
3. Don’t Just Listen. Watch What Your Users Really Do.
If you’re looking for an answer why one solution works better than the other, avoid relying purely on verbally expressed opinions. Instead of analyzing feedback word by word, focus on observing your users and verifying what they really do and how they do it. Just like Nielsen Norman Group suggest:
“The way to get user data boils down to the basic rules of usability:
1. Watch what people actually do.
2. Do not believe what people say they do.
3. Definitely don’t believe what people predict they may do in the future.” Source
4. Test Your Users and Verify Their Actions
User research involves looking at all sorts of data, including watching and verifying actual behaviors. It’s natural that we tend to rationalize our actions, which is why listening is not enough.
‘Countless times I have heard statements like “I would have seen the button if it had been bigger.” Maybe. All we know is that the user didn’t see the button.’ — Jakob Nielsen. Source
We defend our actions with logical explanations to avoid embarrassment and unnecessary conflicts, as well as to demonstrate our rational decision-making process not just to others but also to ourselves.
We are all creatures of emotion and illogicality—let’s not pretend we are not.