Diary Studies: Understanding Long-Term User Behavior and Experiences

Defining Diary Studies

A diary study is a qualitative user research method used to collect insights about user behaviors, activities, and experiences over time and in context.

During a diary study, participants report their interactions and experiences as they occur over a period ranging from a few days, weeks, to a month or longer.

For example, in a diary study investigating how nurses interact with patient health records in a hospital setting, we may ask nurses to log all their activities involving health records.

Collecting contextual, self-reported insights from participants over time makes diary studies great for understanding how users behave within the context of their everyday lives.

Diary Studies vs. Field Studies and Contextual Inquiries

Like field studies and contextual inquiries, diary studies are a type of context method. All these methods are used to understand users’ contexts and environments. However, field studies and contextual inquiries involve directly observing users, usually in person, and can be costly.

In contrast, diary studies are done remotely and asynchronously, allowing researchers flexibility and access to distributed users. Diary studies can often be conducted at a lower cost than field studies or contextual inquiries.

Diary Studies vs. User Interviews and Usability Testing

Contextual and longitudinal insights cannot be collected through single-session moderated user-research methods, like usability testing or user interviews. These types of methods typically remove users from their personal context and take place over a short time.

When to Conduct a Diary Study

Diary studies are useful for exploring a wide variety of research questions about long-term experiences and repetitive activities. You might decide to conduct a diary study when you have research questions about the following aspects of an experience.

Aspect of the Experience	Example Research Questions
Habits and frequency	What time of day do users engage with a product? How often do they interact with the product?
Attitudes and motivations	What motivates people to perform a specific task? How are users feeling and thinking as they do something, and over time?
Changes in behaviors and perceptions	How learnable is a system? How loyal are customers over time? How do they perceive a brand after engaging with the organization over time?
Customer journeys	What does the typical customer journey look like? What is the cumulative perception of experience after interacting with multiple service touchpoints?
Channel and device usage	What channels or devices do users interact with, and why? What motivates them to select one over another?
External factors	How do users collaborate with other people as part of their engagement with a product or service? What competitors, third-party sites, or supplementary tools do they use as part of their processes?

The focus of a diary study can range from very broad to extremely targeted, depending on the topic being studied. Diary studies are often focused on one of the following:

• Broad behaviors
• Targeted product usage
• Targeted activities

Diary Studies with Broad Behavioral Focus

Researching general activities or behaviors helps researchers understand users’ mindsets, mental models, habits, and strategies.

Example:

How do people use intelligent assistants like Amazon Alexa or Google Assistant?

Diary Studies Focused on Targeted Product Usage

Studying users’ interactions with a particular product over time gives insights into their motivations, usage patterns, and comprehensive experience using that product.

Example:

How do people use a specific meal-delivery application in their everyday lives, and what is their overall experience and perception of using it?

Diary Studies Focused on a Targeted Activity

Targeted research around specific activities that take place over time helps researchers understand how users approach broad goals that often include multiple interactions.

Example:

How do customers go about researching and purchasing a new mobile phone plan?

These types of diary studies may target activities involving one particular product or service or may focus on how users approach long-term activities in the larger marketplace.

How Data Is Gathered

Diary studies get their name from the way this type of research was traditionally conducted. Research participants were asked to keep a physical diary, documenting relevant behaviors and experiences during a defined period of time.

Today, we have access to a host of digital tools that can make running diary studies much more efficient for both participants and researchers. However, the underlying method is still the same. Participants are asked to log specific information about activities being studied.

Selecting the right tool(s) for data collection is a decision that should be made based on a variety of factors. We discuss tool selection in detail later in this article. However, one of the primary considerations in tool selection will be driven by your research questions and the focus of your study.

The Diary-Study Timeline

Due to the long-term nature of diary studies and the added complexity of directing participants on how to share insights, these studies require substantial planning and preparation.

After defining the goals of the study, you will need to consider a few important aspects (and document them in your study plan):

1. When Participants Should Create Diary Entries
2. Length of the diary study
3. Number of participants and their profile
4. Incentives and response requirements
5. Tools
6. Communication templates and supporting materials
7. Pilot study
8. Pre-study brief
9. Post-study interview

1.  When Participants Should Create Diary Entries

There are three common methods for gathering insights during diary studies: event-based, interval-based, and signal-based. Each study’s unique research goals, questions, and focus will help researchers determine the best method for gathering information from participants. Many studies use a combination of these methods.

Method	When to Use It
Event-based (also known as event sampling or in-situ logging) Participants are asked to record an entry when an event of interest occurs.	If you want specific details about every related interaction If interactions are not so frequent that logging each would be too time-consuming or cumbersome
Interval-based Participants are asked to report an entry at regular intervals.	If you don’t need specific details about every unique interaction, and you’re okay with capturing details about several interactions within a set interval of time If interactions are so frequent that logging each would be too time-consuming or cumbersome If the timing of the interval won’t result in skewed insights (For example, reporting each afternoon might result in less detail about potentially unique morning activities.)
Signal-based (also known as experience sampling) Participants respond when prompted by a “signal” such as  a text message.	If you don’t need specific details about every unique interaction and you’re okay with capturing details about several interactions within a set interval of time If activities may be varied within set intervals, and you want to capture a mix of insights across different times of day If you’re concerned participants may forget to report on their own

2. Length of Diary Study

Consider for how long you will need participants to report their interactions. The length of a diary study should depend on:

• Your research questions
• What behavior you’re trying to capture
• How frequently these behaviors might happen
• How long it takes typical users to complete a longitudinal activity

You may need to do some exploration to understand the typical frequency or length of activities, but ultimately, the reporting period you select should be one that will give you enough data points across your whole participant sample.

Research Question	Expected frequency or length	Reporting period
How do people use intelligent assistants like Amazon Alexa or Google Assistant in their everyday lives?	A few times a day	1 week
How do people utilize a specific meal-delivery application in their everyday lives?	Once or twice a week	3 weeks
How do customers go about researching and purchasing a new mobile phone?	About 2 weeks to  make a purchasing decision and shop	2 weeks

3. Number of Participants and Their Profile

Consider who and how many participants should be part of your study. Your participants’ typical behaviors should match what you’re looking for in the study.

For example, a study about food-delivery applications may need to involve participants who use such applications several times a week.

Because a diary study takes place over a longer time than a regular usability-testing study, there is a higher chance that participants will drop out of the study due to unexpected circumstances in their lives. Consider overrecruiting in preparation for potential dropouts so that you have enough data points in the end.

Also, consider how many participants will give you enough data to answer your research questions. You want enough participants to reach saturation — that is enough data to ensure your themes will be well established.

Saturation refers to a moment in qualitative research when your data becomes repetitive — you hear the same thing again and again. After this point, there is a diminishing return with any additional participants. In a diary-study context, saturation happens when additional data no longer changes your understanding of the behaviors you’re studying but instead reinforces the themes you’ve already identified.

The best sample size to reach saturation depends on how broad your research questions or the problem space are and on how varied or homogenous your target user group is. The more variety in your user group, the more participants you will need to recruit to get a representative mix of users. And, the broader the research questions, the more people you will need to recruit to ensure enough insight coverage across all questions.

Below we outline some rough guidelines for sample sizes based on these criteria.

	Variance of user group characteristics	Scope of research questions	Approximate number for saturation
Small discovery projects	Fairly homogenous user group	Problem space is relatively small	~ 5–12
Large discovery projects	Fairly heterogenous user group	Problem space is fairly large, with many unknowns	~ 12–30 people
Large research projects or academic research	Very broad user group	Research may be intended to produce generalizable knowledge or theory about a topic	~ 30–50 people

4. Incentives and Response Requirements

Be clear and specific about what participants need to do in your diary study so that people understand what is expected of them and you get the data that you need.

It’s useful to designate a minimum number of responses you expect from participants in exchange for the study incentive. However, this minimum should be realistic to ensure natural behavior from participants.

Expect that your participants’ number of responses may vary. In some situations, you might allow participants to report activities beyond the minimum and earn more for doing so. However, if you do so, you should also designate a maximum number of responses that participants can submit, to avoid having respondents manufacture fake interactions to earn more money.

Diary-study incentives should be higher than those for a typical user-testing session due to the long-term engagement required. Consider the length of the study and the effort required for participants to provide all the information you need, utilizing the tools you select.

As a broad guideline, we recommend about $40 per hour of the total time commitment you expect for a study with a general sample of US-based participants. However, this number should be adjusted accordingly for highly specialized audiences.

For studies longer than a week, think about how you might keep users engaged throughout the length of the study. You can break apart the total incentive and offer smaller installments as participants reach specific milestones (e.g., 3 days of logging), to keep them motivated throughout the duration of the study.

5. Tools

Think about the type of information you need from your participants.

• Is the information simple or complex?
• Do you need basic feedback about behaviors and interactions, or do you need screenshots, images, videos, or screencasts as well?
• Do you have specific questions that each participant should answer about each interaction they report?

These factors will influence the tools you choose.

• For simple text responses, you might choose to use a messaging application like Facebook Messenger, WhatsApp, or Telegram.
• For more complex data, consider tools like Google Forms, Google Docs, Survey Monkey, dScout, or Recollective.

Beyond the type of data you need, there are other considerations to factor into your the choice of a  tool.

Data Security

If your responses are likely to include personal identifying information or private data, select tools that guarantee data security.

Participant Convenience

Think about the tools that are most convenient for your participant sample. For example, teenagers may prefer to report activities through Snapchat rather than email.

Cost

Some tools are free,others can be quite expensive. Free tools are usually general-purpose and not tailored to diary studies, so they often involve a lot of work to onboard participants, write specific directions, communicate with participants, and process the data after the study.

Specialized diary-study tools like dScout are costly, but they alleviate a lot of the researchers’ overhead. They streamline the setup and facilitation of these studies and make them easy to run.

Type of tool	Benefits and drawbacks	Example tools
All-in-one tools	✅ App based: participants follow prompts, record videos, upload photos, and answer questionnaires ✅ Easy export: some support filtering and tagging, transcription, and data visualizations ❌ Expensive	dScout Flex Indeemo Recollective FocusVision
Survey & document tools	✅ Great for long-form responses ✅ Free ❌ Difficult to encourage responses automatically ❌ Prompts sent via another medium ❌ Difficult to share images and videos	SurveyMonkey Google Docs Google Forms
Messaging apps	✅ Easy dialogue with participants ✅ Some automation of prompts possible ❌ More difficult to export entries	Slack Facebook messenger WhatsApp WeChat

6. Communication Templates and Supporting Materials

You should plan to monitor your participants’ responses as they submit them. If you review data as it comes in, you can ask follow-up questions and prompt them for additional details while the activity is still fresh in their minds.

Let participants know up front that you will be reaching out throughout the study and agree on a means of contacting them so you can give encouragement or ask for clarification without being overly intrusive. Give periodic reminders each day or every few days. You’ll also want to track the qualifying submissions in relation to the reporting timeline.

Regardless of the tool you select, some preparation will be required.

It’s a good idea to create templates for various messages you will send to participants, as well as documentation and instructions in advance.

Documentation you should prepare includes:

• Recruiting materials
  • Study description
  • Screener
  • Consent form
• Onboarding materials
  • Welcome message
  • Informational packet: Directions for participation, requirements, incentive details, example logs, detailed instructions for using selected tools, schedule
• Ongoing communications
  • Any proactive logging prompts (Consider using a scheduling tool for automated delivery.)
  • Status and check-in message templates to streamline periodic communications about the number of interactions provided and what is still required
• Wrap-up materials
  • Post-study interview guide
  • Thank you message and incentive delivery

Present the study and expectations in simple and straightforward terms, using consistent terminology throughout. Be as specific as possible about what information you need participants to log without stifling natural variability and differences that you cannot plan for.

Give users example log entries to help them understand the level of detail you need from them. But make sure you don’t bias participants toward those types of entries that you happened to provide as examples.

Communicating periodically with your participants ensures that they stay on track and that they provide you with the right data. Create template messages for the following types of participants:

• Engaged and submitting useful responses: recognize their efforts and ask them to keep up the good work.
• Less engaged or producing incomplete data: give encouragement or offer to answer any questions that may help to get them on track.
• Disengaged or submitting responses that do not fit the reporting requirements: reiterate what they need to do or possibly release these participants from the study if participation issues aren’t corrected.

Diary studies are quite complex, so writing and reviewing all supporting materials in advance will eliminate the risk of confusion and set participants up for success.

7. Run a Pilot Study

Diary studies are fairly expensive and resource-intensive, so it’s helpful to conduct a short pilot study first. Ask pilot participants for feedback about materials and the diary study experience and adjust accordingly.

The pilot study does not need to be as long as the real study and is not meant to garner data for analysis. Its purpose is to test your study design, tools, and related materials, and practice the process. It will allow you to tweak your instructions and approach to ensure you get the data you need.

8. Pre-Study Brief

In a diary study, a pre-study brief is a short meeting you conduct with participants in advance of the reporting period to get them ready for the study. You should communicate what they should report, the required number of reports, and incentives, and answer any questions they may have.

For very simple studies, a pre-study brief may not be necessary — your onboarding materials may be able to serve this purpose.

However, if your study has a complex setup or if it requires users to report complex information or to use unfamiliar tools, take the time upfront to get participants ready to log their responses.

Schedule a meeting with each participant to discuss the details of the study. Walk through the schedule or calendar for the reporting period, answer questions, and discuss expectations.

Discuss the tools they will be using and be sure each participant has familiarized themselves with the technology; answer any questions they may have before beginning.

9. Post-study Interview

After the study, you will evaluate all the information provided by each participant. Plan a follow-up interview with each participant to discuss their responses in detail. This is your last opportunity to get insights from your participants before the study ends.

Ask probing questions to uncover specific details needed to complete the story and clarify as needed. For example, if you’re studying how people use meal-delivery applications, you may ask a participant to clarify a vague response. For example,  “I see that you said you were unsatisfied with the experience using this app. Can you elaborate a little bit about why you were unsatisfied?”

You might also ask general reflection questions about the broader experience you are studying, such as “Overall, what did you like or dislike about using these meal-delivery applications over the last few weeks?”

Data Analysis

Because diary studies are longitudinal, they generate a large amount of qualitative data. Re-visit your research questions before you dig into all the rich insights you’ve collected to find the answers.

Evaluate the behaviors you’ve captured throughout the study. How did they evolve and change over time? What influenced these behaviors? If the focus of your study was around a particular product or service relationship, look at the entire customer journey.