After my previous blog on the Two Pillars of Accessibility, I was asked on LinkedIn from a peer in the accessibility field how a program could measure and report the empathy pillar of an accessibility program i.e. how much does your company care about accessibility. Measuring the impact of accessibility is relatively easy — you can track the quantitative aspects of issues: number of bugs, number of tickets, number of issues, testing errors, and other accessibility issues found during the product development cycle. These immediately show your impact analytics for accessibility and these generally should be collected by every accessibility team as Key Performance Indicator (KPI) or milestones for that program.
However, how do you measure quantitatively for how much people care when there are no obvious metrics? Unfortunately, the answer is that they must be manufactured and generated with programs and processes that may also need to be created. In this article, I will describe three possible ways of measuring empathy that you can use as metrics or KPIs for an accessibility program.
One disclaimer I want to make at the outset is that these measurements are purely meant for the empathy of accessibility in building products and services. A company’s treatment, culture, and empathy of internal employees with disabilities and their accessibility needs is a whole other set of challenges and I would point you to experts who are far more versed in these areas of accessibility.
Internal Employee Experience Survey
The first suggested way to determine the amount of empathy for accessibility is to measure it in a company-wide survey. Ideally, if your company has a survey sent out on a regular basis to measure other areas of employee satisfaction adding a few questions about accessibility is a great resource to tap into. If this resource doesn’t exist, or for other reasons you can’t add on questions to this survey, then you will need to create a survey on your own (and make sure it’s accessible!). However, getting folks to respond to just a few survey questions on a specific topic can be like pulling teeth and adds more time needed to collect these metrics.
The key here is to focus your questions to not focus on specific knowledge, but to keep it generic and remove open-ended questions. Here’s a few examples of what to ask employees:
- Rate their knowledge of accessibility on a scale of 1–5
- How many trainings has the employee attended, or courses taken, on accessibility
- Rate how important is accessibility to them on a scale of 1–5
- How many times accessibility is brought up during your product development life cycle
Avoid questions that will be one sided or biased such as “Do you care about accessibility?” that will skew the results. There may be a few bad apples who answer this with “No”, but generally most folks will answer yes. The key is to dig into the psyche of the approach that your company uses. Also try to avoid answers that would ask the user to provide subjective answers and use such phrases as “How do you feel”. Utilize rating systems and scales that provide more quantitative data. Overall, your goal from this survey is to collect metrics. Getting metrics from your initial survey and then comparing it quarterly/yearly is a sign of how effectively empathy is working.
A bonus opportunity that these survey questions offer is to use these to inform training, processes /programs, and where you should add resources. If you have the resources and time you could also add role specific surveys to get more precise on your offerings to employees, but this is tricky unless there’s a mechanism or survey structure already built. Many large tech companies have a yearly engineering or design survey sent out that provides a great opportunity to focus on that area of your company’s product development.
If the opportunity is available, beg/borrow/steal some User Researcher’s time to help write and phrase the questions. I recognize that even having this role outside of a large tech or research company is a luxury, but there are always resources/acquaintances on LinkedIn that specialize in this as well. As you dig deeper and phraseology into what you want to get out of these surveys, refreshing the questions to avoid survey fatigue, as well as the subject matter expertise, makes a user researcher a very nice to have.
Metrics from Internal Accessibility Training and Programs
Another great way to measure empathy is the metrics from internal training, speakers on accessibility, Office Hours, and accessibility resources. The obvious downside is that these programs and resources will need to already exist to gather these metrics, otherwise they will need to be created in order to collect these metrics.
These programs can be challenging to establish and create buy-in from your leadership, let alone have a culture of continuing education and dedicated time for training. Additionally, unless you have experts in-house, you will need to rely on external resources and expertise to create those resources and training, and staff Office Hours. However, once established the metrics from these programs and resources can be invaluable in measuring your company’s empathy for accessibility.
There are a tremendous number of metrics from these programs. Here’s a few examples:
- The number of people who attended training
- Requests for training
- Requests for Office Hours
- Topics for Office Hours i.e. Design/Development evaluation, testing, Program/Process question
- Daily/Weekly/Monthly/Quarterly statistics on how many employees access resources
- Topic and number of requests for speakers
- Requests for resources
These metrics are much more impactful if they can be attributed and broken down by role ( i.e. Designer/Developer, Program/Product Manager), organization, and geographic location (if your company has multiple locations). These metrics can then be attributed to products/features that launch into production with accessibility issues and show the impact that training, Office Hours, speakers, and resources provided by you and your team have on final delivery. Eventually these numbers could stagnate in smaller companies so it’s important to introduce new accessibility topics/resources, such as new accessibility laws and guest speakers with lived experience.
Customer Sentiment
The third area I’d recommend to measure empathy is gauging your customers views on the accessibility of your products/services, and the opinion of customers with disabilities. This can be done in a number of ways:
- Conduct user interviews and surveys
- Search your social media for customer opinion
- Review customer reviews
- Interview previously self-identified customers with disabilities
- Review your customer support database
For each one of these examples you will want to focus on keywords, or tags, such as “accessibility”, “disability”, “disabled” and other terms related to accessibility. Try to avoid specific disabilities when using keywords as your dataset is going to be too small and is best left to to direct user research or interviews. Measuring customer sentiment requires a large amount of sifting through data and setting up (and fine tuning) queries the first time around. Once setup though there should be minimal work when automated and the data is visualized.
I would encourage everyone to partner with associations, or other organizations, that represent or specialize in working with people with disabilities and lived experience to conduct surveys and user interviews. This is a resource that I’ve used before in many different use cases, particularly when there was a specific disability or assistive technology that needed to be reviewed and tested by people with lived experience. Here’s just a few organizations I’ve worked with before, but there are many more excellent options available:
- American Foundation for the Blind (AFB)
- National Federation of the Blind (NFB)
- Accessibility User Research Collection
- Answer Lab
The best way I’ve found to leverage customer sentiment is to analyze the data to find patterns with specific products, services, and to work with the teams responsible and engage that team directly. Make sure to test the issues first and come with a specific problem, customer impact, and be ready to propose a solution. I see this often with accessibility programs where a bug or issue will be assigned to the owning team seemingly out of nowhere, and there is little context as to what the impact or solution is. It’s important to not only focus on the negative sentiment, even though that will be the majority, but to celebrate the wins of the positive feedback. Make sure those teams and individuals are given credit or a shout out to the company and their management to help emphasize the importance of accessibility.
Conclusion
Utilizing these three areas can create a holistic picture of your company’s internal approach to accessibility, and give great insight on the empathy your company has for customers with disabilities. Creating a full product lifecycle view of the company’s accessibility empathy at each step (ideation, research, design, development, delivery, and support) can be a lot of work, so break it down and tackle each area one at a time. All three of these areas also provide insight to determine if you are providing the right resources and training needed by your company for each role, technical detail, and specificity. Scaling these areas of an accessibility program takes work and time, but if highly beneficial to your company and your customers.
Measuring Empathy for Accessibility was originally published in UX Collective on Medium, where people are continuing the conversation by highlighting and responding to this story.