New ways to listen: Understanding accessibility through the analysis of feedback

The use of technology has become integral to the way we connect, learn and work – and with it, the need to ensure universal access and experience for all. Creating accessibility for employees and customers with disabilities is a highly involved process that is far from one dimensional.

In addition to strong guidelines in the development and design phases of technological platforms, accessibility must also be influenced by user feedback throughout a user’s journey of a service or process. But the capturing of user feedback is only the first step. How do organisations bring that feedback to life, to better understand underlying accessibility issues, and; how do they create structures to ensure that accessibility feedback is acted upon?

Research conducted by Tim Coughlan, Thomas Ullmann and Kate Lister (The Open University, UK) explored how different methods of analysing qualitative user feedback data can elicit rich insights into accessibility in a higher education context.

In essence, the research found that the combination of automated key word methods with deeper methods of qualitative analysis is highly effective in understanding large scale open-comment feedback data in relation to improving accessibility in online learning. Additionally, the researchers found hidden time imposts on students with a disability, which are likely to also present for employees, and that these were not considered when the duration of projects was estimated or new processes introduced.

Aim

This research primarily aimed to better understand different approaches to analysing user feedback to distill insights that could inform enhanced accessibility. The researchers also sought to interpret feedback data collected to understand opportunities to improve accessibility in The Open University specifically.

Method

All students from The Open University were invited to respond to a survey on modules they undertook between 2014 and 2015.The survey included 40 closed questions and four open comment questions. Data collected from three of the open comment questions were analysed during this study, exploring aspects of teaching, resources and assessment that were found to be helpful or unhelpful. In total, data from 93,148 open comments was collected, with 6,792 comments from students who had declared having a disability.

Three methods of analysis were undertaken to draw insights from the data collected:

  • Analysis of key words: R software was used to analyse the frequency of nouns identified in open comment text from students who had declared having a disability, compared to the broader population. This frequency was then analysed in relation to each type of disability declared
  • Categorisation of key words: Words identified as unusually frequent or infrequent were then categorised by 5 coders as either ‘Course-Related’, ‘Disability-Related’, ‘People and Organisations’, or ‘Tools and Resources’ in order to better understand how different disabilities related to different categories of comments
  • Thematic analysis: Using NVivo 10 software, the top ten key words for each category of disability were analysed thematically to validate areas for improvement.
Findings

Analysis of key words:

  • Across all respondents who declared a disability:
    • Unusually frequent key words included: disability, dyslexia, transcripts, screen reader, examiner, health issues, mini lectures, voice, help, computer skills
    • Unusually infrequent words included: readings, examples, study guide, role, theories, material, level, process.
  • These words changed when frequency was analysed across different disability type, reiterating that the issues and language used by students with disabilities varies between disabilities, as differing disabilities manifest differently in learning
  • The strong response rate and its proportionality to the overall student population indicated that students with a disability were likely to provide feedback when asked.

Categorisation:

  • A large number of key words could not be categorised within the four chosen categories and there appeared to be some area of semantic overlap between categories by the coders, which indicated a need for further investigation into suitable categories and an agreed dictionary for key words to support alignment in theming.

Thematic analysis:

  • Course related themes: Prevalent points of discussion from students with a disability included the relationship between the course structure and time pressures, particularly the form and style of course materials and aspects of the course where scheduling was less flexible to unforeseen events in the person’s life (such as exams and deadlines)
  • Disability related: The researchers found that recommended study hours did not tend to take into account disabilities, which had flow on impacts to students’ other commitments such as employment. In addition, the researchers noted that the nature of open comment feedback places the onus on the student to provide context of their disability, which if left unclear or unexplained can detract from the value of the feedback contributing to building accessibility
  • People and organisation related: The consistency in the support provided from staff to a student’s development over time was found to be critical in aiding accessibility, due to the level of confidence that students build through relationships with staff and peers over time
  • Tools and resources related: The accessibility of tools and resources such as screen readers and audio transcripts can be challenging, and there was found to be a need for greater recognition of the time required to build digital literacy.
Implications

Be holistic – Accessibility is best understood in context of the changing individual user. To enrich the use of feedback to enhance accessibility, it is critical to take into account the context of that particular user’s journey and their unique circumstances. To achieve this whilst avoiding data collection fatigue, combine feedback data with other available information (such as personal data captured at enrolment) as part of the analysis.

Embrace automation – In analysing more than 90,000 open-comment texts, the research highlighted that it is feasible to draw rich insights on a such large scale through open-comment feedback when leveraging automated approaches. However, this should be coupled with manual interpretation to allow contextual analysis in understanding the cause and effect of underlying issues.

Consistency is key – The occurrence of inconsistencies or frequent changes in processes can have a significant impact on the confidence and skills of someone with a disability and where change is necessary this should be communicated well in advance and additional support provided where necessary.

Act – It is critical to integrate feedback into organisational processes to improve accessibility. Utilising better articulation of responsibility for accessibility in the organisation, implementing feedback loop processes to formalise the flowing of accessibility data to the appropriate areas for action, and leveraging feedback collected in the design of training programs for staff to ground their understanding of their users’ experiences in real life case studies.

For more information, contact Rachael Salamito

To read the full article, see Coughlan, T., Ullmann, T.D. and Lister, K., 2017, April. Understanding Accessibility as a Process through the Analysis of Feedback from Disabled Students. In Proceedings of the 14th Web for All Conference on The Future of Accessible Work (p. 14). ACM.
https://doi.org/


Want to stay up-to-date?

Stay on trend and in the know when you sign up for our latest content

Subscribe