Getting accessibility testing right

When it comes to accessibility testing, there are three approaches. Automated testing, manual testing and user centred testing. All three have their uses, and all three have their limitations. Understanding how they fit together is the key to successful testing.

Automated testing

Automated testing is carried out using specifically designed software or web based services. Once you’ve set up your tool of choice, you can go and grab a cup of tea whilst it gets on with the job of testing your website. The benefit of automated testing is that it covers a large number of pages in a very short space of time. It’s also very convenient, and very easy to run repeat tests at regular intervals. The drawback of automated testing is that it doesn’t check everything. Something less than 50% of WCAG 2.0 success criteria can be checked automatically for example, and many of those can only be checked partially. A good example is text descriptions for images. An automated testing tool can tell you whether the text description is present, but it can’t tell you whether the description is any use to a person at all.

Manual testing

Manual testing is carried out by a person, usually with strong accessibility and technical knowledge. It’s a hands on process that really lets you get into the detail of every web page. The benefit of manual testing is that it’s extremely thorough. Every page can be tested against all 61 WCAG 2.0 success criteria for example, which means that it’s possible to gather a lot of detailed information. The drawback of manual testing is that it takes time. A lot of time. Evaluating a website with a rich feature set and masses of different content against 61 success criteria takes a lot longer than making that cup of tea.

User centred testing

User centred testing is carried out with people who’ll be using the website. It can be done with any group of people, but for accessibility testing the focus is usually on people with different disabilities. The benefit of user centred testing is that it gives you an insight into the experiences of people as they explore your website. It goes beyond the technical guidelines, and joins up all the dots in terms of understanding the challenges different people may face. The drawback of user centred testing is that it’s often out of budget, or only factored in at a single point. It can also be hard to find people to take part, particularly if you’re trying it for the first time and have yet to make contact with relevant groups.

Accessibility testing

Each of these testing techniques has strengths and weaknesses. Bringing all three together to complement each other offers the best approach of all.

  1. Use automated testing for regular accessibility snapshots. It won’t tell you everything you need to know, but it will help you keep an eye on things at frequent intervals. The results may also flag up trouble spots that will help you focus your manual testing.
  2. Use manual testing to focus on a few sample pages at a time. Try to encompass as many different page types within your sample to get the best coverage. Focus your manual testing on trouble spots highlighted by automated testing, and use the results to really understand how to put things right.
  3. Use user centred testing to get in touch with your audience. At the end of the day, accessibility is about people not technology, and user testing will often put things into a completely different perspective. Use the results to prioritise the outcomes of the manual testing into a practical action plan.