Should you use manual, automated, or hybrid accessibility testing?

Test early and often. This classic motto holds true when it comes to maintaining an accessible website.

But it can be difficult to know where to start and how to ensure you get accurate results. There are many free tools and checkers available online and although they are a good way to quickly scan for the more obvious issues, if you want to catch the deeper issues on your website, you should not rely solely on these.

There are three approaches when it comes to web accessibility audits and maintenance:

1. In-depth manual testing
2. Automated testing using a quality software solution
3. A hybrid approach that combines manual and automated methods

Here, we will review the advantages of each testing method so you can assess which approach is the best fit for your organization.

Manual accessibility testing

Manual accessibility testing entails a person or team of people combing through a website’s individual pages or code to check for accessibility issues.

Manual testing is most effective for checking compatibility with:

• Assistive technology, e.g., screen readers
• Keyboard navigation
• Browser plugins, e.g., HTML and JavaScript

Key advantages: The advantage of manual testing is that with trained eyes and knowledge of assistive tech, you can catch issues that might go undetected by automated tools. Even if you don’t have accessibility knowledge in-house, there are manual testing services available that give you access to accessibility experts who can review your website.  

User testing

One of the most effective ways to manually check your website for accessibility issues is to employ the help of people with disabilities. Their first-hand experience with your website is incomparable and is likely to reveal issues on your site that you might overlook.   

When carrying out user testing, you should always plan ahead to ensure your tester is comfortable and you get actionable results. Here are some things to consider before user testing:

• Have a clear scope and goals for what conclusions you hope to gain.
• Prepare test materials and ensure functioning assistive tech (e.g., screen reading software).
• Create a comfortable and quiet environment to ensure there are no interruptions.
• Carefully observe and consider all feedback. Don’t assume feedback from one person with a disability applies to all people with disabilities.
• Consider recording testing sessions to help you track and report your progress.

It’s important to note that although user testing can effectively give you insight into accessibility issues, it is not a catch-all. If you’re aiming for compliance with accessibility regulations such as the EU Web Accessibility Directive, you need to make sure you conform to the requirements set out for you.

Automated accessibility testing

Automated accessibility testing uses software solutions to automatically check your websites and documents. These automated solutions can do so by scanning your website’s code to find definitive errors or flag potential issues that might require your attention. Usually, these automated software solutions check your website against a recognized standard like Web Content Accessibility Guidelines (WCAG).

Key advantages: Using an automated solution saves you the time and effort of looking through lines of code. It also makes it easier to pinpoint recurring issues, which can be critical.

An advantage that is perhaps not as obvious is the training benefit for your organization. A quality automated accessibility solution will not simply flag errors but will also guide and educate on accessibility issues, so that the more testing you run, the deeper your knowledge of accessibility will develop.

An ideal automated testing solution should also include tools to make it easier to track your progress, prioritize your tasks, and create reports. Since reporting is a key component to maintaining compliance for accessibility regulations, a tool that can automatically generate reliable graphs that track your progress is a big timesaver.

Hybrid accessibility testing

By its nature, accessibility is an issue that requires a collaborative approach, as it is an issue that overlaps across different departments, like web, content, design, and QA. Using a hybrid mix of manual and accessibility testing therefore gives you the flexibility it takes to achieve an accessible website as efficiently as possible.

Key advantages: Hybrid accessibility testing leverages the benefits of automated and manual testing by allowing organizations to customize the testing according to their needs and goals. For example, if you do not have accessibility-savvy people within your organization and manual testing services are not an option, automated testing software can handle the bulk of the work. The manual load is therefore reduced and more manageable to take on in-house.

Other factors to consider when mapping out a hybrid testing approach are:

• the size of your website
• the number of websites you’re responsible for
• how accessible and usable your Content Management System (CMS) is

Assisted testing

Assisted testing is a semi-automated way of testing in which you answer questions that help the automated software diagnose your web content.

This type of testing is a great example of how automated and manual testing approaches can be combined to achieve accurate results. It gives you the ease of use of automated testing and the ability to fine-tune the results with human knowledge.  

The advantage of assisted testing is that you do not have to be an accessibility expert. With basic tech knowledge, anyone on your team can use this solution to test more quickly, while also gaining more insight into accessibility issues.

Choosing the right method for your organization

Testing your website for accessibility issues is an ongoing process that should be a part of your team’s regular workflow. That is why it is important to talk with your web, QA, content, and design teams from the very start, as these are the teams who will work most closely with accessible practices on your website.

When choosing the right testing method for your organization, you do not need to choose one or the other. For the most optimal results, try mixing automated and manual testing approaches depending on your needs, the number of websites you will test, your budget, and your accessibility knowledge in-house.

Let’s review a few different scenarios that show how automated and manual testing can work together.  

Automated testing early in the process

Consider taking a bottom-up approach by requiring that all code is checked with an automated accessibility tool, such as the Siteimprove Accessibility Checker Chrome Extension, before it is considered done. By employing a solid base of automated testing early in the development process, you can kill issues at the root. If your team is responsible for multiple websites with multiple pages, this simple addition to your web team’s work process can save your organization hours of manual work, compared to if you had to fix issues after the code has been implemented on your website.

Follow up with routine manual testing

That’s not to say you should eliminate manual testing altogether. You should plan for user testing at regular intervals. This manual testing method is an integral way to assess your website’s actual usability with a human perspective that cannot be replaced by software. In this way, you can be sure that all your work designing an accessible website is indeed producing the best user experience for your website’s visitors.

Track your progress with automated testing tools

In the daily hustle and bustle, it is easy to put off tracking and documenting your progress. But then you’re left with the task of retracing your steps when it comes time to send in your reports to management. This is where an automated testing solution with built-in reporting tools is essential. With these automated solutions, you can track issues, assign tasks, and document your results all in one go.

Whichever testing method, or combination of methods, you choose, you are sure to gain a better understanding of accessibility and be well on your way to creating an inclusive user experience that everyone can enjoy.  

Want to get started testing your website? Try the Siteimprove Accessibility Checker to check your website for accessibility issues and see instant results.