Nothing can replace the eyes of a skilled professional when it comes to web accessibility testing. This is because so much of determining a website’s accessibility is subjective enough that an automated tool cannot help you when it comes to making definitive judgments about whether disabled people who visit your site and interact with it can actually use it.
But before I tell you why automated web accessibility testing is bad in more detail, I want to tell you what it’s good at, because despite not being a complete solution, there are some aspects of accessibility testing they excel at.
Piles of Code
A good web accessibility audit involves an in-depth, manual look at all the generated source code that makes up a website. Doing this is the best way to find out if problems exist, where they are, why they’re occurring, and how to fix them once they’re found. Doing everything manually is great if the website I’m working on is small. But once websites started becoming larger, web accessibility professionals needed a way to quickly scan through thousands of lines of generated source code to get a basic idea of where things stood.
This is where automated tools come in. They help by providing a general layout of the accessibility landscape, and they provide a list of things which need further attention in a manual audit. When you’re staring at a page that contains hundreds or thousands of lines of JavaScript, HTML and CSS, (like, for example, the Yahoo homepage or the Facebook homepage), automated tools can be a great help. On large sites, there really is no other way to find all the errors than automating the task. This includes sites that are using a templating system, such as WordPress sites. When conquering these, you’ll need to use an automated tool that crawls the entire site to get the assessment done. When I began working on Utility Pro, running it through an automated tool was the first thing I did in order to find out what I needed to start work on first, and in what priority.
Some errors don’t need human inspection
One of the best examples of errors that an automated tool can save time on is images without alternative text. In cases where there is no alternative text, an automated tool is used to document every occurrence, instead of an auditor manually going through and finding each one. But in cases where images have alternative text, and it’s not meaningful, a manual audit will find these and suggest meaningful text.
An automated tool lays the groundwork for a manual audit
The most appropriate way to use an automated web accessibility tool is not to generate a comprehensive list of pass or fail judgments on the site being tested, but to give an auditor a list of things that need a closer look. A good tool will provide a list of errors with line numbers and snippets of code that’s causing them. This information helps an auditor decide whether the problem really is a problem, so they can then tell the client why it’s a problem and suggest ways to fix it. More importantly, it’s a good way to find patterns of errors, which then helps an auditor determine the best way to help the client integrate accessibility best practices into their workflow.
Automated web accessibility tools can also help find exceptions in an otherwise accessible site. I’ve created sites or added content which contained good markup, but since underlying tools may change what happens to that markup, I’ll run the site through an automated tool and sometimes find errors I hadn’t realized were introduced. So in short, automated tools are great for sanity checks, and make sure that a manual audit doesn’t miss the obvious.
And now, the bad
Most of the bad things that get mentioned about automated web accessibility testing are related to the subjective nature of accessibility. There are also numerous ways to make a site more useable for people with disabilities that an automated tool will never alert you to. Automated tools can offer false positive and false negative results, and they have a lot of trouble with dynamic features, which have become more and more prevalent on the web and are not going away any time soon.
Now for some statistics
While working with WordPress and the Genesis framework, I’ve found that automated tools give me about twenty percent accurate results when it comes to web accessibility best practices. Thirty percent of what the tool finds I can use as guideance for further manual review. That leaves fifty percent that is straight up manual review. That fifty percent includes everything related to forms, and, since automated tools can only test HTML and CSS, also includes JavaScript, AJAX, and any other client-side scripting.
Does this mean you shouldn’t use automated tools?
I don’t mean to suggest by any of the above that you should never use an automated tool. But the thing to keep in mind is that an automated tool is just that, a tool. It can help with productivity, and helps with the accuracy of the final report. They also help with getting started with the basics of accessibility and can help developers and designers begin learning about it. But when using automated tools, in order to get the most milage, you’ll need to be well-trained in accessibility, understand how to read their results, and remember that what that tools tells you is not the final say in determining whether your site is accessible.