Something that’s often overlooked in the marketplace is not all accessibility audits are the same.
Importantly for clients, some audits are worse quality than others. Let’s use a hypothetical to illustrate:
Scenario A) You purchase a website accessibility audit from Accessible.org and get an easy to understand report back that lists and explains all accessibility issues under the WCAG 2.1 AA standard.
Scenario B) You purchase an audit from another digital accessibility company and get a report back that is easy enough to understand, but misses 15 issues, per WCAG 2.1 AA, for the pages and screens within scope. The report also curiously adds in 5 issues that aren’t actually issues per WCAG 2.1 AA.
Given all that’s at stake – including access and legal risk, that’s a big deal. You never want to pay thousands of dollars and get back a poor quality report.
How Do Bad Audits Happen?
There are number of causes for bad quality audits:
- Inexperienced or unknowledgeable auditors
- Audit farming to large agencies with no quality checks
- Auditors who are rushing too fast
- Automated scan results copied and pasted as issues
- Digital accessibility companies who think that automation can be combined with manual evaluation
- A lack of screen reader or keyboard testing
- A lack of ethics from digital accessibility companies
Put simply, if you shortchange any aspect of an audit, it can and usually will compromise the quality of the report.
What Factors Into Good Quality Audits?
There are several important factors that go into an excellent audit report, but let’s focus on three. The #1 factor might surprise you: an auditor who actually cares about accessibility.
Caring
While technical skill, knowledge, and expertise are essential for any auditor, what truly sets exceptional auditors apart is their genuine commitment to and interest in accessibility. An auditor who truly cares will never take shortcuts.
We’re extremely selective in choosing our auditors, requiring them to demonstrate a genuine and longstanding interest in accessibility. Our clients trust us with their accessibility needs, and we honor that trust through obsessive attention to quality in every deliverable.
Attention to Detail
Another big factor is attention to detail.
When auditors are meticulous and thorough, this means it’s much, much less likely that any accessibility issue is missed.
Can an accessibility issue be missed?
Of course. This is a manual process whereby a technical accessibility expert is evaluating every page and screen (within scope) of a digital asset. If there are a large number of pages and many accessibility issues, it’s easy to envision how their could be an oversight. However, if auditors are paying close attention to detail, missed issues are rare.
Diverse Methodologies
A high quality audit will involve a diverse set of evaluation methodologies. One common problem amongst audit service providers is they’ll rely exclusively on screen reader testing to evaluate digital assets.
While screen reader testing is absolutely essential, it isn’t everything. At Accessible.org, our evaluation includes:
- Screen reader testing
- Keyboard testing
- Visual inspection
- Code inspection
- Color contrast analyzer tool
We use scans to complement our manual efforts, but never replace them. And after we complete the manual inspection, we’ll use an automated scan to review our work and ensure all issues that are correctly flagged by automation are included in our final report.
What to Look For As a Client
As a prospective client, it’s virtually impossible to have the full scoop on the digital accessibility company that lands on your final list of choices.
However, there are some key observations you can make and questions you can ask to get a better feel for whether a service provider will provide an excellent quality audit:
Observations
- Does their price stay the same regardless of scope?
- This is a red flag
- Do they seemed concerned at all with scope?
- Do they list out the URLs/pages in scope?
- It’s a big red flag if they don’t discuss / ask about scope
- Do they list out the URLs/pages in scope?
- Do they promote or sell overlays / overlay widgets?
- Immediate red flag, look elsewhere
- overlays are a worst practice for accessibility
- overlays have been discredited by over 900 accessibility professionals
- Do they tout or feature automated “testing” or “solutions”?
- Another immediate red flag
- Do they inflate how what percentage of issues can be flagged by a scan?
- It’s approximately 25%, look out for answers higher than 30%
- Are they conflating user testing with auditing?
- User testing means one or more professionals with disabilities are testing the accessibility
- This is different from an audit
- This is a concern because they think that user testing and an audit are one and the same
- User testing means one or more professionals with disabilities are testing the accessibility
- Are they going over specifics or talking in generalities?
- Specifics are a good sign, generalities are a bad sign
- Do they detail their audit methodology on their website?
- Concerning if they don’t
- Can they answer your questions without hesitation?
- At no point should at accessibility company representative be unsure on questions about their audit
- Is the audit something they feature or gloss right over?
- Do they ever use the phrase “run an audit?”
- You don’t “run” an audit, you run a scan
- Do they insinuate that manual and automated “testing” can be combined?
- You don’t combine manual efforts and scans because all WCAG success criteria need to be manually accounted for (scans can have false negatives and false positives)
- Scans should be used as a complement to manual efforts, not a replacement
Questions
- Why do I need to get an audit?
- What does an audit do for me?
- The answer should include to find all accessibility issues within scope and fix them
- Can you talk about your audit process / methodology?
- The answer should contain specifics
- (If they sell a scan) If you’re auditing my website, what do we need a scan for?
- Should we use the scan while we wait for the audit?
- The answer is no, you shouldn’t need the scan.
- And the only acceptable answer is that you will use the scan after remediation and re-audit takes place.
- Should we use the scan while we wait for the audit?
- Do you offer certification?
- What is your certification process?
- If we’re pushing changes to live soon, can we get certification?
- Certification should be contingent upon full conformance
- The answer should be not until we’ve ensured the scope is conformant
- What information does your audit report contain?
- Their answer should include most of the following
- Criterion
- Issue
- Location
- Recommendation(s)
- Screenshot / recording
- Their answer should include most of the following
- Who actually conducts the audit?
- Some companies farm out audits which may compromise quality
- If they farm out audits, ask how they ensure quality
- How do you determine the price of an audit?
- There should be a simple formula that they can recite to you
Summary
As a client in the market for an audit, it’s important to research different sellers and gauge who you feel will provide you the best quality audit. Remember, you are relying upon this report in multiple different ways including user access, compliance, and legal risk.
If your audit is of poor quality, it can easily come back to harm you.
We recommend you gravitate towards companies that are specific, detailed, and transparent.
We recommend you avoid companies that are general, vague, hesitant and promote overlays or any type of automated “solutions.”
If you need help with an audit, we’re happy to help. Accessible.org offers excellent quality audit services with quick turnarounds.
Just send us a message and we’ll respond ASAP – usually within a few hours. We deliver audits for most clients within 1-2 weeks after they reach out.
