When you’re looking to use someone else’s code in a web project, how do you decide between the alternatives?
You might look at some key numbers:
- When was it last updated?
- How many people are using it?
- How many open GitHub issues does it have?
- How many other people using it?
To answer these questions you can use the figures for ‘Weekly Downloads’, ‘Last publish’, ‘Issues’, and ‘Dependents’ (the number of other packages on npm that have this package as a dependency) from the npm package page.
You might also want to consider:
- How big is it?
- How will it affect my site’s performance?
For these, you can use Bundlephobia to check how a package might increase your bundle size and how long it will take to download.
What about “Is it accessible?”
This one is much harder to distil down to a few numbers. But that hasn’t stopped me from trying.
Measuring accessibility
Automated testing
Before building an automated tool, I thought about my manual process for evaluating a package’s accessibility. The first thing I look for is a link to an example of it running in the browser. From here, I’ll do some quick tests: if the package generates markup, I’ll check that it uses semantic HTML; for interactive elements, I’ll check that focus is managed correctly and that everything can be controlled using a keyboard.
If you wanted to automate this step — you could use an automated testing tool like axe, which would give you a figure for the number of accessibility issues. But automated tests only pick up a proportion of all issues and while this could work for some packages, there’s no standard way of linking to examples from an npm or GitHub page.
GitHub issues
After discarding that idea, I decided to look at GitHub issues. These act like a crowd-sourced database of bugs, feedback, and areas for improvement. We can filter for issues that contain certain keywords, such as ‘a11y’ (a common numeronym for ‘accessibility’) or ‘screenreader’ to get those issues we’re interested in. The fundamental assumption: if a maintainer prioritises these kinds of issues and responds to accessibility concerns in a timely manner, it’s likely that they have developed a piece of software with accessibility in mind.
To automate this, I started off by querying the npms.io API. This gives us lots of useful data for each package, including a link to the repository containing the package's source code. If this repository is on GitHub (and the vast majority are) then we can make use of GitHub’s REST API to search issues.
To determine how much a package’s maintainers prioritise accessibility-related issues, I’ve used two metrics:
- Score: percentage of issues closed
- Average age: median duration between issue open and closed dates
I’ve also found it useful to see a listing of open issues, showing the oldest first. This is handy for highlighting serious red flags — if there’s an issue called something like ‘Add accessibility’ that’s been open for six years, there’s probably something wrong.
Putting it all together
I built a prototype over a year ago which I’ve been using myself. It hasn’t completely changed my workflow but it automates a step that used to take a few minutes.
After deciding that other people might find it useful, I have rebuilt the site in Next.js and published it at IsItAccessible.dev. I’ve also added an autocomplete search, powered by the npms.io API, to make finding packages easier.
Of course, GitHub issues are by no means a perfect way of measuring accessibility:
- Less popular packages often lack the user base for accessibility issues to be detected.
- Some packages have the search keywords ‘accessibility’, ‘a11y’, ‘aria’, ‘screenreader’ in their name. This results in a large number of false positives.
- Lots of packages use monorepos, where multiple packages originate from the same source git repository. This makes working out which GitHub issue corresponds to which npm package difficult.
- Some maintainers can be trigger-happy closing issues that they don't think are issues. If a maintainer doesn't value accessibility at all, there's nothing to stop them closing accessibility-related issues without any further action.
Roadmap
- Reduce the number of false positives by filtering issues further based on their content.
- Give a rough guide to how an accessibility score or average age compares with other packages.
- Suggest alternative packages which may have fewer accessibility issues.
If you have any ideas on how to improve the site, please get in touch, either by creating a GitHub issue or on Twitter.