As with any fields of design, confirmation bias can stop us from seeing the reality. Eric Eggert points out how (in this field of design, too, one may add) only verifying solutions with real users – in this case people with disabilities – will lead to good results.
Accessibility removes actual barriers for people with disabilities. And you have to learn about the barriers and how they manifest for different disabilities. That will also highlight why the correct answer to accessibility questions often is “it depends”.
The links in his blog posts are a good starting point – “How People with Disabilities use the web” and the videos of assistive technology in use in particular –, and even better it is to test designs in their real use context.