The UX design case of closed captions for everyone

2019-04-29 08:15 Added a sentence of caution to the final paragraph: inclusion must not come at the cost of exclusion

Are video subtitles really chiefly for users who cannot hear or lack an audio device? A recent Twitter thread on “closed captions for the hearing” triggered a brief qualitative exploration and thought experiment – there may well be a growing group of users being forgotten in the design of closed captions.

Most commonly perceived as an auxiliary means for the hearing impaired, video subtitles, a.k.a. closed captions (CC), have only recently started to be  widely considered as an affordance for users in situations with no audio available/possible (think mobile devices in public settings, libraries, shared office spaces); the latter to the extend that contemporary “social media marketing guidelines” strongly recommend subtitling video clips uploaded to Facebook, Twitter et al.

So: subtitles are for those who cannot hear, or with muted devices?

Who else uses closed captions?

I’m personally a great fan of closed captions, for various reasons unrelated to either of the above, and have often noticed certain limitations in their design. Hence, the user researcher inside me just did a somersault as I randomly encountered a Twitter thread following Jason Kottke asking his 247.000 followers:

After seeing several photos my (English-speaking, non-deaf) friends have taken of their TV screens over the past week, I’m realizing that many of you watch TV with closed captions (or subtitles) on?! Is this a thing? And if so, why?

The 150+ replies (I guess this qualifies as a reasonable sample for a qualitative analysis of sorts?) are a wonderful example of “accessibility features” benefiting everybody (I wrote about another instance recently). The reasons why people watch TV with closed captions on, despite having good hearing abilities and not being constrained by having to watch muted video, are manifold and go far beyond those two most commonly anticipated use cases.

Close-up image of a video with subtitles
Image caption: Closed captions are used by people with good hearing and audio playback turned on. An overseen use case?

Even applying a rather shallow, ex-tempore categorisation exercise based on the replies on Twitter, I end up with an impressive list to start with:

  • Permanent difficulties with audio content
    • audio processing disorders
    • short attention span (incl., but not limited to clinical conditions)
    • hard of hearing, irrespective of age
  • Temporary impairments of hearing or perception
    • watching under the influence of alcohol
    • noise from eating chips while watching
  • Environmental/contextual factors
    • environment noise from others in the room (or a snoring dog)
    • distractions and multitasking (working out, child care, web browsing, working, phone calls)
  • Reasons related to the media itself
    • bad audio levels of voice vs. music
  • Enabler for improved understanding
    • easier to follow dialogue
    • annoyance with missing dialogue
    • avoidance of misinterpretations
    • better appreciation of dialogue
  • Better access to details
    • able to take note of titles of songs played
    • ability to understand song lyrics
    • re-watching to catch missed details
  • Language-related reasons
    • strong accents
    • fast talking, mumbling
    • unable to understand foreign language
    • insecurity with non-native language
  • Educational goals, learning and understanding
    • language learning
    • literacy development for children
    • seeing the spelling of unknown words/names
    • easier memorability of content read (retainability)
  • Social reasons
    • courtesy to others, either in need for silence or with a need/preference for subtitles
    • presence of pets or sleeping children
    • avoiding social conflict over sound level or distractions (“CC = family peace”)
  • Media habits
    • ability to share screen photos with text online
  • Personal preferences
    • preference for reading
    • acquired habit
  • Limitations of technology skills
    • lack of knowledge of how to turn them off

An attempt at designerly analysis

The reasons range from common sense to surprising, such as the examples of closed captions used to avoid family conflict or the two respondents explicitly mentioning “eating chips” as a source of disturbing noise. Motivations mentioned repeatedly refer to learning and/or understanding, but also such apparently banal reasons like not knowing how to turn them off (a usability issue?). Most importantly, though, it becomes apparent that using CC is more often than not related to choice/preference, rather than to impairment or restraints from using audio.

At the same time, it becomes very clear that not everybody likes them, especially when forced to watch with subtitles by another person. The desire/need of some may negatively affect the experience of others present. A repeat complaint that, particularly with comedy, CC can kill the jokes may also hint at the fact that subtitles and their timing could perhaps be improved by considering them as more than an accessibility aid for those who would not hear the audio? (It appears as if the scenario of audio and CC consumed simultaneously is not something considered when subtitles are created and implemented; are we looking at another case for “exclusive design”?)

And while perceived as distracting when new – this was the starting point of Kottke’s Tweet – many of the comments share the view that it becomes less obtrusive over time; people from countries where TV is not dubbed in particular are so used to it they barely notice it (“becomes second nature”). Yet, there are even such interesting behaviours like people skipping back to re-read a dialogue they only listened to at first, as well as that of skipping back to be able to pay better attention to the picture at second view (e.g. details of expression) after reading the subtitles initially.

Last but not least, it is interesting how people may even feel shame over using CC. Only a conversation like the cited Twitter thread may help them realise that it is much more common than they thought. And most importantly that it has nothing to do with a perceived stigmatisation of being “hard of hearing”.

CC as part of video content design

The phenomenon is obviously not new. Some articles on the topic suggest that it is a generational habit of generation Z (though Kottke’s little survey proves the contrary), or even sees it as paranoid and obsessive-compulsive behaviour of “postmodern completists” as facilitated by new technological possibilities. Research on the benefits of CC for language learning, on the other hand, reaches back several decades.

No matter what – the phenomenon in itself is interesting enough to make this a theme for deeper consideration in any design project that contains video material. Because, after all, one thing is for sure: closed captions are not for those with hearing impairments or with muted devices alone – and to deliver great UX, these users should be considered as well. At the same time, it is designers’ responsibility to ensure that extending the inclusiveness of video content through CC does not come at the cost of breaking the established science of providing access to those who need CC the most.

Copy of this article syndicated on Twitter (2 likes, 4 retweets)
Responding with a post on your own blog? Submit the URL as webmention (?)
  • TFW when you realise you are not alone.

    2019-04-27

  • Comment by Mark:

    Your list, and sampling, leaves out one important and constantly growing demographic - English as a second language speakers. I was introduced to always on captions by my Turkish wife and in-laws. They all have them on because reading straight English helps them understand words that may be new to them or said in an accent they don't understand. Southern English is especially difficult for them to parse.

    2019-04-28

  • Comment by Sebastian Greger:

    Mark, thank you for your comment. The demographic you mention does appear in the list, yet I admit only implicitly (under "Language-related reasons" and "Educational goals, learning and understanding"). I full-heartedly agree this cannot be stressed enough: making language accessible to non-native speakers is one of the most significant accessibility considerations addressed by CC.

    2019-04-28

  • Comment by Richard James:

    The comment below is highly problematic:

    “Because, after all, one thing is for sure: closed captions are not for those with hearing impairments or with muted devices alone – and to deliver great UX, these users should be considered as well.”

    The reason for this is simple: the closed captioning technology was optimized for hearing impaired/deaf people. It is an accessibility device, first and foremost. Any changes to this tech that degrades the accessibility for the deaf user would be the disability equivalent of cultural appropriation but with highly negative impact on the original users. It is kind of like saying that we should change/modify Braille to accommodate non-blind people, without considering how it would affect the blind.

    2019-04-29

  • Comment by Sebastian Greger:

    Richard, thank you for your reply. I understand your concern and appreciate that you took the time to leave a comment.

    I believe that addressing CC usability beyond those hearing impaired or deaf does not have to automatically imply a degradation of accessibility for those groups. Inclusive design, for me, means to maximise access; improving UX for one group at the expense of others is never my intention. I do, however, consider it important to acknowledge that there are more people relying on CC than is often thought and that, in an inclusive design process, these groups deserve consideration.

    As in every design process touching established principles, there is an inherent risk of breaking things. To avoid the quoted statement to be misread as an invitation to unreflectedly "disrupt CC", I added a sentence about designers' responsibility to my original paragraph above. Thank you again for highlighting this.

    2019-04-29

  • Comment by an anonymous reader:

    When I travel in the USA, I find that many hotel room TVs position the CC in the middle of the screen, which is just ridiculous.

    CC options for positioning and foreground/background colors are really useful. An option that I'd like to see is disabling CC temporarily while looking at a guide or channel listings.

    2019-04-30