Adding Closed Captions to an Amazon IVS Live Stream

There are two types of people in this world: those who watch TV shows with captions on, and those who are weird. All joking aside, the importance of closed captions for video can not be understated. Although crucial for the deaf and hard-of-hearing, captions are also important when audio is unavailable, or not clearly audible. Maybe you're watching a video in a public place and the audio is drowned out by ambient noise. Or, maybe the person speaking in the video is using a microphone that isn't the best quality, or speaks with an accent or dialect that is unfamiliar to the viewer. Captions are always a good thing. Unfortunately, captioning audio in a live stream is a really tricky problem to solve.

Before we dig into the problem of captioning live streams, let's talk about semantics a bit. Did you know that there is a difference between the terms closed caption and subtitle? The HTML spec describes subtitles as:

transcription or translation of the dialogue, suitable for when the sound is available but not understood (e.g. because the user does not understand the language of the media resource's audio track). Overlaid on the video.

The spec describes as captions like so:

Transcription or translation of the dialogue, sound effects, relevant musical cues, and other relevant audio information, suitable for when sound is unavailable or not clearly audible (e.g. because it is muted, drowned-out by ambient noise, or because the user is deaf). Overlaid on the video; labeled as appropriate for the hard-of-hearing.

This means that when we talk about "closed captions" for live videos, we're usually referring to subtitles since captions usually include descriptive information. Think about a scene in a TV show where an actor gets in their car to leave their home and says goodbye to their spouse. The caption for this scene might read "Goodbye, dear. [car engine starts]". We're not quite close to having AI systems describe contextually information like this for us, so we'll be limited to just adding pure "speech to text" subtitles to our live stream using the method below.

Adding Captions to Amazon IVS Live Streams

The solution that we will look at in this post will focus on broadcasting to an Amazon Interactive Video Service (Amazon IVS) live stream from OBS Studio. OBS doesn't offer native support for captioning, but there are a number of plugins available that can perform the necessary speech to text conversion and publish the captions to an RTMP stream in the CEA-708/EIA-608 format that is supported by Amazon IVS. For this demo, I've chosen to use the OBS-captions-plugin by ratwithacompiler (GitHub and ). To get started with this plugin, download it and install it. Once you've got it installed in OBS, click on Docks and make sure the Captions dock is enabled.

Next, click the 'gear' icon in the Captions dock to modify the settings.

Make sure that a Caption Source is selected, and modify the plugin configuration to suit your needs. For example, the default Caption Timeout for me was set to 15.0 seconds, but I found 5.0 seconds to be a better value for me.

Once you've saved your configuration and started a new live stream, the plugin will handle converting your speech to text and produce the required caption information to the live stream.

To playback the caption data with the Amazon IVS player, we can add an event listener to listen for the TextCue event (docs).

ivsPlayer.addEventListener(IVSPlayer.PlayerEventType.TEXT_CUE, (evt) => {  console.log(evt);}

The handler as configured above will log all of the incoming TextCue events to the console.

The text property of the TextCue event will contain the caption data.

With some HTML and CSS, we can render the caption data as an overlay on the <video> element. This implementation is highly dependent on your needs, but you should take into account auto-hiding the overlay after a specified period of no caption data.

Summary

In this post, we looked at how to use an OBS plugin to convert speech to text and publish that text as caption data on an Amazon IVS live stream.