ExoPlayer ignores embedded 608 captions in Dolby Vision streams

[Halmo]

Version

Media3 main branch

More version details

627b7a3

Devices that reproduce the issue

All

Devices that do not reproduce the issue

No response

Reproducible in the demo app?

Yes

Reproduction steps

1. Play a Dolby Vision DASH stream with embedded 608 captions in the Demo app.
2. Enable captions.

Expected result

Captions are enabled and are rendered.

Actual result

Captions are enabled but are not rendered.

This occurs because the Dolby Vision metadata in the video track causes the mime type to be video/dolby-vision instead of video/hevc or video/avc.

In this line:

media/libraries/extractor/src/main/java/androidx/media3/extractor/mp4/FragmentedMp4Extractor.java

Line 1609 in 49337d9

processSeiNalUnitPayload =

NalUnitUtil.isNalUnitSei always returns false because the function only checks for H264 or H25 mime types: https://github.com/androidx/media/blob/release/libraries/container/src/main/java/androidx/media3/container/NalUnitUtil.java#L342

If the mime type is Dolby Vision the Sei NAL units get ignored.

Secondly for H265 Dolby Vision this check also fails:

media/libraries/extractor/src/main/java/androidx/media3/extractor/mp4/FragmentedMp4Extractor.java

Line 1632 in 49337d9

nalBuffer.setPosition(MimeTypes.VIDEO_H265.equals(track.format.sampleMimeType) ? 1 : 0);

If Dolby Vision is or'd with the H265 format for the above 2 cases the issue is resolved. However, the fix obviously isn't that simple because H264 or H265 can in theory be Dolby Vision.

Media

Not applicable

Bug Report

• You will email the zip file produced by adb bugreport to android-media-github@google.com after filing this issue.

[icbaker]

Thanks for the report.

However, the fix obviously isn't that simple because H264 or H265 can in theory be Dolby Vision.

Can you elaborate a bit on this part? Are you saying that this would be safe/correct:

public static boolean isNalUnitSei(@Nullable String mimeType, byte nalUnitHeaderFirstByte) {
  return ((MimeTypes.VIDEO_H264.equals(mimeType) 
              || MimeTypes.VIDEO_DOLBY_VISION.equals(mimeType))
          && (nalUnitHeaderFirstByte & 0x1F) == H264_NAL_UNIT_TYPE_SEI)
      || ((MimeTypes.VIDEO_H265.equals(mimeType) 
              || MimeTypes.VIDEO_DOLBY_VISION.equals(mimeType))
          && ((nalUnitHeaderFirstByte & 0x7E) >> 1) == H265_NAL_UNIT_TYPE_PREFIX_SEI);
}

But this would be incorrect for H.264-in-dolby-vision (since it assumes dolby-vision implies H.265)?

// If the format is H.265/HEVC the NAL unit header has two bytes so skip one more byte.
nalBuffer.setPosition(
    MimeTypes.VIDEO_H265.equals(track.format.sampleMimeType)
        || MimeTypes.VIDEO_DOLBY_VISION.equals(track.format.sampleMimeType) 
    ? 1
    : 0);

If I've correctly understood your concern, we can probably modify NalUnitUtil.isNalUnitSei to return an int indicating whether it's an H.264 or H.265 SEI NAL unit, and use this to drive the ? 1 : 0 decision later. Does that sound like it would work?

Do you have a piece of sample media that demonstrates this problem? It would be really helpful if you could share it with us for testing. Please either upload it here or send it to android-media-github@google.com with the subject Issue #1820. Please also update this issue to indicate you’ve done this.

It would be even better if you could modify our existing dolby vision test asset to add CEA-608 captions and share the result with us, so we can write an automated test against it to avoid this breaking again in future.

[akhilesh-dubey]

Thanks @icbaker
@Halmo can provide more details on this and probably share the sample assets if required.
The main issue is that the check for Dolby Vision MIME type is not included, which leads to a failure in displaying CEA-608 captions when playing Dolby Vision (HEVC-HDR) tracks. It works fine with MIME type AVC (AVC-SDR) tracks. Although adding a following check for Dolby Vision in both places resolves the issue, Not sure if this is the correct approach for Dolby Vision support. Unfortunately, there's no quick workaround available to implement a fix right away.

  public static boolean isNalUnitSei(@Nullable String mimeType, byte nalUnitHeaderFirstByte) {
    return (MimeTypes.VIDEO_H264.equals(mimeType)
            && (nalUnitHeaderFirstByte & 0x1F) == H264_NAL_UNIT_TYPE_SEI)
        || (MimeTypes.VIDEO_H265.equals(mimeType)
            && ((nalUnitHeaderFirstByte & 0x7E) >> 1) == H265_NAL_UNIT_TYPE_PREFIX_SEI)
        || (MimeTypes.VIDEO_DOLBY_VISION.equals(mimeType)
            && ((nalUnitHeaderFirstByte & 0x7E) >> 1) == H265_NAL_UNIT_TYPE_PREFIX_SEI);
  }

and

            nalBuffer.setPosition(MimeTypes.VIDEO_H265.equals(track.format.sampleMimeType)
                || MimeTypes.VIDEO_DOLBY_VISION.equals(track.format.sampleMimeType) ? 1 : 0);

@icbaker, to answer your question, yes if we can identify the SEI NAL unit for Dolby Vision stream, and based on that can decide to skip the header bytes, it will resolve the issue. Thanks a lot.

[icbaker]

@akhilesh-dubey

The main issue is that the check for Dolby Vision MIME type is not included, which leads to a failure in displaying CEA-608 captions when playing Dolby Vision (HEVC-HDR) tracks. It works fine with MIME type AVC (AVC-SDR) tracks.

Does the bit I've bolded mean that when you play H.264-in-dolby-vision with CEA-608 subtitles, they work? If so, how does that work, because surely the same MIME type checks you've flagged here should fail with video/dolby-vision MIME type.

[akhilesh-dubey]

We haven't tried playing H.264(with CEA-608 subtitles) with Dolby Vision yet, but my guess is that it likely won’t work.
However, we did test H.264(with CEA-608 subtitles) without Dolby Vision, and that worked fine.
I’m not certain if we have an H.264(with CEA-608 subtitles) stream with Dolby Vision, but if I can find one, I’ll definitely test it.

[Halmo]

@icbaker the problem with

public static boolean isNalUnitSei(@Nullable String mimeType, byte nalUnitHeaderFirstByte) {
  return ((MimeTypes.VIDEO_H264.equals(mimeType) 
              || MimeTypes.VIDEO_DOLBY_VISION.equals(mimeType))
          && (nalUnitHeaderFirstByte & 0x1F) == H264_NAL_UNIT_TYPE_SEI)
      || ((MimeTypes.VIDEO_H265.equals(mimeType) 
              || MimeTypes.VIDEO_DOLBY_VISION.equals(mimeType))
          && ((nalUnitHeaderFirstByte & 0x7E) >> 1) == H265_NAL_UNIT_TYPE_PREFIX_SEI);
}

Is that it would allow H265 Dolby Vision content to check (nalUnitHeaderFirstByte & 0x1F) == H264_NAL_UNIT_TYPE_SEI) and presumably that's not valid for H265. I'm not familiar enough with low level H265 spec details to know if that could give a false positive.

Similarly you could also have the inverse where you are playing H264 Dolby Vision and it fails the first check but then checks ((nalUnitHeaderFirstByte & 0x7E) >> 1) == H265_NAL_UNIT_TYPE_PREFIX_SEI) not sure if there could be false positives there.

To be honest I think ExoPlayer really needs to append Dolby Vision to the H264/H265 mime types vs having it as its own mime type since to my knowledge there's no dolby vision only codec it's always built on top of another existing codec. But this of course is a large change that touches a lot of pieces in ExoPlayer. We have other workarounds in our code due to the mime type being reported as Dolby Vision and the original codec being dropped from the mime type.

As for content I'll reach out to our internal teams to see if we can help in adding the 608 embedded to your existing test asset.

[Halmo]

@icbaker I've sent a sample asset to android-media-github@google.com. It doesn't seem to want to play in ExoPlayer for some reason (didn't investigate too much) but it does execute the code path in FragmentedMp4Extractor.

[icbaker]

Thanks for the sample, and for flagging the concerns with false-positives if we just inspect the NAL byte.

I've gone for a different approach using the existing MimeTypes.containsCodecsCorrespondingToMimeType() method, which I think should resolve this - the commit with this change is linked above.