Dubbing and Audio description Profiles of TTML2

An Audio Recording is an Audio object that references an audio resource. It has the following properties:

• One or more alternative Sources, each of which is either 1) a link to an external audio resource or 2) an embedded audio recording;
• For each Source, one mandatory Type that specifies the type ([MIME-TYPES]) of the audio resource, for example audio/basic;
• An optional Begin property and an optional End and an optional Duration property that together define the Audio Recording's time interval in the programme timeline, in relation to the parent element's time interval;
• An optional In Time and an optional Out Time property that together define a temporal subsection of the audio resource;

  The default In Time is the beginning of the audio resource.

  The default Out Time is the end of the audio resource.

  If the temporal subsection of the audio resource is longer than the duration of the Audio Recording's time interval, then playback MUST be truncated to end when the Audio Recording's time interval ends.

  Note

  If the temporal subsection of the audio resource is shorter than the duration of the Audio Recording's time interval, then the audio resource plays once.

• Zero or more Mixing Instructions that modify the playback characteristics of the Audio Recording.

When a list of Sources is provided, a presentation processor MUST play no more than one of the Sources for each Audio Recording.

Implementations can use the Type, and if present, any relevant additional formatting information, to decide which Source to play. For example, given two Sources, one being a WAV file, and the other an MP3, an implementation that can play only one of those formats, or is configured to have a preference for one or the other, would select the playable or preferred version.

An Audio Recording is represented in a DAPT Document by an element child of a

or element corresponding to the Text to which it applies. The following constraints apply to the element:

• The begin, end and dur attributes represent respectively the Begin, End and Duration properties;
• The clipBegin and clipEnd attributes represent respectively the In Time and Out Time properties, as illustrated by Example 5;
• For each Source, if it is a link to an external audio resource, the Source and Type properties are represented by exactly one of:
  1. A src attribute that is not a fragment identifier, and a type attribute respectively;

     This mechanism cannot be used if there is more than one Source.

     Example 28

  2. A child element with a src attribute that is not a fragment identifier and a type attribute respectively;
     Example 29

  A src attribute that is not a fragment identifier is a URL that references an external audio resource, i.e. one that is not embedded within the DAPT Script. No validation that the resource can be located is specified in DAPT.

  Editor's note

  Do we need both mechanisms here? It's not clear what semantic advantage the child element carries in this case. Consider marking use of that child element as "at risk"?

  Issue 113: Support both `@src` and `` child of `

            While working on the specification for adding audio recordings I reminded myself of the various ways in which an audio recording can be embedded and referenced, of which there are at least 5 in total. Requirement R15 of [DAPT](https://www.w3.org/TR/dapt-reqs/#requirements) is clear that both referenced and embedded options need to be available, but should we be syntactically restricting the options for each? Will raise as separate issues.
  

  Originally posted by @nigelmegitt in #105 (comment)

  The following two options exist in TTML2 for referencing external audio resources:

  1. src attribute in element.

  2. element child of element.

  This second option has an additional possibility of specifying a format attribute in case type is inadequate. It also permits multiple child elements, and we specify that in this case the implementation must choose no more than one.

  [Edited 2023-03-29 to account for the "play no more than one" constraint added after the issue was opened]

  Issue 218: At-risk: support for `src` attribute in `

  Possible resolution to #113.

  Issue 219: At-risk: support for `` element child of `

  Possible resolution to #113.

• If the Source is an embedded audio resource, the Source and Type properties are represented together by exactly one of:
  1. A src attribute that is a fragment identifier that references either an element or a element, where the referenced element is a child of /tt/head/resources and specifies a type attribute and the xml:id attribute used to reference it;

     This mechanism cannot be used if there is more than one Source.

     Example 30

     <tt>
       <head>
         <resources>
           <data type="audio/wave" xml:id="audio1">
             [base64-encoded WAV audio resource]
           data>
         resources>
       head>
       <body>
         ..
         <audio src="#audio1"/>
         ..
       body>
     tt>

  2. A child element with a src attribute that is a fragment identifier that references either an element or a element, where the referenced element is a child of /tt/head/resources and specifies a type attribute and the xml:id attribute used to reference it;
     Example 31

     <tt>
       <head>
         <resources>
           <data type="audio/wave" xml:id="audio1wav">
             [base64-encoded WAV audio resource]
           data>
           <data type="audio/mpeg" xml:id="audio1mp3">
             [base64-encoded MP3 audio resource]
           data>
         resources>
       head>
       <body>
         ..
         <audio>
           <source src="#audio1wav"/>
           <source src="#audio1mp3"/>
         audio>
         ..
       body>
     tt>

  3. A child element with a element child that specifies a type attribute and contains the audio recording data.
     Example 32

     <audio>
       <source>
         <data type="audio/wave">
             [base64-encoded WAV audio resource]
         data>
       source>
     audio>

  In each of the cases above the type attribute represents the Type property.

  A src attribute that is a fragment identifier is a pointer to an audio resource that is embedded within the DAPT Script

  If elements are defined, each one MUST contain either #PCDATA or child elements and MUST NOT contain any child elements.

  and elements MAY contain a format attribute whose value implementations MAY use in addition to the type attribute value when selecting an appropriate audio resource.

  Editor's note

  Do we need all 3 mechanisms here? Do we need any? There may be a use case for embedding audio data, since it makes the single document a portable (though large) entity that can be exchanged and transferred with no concern for missing resources, and no need for e.g. manifest files. If we do not need to support referenced embedded audio then only the last option is needed, and is probably the simplest to implement. One case for referenced embedded audio is that it more easily allows reuse of the same audio in different document locations, though that seems like an unlikely requirement in this use case. Another is that it means that all embedded audio is in an easily located part of the document in tt/head/resources, which potentially could carry an implementation benefit? Consider marking the embedded data features as "at risk"?

  Issue 114: Support both `@src` and `` child of `

            While working on the specification for adding audio recordings I reminded myself of the various ways in which an audio recording can be embedded and referenced, of which there are at least 5 in total. Requirement R15 of [DAPT](https://www.w3.org/TR/dapt-reqs/#requirements) is clear that both referenced and embedded options need to be available, but should we be syntactically restricting the options for each? Will raise as separate issues.
  

  Originally posted by @nigelmegitt in #105 (comment)

  Given some embedded audio resources:

  
    
      
      
        [base64 encoded audio data]
      
    
  

  The following two options exist in TTML2 for referencing embedded audio resources:

  1. src attribute in element referencing embedded or :

  2. element child of element.

  This second option has an additional possibility of specifying a format attribute in case type is inadequate. It also permits multiple child elements, though it is unclear what the semantic is intended to be if multiple resources are specified - presumably, the implementation gets to choose one somehow.

  Issue 115: Support both referenced and inline embedded audio recordings? questionPR-must-have

            While working on the specification for adding audio recordings I reminded myself of the various ways in which an audio recording can be embedded and referenced, of which there are at least 5 in total. Requirement R15 of [DAPT](https://www.w3.org/TR/dapt-reqs/#requirements) is clear that both referenced and embedded options need to be available, but should we be syntactically restricting the options for each? Will raise as separate issues.
  

  Originally posted by @nigelmegitt in #105 (comment)

  If we are going to support embedded audio resources, they can either be defined in /tt/head/resources and then referenced, or the data can be included inline.

  Do we need both options?

  Example of embedded:

  
    
      
      
        [base64 encoded audio data]
      
    
  

  This would then be referenced in the body content using something like (see also #114):

  Example of inline:

  Issue 220: At-risk: support for `src` attribute of `

  Possible resolution to #114 and #115.

  The link to #115 is that this implies the existence of some referenceable embedded audio resource too, which one of the options described in #115.

  Issue 221: At-risk: support for `` child of `

  Possible resolution to #114 and #115.

  The link to #115 is that this implies the existence of some referenceable embedded audio resource too, which one of the options described in #115.

  Issue 222: At-risk: support for inline audio resources PR-must-have

  Possible resolution to #115.

  Issue 116: Add non-inlined embedded audio resources to the Data Model? questionPR-must-have

  See also #115 - if we are going to support non-inline embedded audio resources, should we make an object for them and add it into the Data Model?

  Issue 117: Embedded data: Do we need to support all the permitted encodings? What about length? questionPR-must-have

  In TTML2's element, an encoding can be specified, being one of:

  • base16
  • base32
  • base32hex
  • base64
  • base64url

  Do we need to require processor support for all of them, or will the default base64 be adequate?

  Also, it is possible to specify a length attribute that provides some feasibility of error checking, since the decoded data must be the specified length in bytes. Is requiring support for this a net benefit? Would it be used?

  Issue 223: At-risk: each of the potential values of `encoding` in `` PR-must-have

  Possible resolution to #117.

  Issue 224: At-risk: support for the `length` attribute on `` PR-must-have

  Possible resolution to #117.

• Mixing Instructions MAY be applied as specified in their TTML representation;
• The computed value of the xml:lang attribute MUST be identical to the computed value of the xml:lang attribute of the parent element and any child elements and any referenced embedded elements.

A Synthesized Audio is an Audio object that represents a machine generated audio rendering of the parent Text content. It has the following properties:

• A mandatory Rate that specifies the rate of speech, being normal, fast or slow;
• An optional Pitch that allows adjustment of the pitch of the speech.

A Synthesized Audio is represented in a DAPT Document by the application of a tta:speak style attribute on the element representing the Text object to be spoken, where the computed value of the attribute is normal, fast or slow. This attribute also represents the Rate Property.

The tta:pitch style attribute represents the Pitch property.

The TTML representation of a Synthesized Audio is illustrated by Example 6.