Is there a way to associate a Renderer with a MediaItem?

[godkingwashington]

Hey everyone, this is a more conceptual question about exoplayer. I'm trying to understand where I can find the relationship between a media file (e.g. a mp3 file / MediaItem) and the renderer within the code itself.

My use case has me downloading several mp3 files from my server, each with specific file names, and loading them into several renderers, one for each file, for playback. The goal is to be able to access a renderer for a given file--for example: sending a MSG_SET_VOLUME message to a renderer given the filename. I can create the renderers just fine using a custom RenderersFactory, but beyond this I'm having some trouble tracing the relationship between a renderer and the source of its data.

I've looked at TrackSelection and LoadControl, but each have TrackGroups and TrackSelections within their purview, not a direct relationship to a MediaItem. In other words, I don't believe I can tell which file the TrackSelection or TrackGroup is associated with. I've also looked at setting custom metadata for the MediaItems I'm giving my player via setting MediaMetadata on the MediaItem.Builder in the hopes of retrieving it somewhere like LoadControl, but I cannot seem to access the metadata I set, only the ID3 metadata.

For example, here's how I tried to see the metadata I set on a singular MediaItem. I attempted to put the filename as the track title:

//In our activity when we are setting up the player
private String PrepareDebugFiles(){/* ...Ensure our debug files are properly downloaded and present on the device for testing */}

MediaMetadata metadata = new MediaMetadata.Builder().setTitle("MyCustomFileName").build();
MediaItem myFile = new MediaItem.Builder().setMediaMetadata(metadata).setUri(PrepareDebugFiles()).build();

ExoPlayer player = new ExoPlayer.Builder(context)
                    .setLoadControl(myCustomLoadControl)
                    .build();
player.setMediaItem(myFile);
player.prepare();
player.play();



//In a custom load control object
public void onTracksSelected(Timeline timeline, MediaSource.MediaPeriodId mediaPeriodId, Renderer[] renderers, TrackGroupArray trackGroups, ExoTrackSelection[] trackSelections){

	for(int i = 0; i < trackGroups.length; i++) {
		Metadata metadata = trackGroups.get(i).getFormat(0).metadata;
		if (metadata != null) {
                        //Returns some values but not "MyCustomFileName"
			Log.d("CustomLoadControl", "Metadata " + metadata.get(0));
		}
	}

	for(int i = 0; i < trackSelections.length; i++){
		if (trackSelections[i] != null) {
			Metadata metadata = trackSelections[i].getSelectedFormat().metadata;
			if (metadata != null) {
                                //Returns some values but not "MyCustomFileName"
				Log.d("CustomLoadControl", "Metadata " + metadata.get(0));
			}
		}
	}
}

Based on the diagram in the glossary, it seems like everything has a one-way relationship when loading from a file and I cannot really walk back up the chain easily.

Any help pointing me in the correct direction would really be appreciated. I'm hoping I missed something obvious.

Thanks.

[icbaker]

Stepping back slightly: Would your use-case be served with effects or custom audio processors, rather than needing a whole separate renderer instance for each item (presumably these items are in a playlist?)? I only suggest this because the example you give is about adjusting the volume, which should be possible with effects/processors. Manually creating a separate renderer for each media item sounds a little unusual.

---

To answer what you actually asked: I think @christosts was discussing recently how to access the currently playing MediaItem from the renderer part of the pipeline - so he might have some thoughts.

[godkingwashington]

It's funny you mention processors because that would be the next step beyond this. I need to control each piece of media individually before mixing them down into a single output buffer and sending it off to the audio track for consumption (which, from what I understand, can be achieved via custom audio sinks and processors). The example was just to show that I want to interact with the renderer in some way given the information I receive from the server (the filename).

To be extra clear, I could potentially need to do that following:
Given five mp3 files named FileOne, FileTwo, FileThree, FileFour, FileFive

1. Reduce the volume of FileOne by 50%, FileThree by 33%, and FileFour by 90%, and omit FileTwo's buffer entirely (or reduce volume by 100%)
2. Mix the output buffers of all files into a single result buffer
3. Output this result buffer via AudioTrack

[christosts]

To answer the very specific question asked:

find the relationship between a media file (e.g. a mp3 file / MediaItem) and the renderer

As of version 1.2.0, there are two changes in Renderer:

• We added Renderer.setTimeline(). ExoPlayer updates the timeline on the renderers every time there is a timeline change, so the timeline is always updated.
• If you're re-using existing renderers classes, or you implemented your own renderers but still extend BaseRenderer, then the timeline instance is available via the protected method BaseRenderer.getTimeline().
• We added a MediaPeriodId argument in Renderer.replaceStream().
• If you're re-using existing renderers classes, or you implemented your own renderers but still extend BaseRenderer, then the MediaPeriodId is available in BaseRenderer.onStreamChanged().

With a Timeline and a MediaPeriodId, you can get the MediaItem currently played, eg with

Timeline.Period period =  getTimeline().getPeriodByUid(mediaPeriodId, new Timeline.Period());    
MediaItem mediaItem = getTimeline().getWindow(period.windowIndex, new Timeline.Window()).mediaItem;

That said, on your overall plans for mixing mp3 files together:

• Do you plan to do it using one ExoPlayer instance?
• Is the one player instance playing the 4 mp3 files in parallel or are you playing them one after the other?
• Do you need to play them to the use in real-time as the mp3 files are being loaded, or it is OK if you pre-process the entire operation, store the result in a separate file and then play out the new file?

The ability to play multiple videos and audios together, apply visual and audio effects, and preview the result in real-time is something we are working on. But for your immediate needs, you might be able to use Transformer to first mix the audios and produce an intermediate file.

cc-ing @Samrobbo for audio processing related information

[godkingwashington]

Oh interesting. I tried to find the MediaItem relationship on the Timeline and MediaPeriodId in the LoadControl object I set up as they are exposed there but had no luck since onTracksSelected only exposes a Renderer array, not a BaseRenderer. I was hoping to set it in LoadControl as I only really need to set this relationship once during initialization. The tracks will never change, but the entire set of files will as the app is used.

Do you plan to do it using one ExoPlayer instance?

Yes, that was my intention. I tried using several ExoPlayer instances but found that it was too out of sync with one another to get the results I needed.

Is the one player instance playing the 4 mp3 files in parallel or are you playing them one after the other?

The goal is to get up to 16 tracks mixed down and played back simultaneously. The average may be less, but that's the most extreme case I'm preparing for. I don't really know what the upper-bound is for simultaneous playback using ExoPlayer, but I rolled my own mixing solution using AudioTrack, MediaCodec, and MediaExtractor and got 12 to work fairly well, only bottlenecked by I/O. I saw that ExoPlayer has a more elaborate solution for loading files so I wanted to get a prototype working using something more established and tested.

Do you need to play them to the use in real-time as the mp3 files are being loaded, or it is OK if you pre-process the entire operation, store the result in a separate file and then play out the new file?

I do need the operation to happen in real-time as there will be some manipulation happening during mixing and playback. I will also be working with multiple sets of files coming from my server. As a first step, I intend to tear down the ExoPlayer instance and set it up for each new set of files loaded to the device for simplicity's sake. So, one set may have 5 files while another has 10 and yet another has 16, each being mapped to a renderer for playback and real-time manipulation.

[christosts]

The goal is to get up to 16 tracks mixed down and played back simultaneously.

How are the media items given to player? As a List with setMediaItems()? If multiple media items are played simultaneously, I'm not sure the Timeline/MediaPeriod abstractions will give you what you need - only one MediaItem plays at the time.

Aside:

• If you're using one ExoPlayer, it will be interesting to see if the single-thread that feeds and drains MediaCodec instances will be able to iterate all 16 codecs fast enough.
• On the other hand, 16 individual ExoPlayer instances means 16 threads, so that may also not scale.

[godkingwashington]

How are the media items given to player? As a List with setMediaItems()? If multiple media items are played simultaneously, I'm not sure the Timeline/MediaPeriod abstractions will give you what you need - only one MediaItem plays at the time.

That could be an issue. Would a MergingMediaSource solve this? I'm assuming if this is the case that the MergingMediaSource would have tracks rather than individual MediaItems and I'd need to create an association a different way between the Renderers and the files for which they are responsible.

[tonihei]

You'd need to do the association on track level when using MergingMediaSource. We already mark the TrackGroup instances with unique id prefixes corresponding to the index of the source. But unfortunately, we don't pass down the group to the Renderer. Would it be helpful to also prefix Format.id (which is available in the renderers)?

[godkingwashington]

I tried to experiment with your suggestion, but it seems I've hit a bit of a snag. I've created a MergingMediaSource and loaded it into the player using the setMediaSource function but it seems like trying to get the format of the selected tracks from both the TrackGroup and TrackSelection later down the pipeline returns null. For example, in my CustomLoadControl class where I override onTracksSelected:

//CustomLoadControl
public void onTracksSelected(Timeline timeline, MediaSource.MediaPeriodId mediaPeriodId, Renderer[] renderers, TrackGroupArray trackGroups, ExoTrackSelection[] trackSelections){

        for(int i = 0; i < trackGroups.length; i++) {
            Log.d("CustomLoadControl", "Format ID?  " + trackGroups.get(i).getFormat(0).id); //null
        }

        for(int i = 0; i < trackSelections.length; i++){
            if (trackSelections[i] != null) {
                Log.d("CustomLoadControl", "Format ID? " + trackSelections[i].getSelectedFormat().id); //null
            }
        }

I'm guessing that I need to override the TrackSelector and create a custom implementation rather than using the DefaultTrackSelector? Otherwise, I'm not sure why I do not get the Format.id at this point from the groups. I think prepending the id to my map of the filename to renderer reference could work, but I'd need to really see how the pieces fit before I can say for sure.

[tonihei]

This was just a suggestion that isn't implemented yet :)

You probably find that all trackGroups.get(i).id start with "0:", "1:" etc depending on which source they are from. The suggestion was to use the same pattern for the Format.id fields too, so that you can identify which source a Format is from even from within a Renderer for example.