TL;DR
How to route remote participants voice in WebRTC call so they can be treated as internal sounds by Streamlabs or similar streaming apps?
Long story
My game is using Google WebRTC library for the main functionality - audio/video chat. I want to promote my players to stream the screen cast to YouTube and Twitch right from their Android phones. I tested a few apps and found that Streamlabs works great for my purpose.
My players can spectate a game and stream the app screen and sounds to Twitch/YouTube using Streamlabs app.
Streamlabs can pick audio from 2 channels - Internal & Microphone (you can adjust volume for each one). All ambient game sounds, like clicks, music etc. goes to Internal which is great. The WebRTC sounds from other participants (players) routed by my app (using Android AudioManager) to Speakerphone, then they picked up by phone's Microphone. So, this works, but this is not great.
The problem is that the streamer should always keep his mic on, even if he doesn't say anything because WebRTC speech from remote players goes this route (WebRTC -> Speakerphone -> Mic -> Streamlabs). And he also has always be in quite environment. Plus, the quality of sound obviously degraded.
Is there a way to route WebRTC sounds in a way, so they will be treated as Internal by Streamlabs?
Note that the problem is common for all WebRTC / VOIP call / conference call apps on Android. This can be easily reproduced with WhatsApp call, or Facebook Messenger call or any similar app. You can also use YouTube or Twitch app instead of Streamlabs and get similar results.
I tried different setting with Android AudioManager (using USAGE_MEDIA instead of USAGE_VOICE_COMMUNICATION, setting Audio Mode to NORMAL instead of MODE_IN_COMMUNICATION and more), but none of them works. I guess this is because the streaming app (Streamlabs etc) grabs Internal sounds even before it reaches AudioManager, probably on Android MediaPlayer level.
It would be great if I can route remote WebRTC participants voice in a way that it can be picked up by Streamlabs as Internal sounds, so it won't need to travel by air from Speakerphone to the Mic loosing quality and mixing with environment sounds.
I can alter Google WebRTC if needed (I build it myself anyway).
From the look of things, you’re trying to route WebRTC audio directly to the internal audio stream that Steamlabs can capture rather than having it go through the speaker-microphone path. You should try these approaches:
Investigate AudioTrack since WebRTC likely uses AudioTrack for audio output. You might be able to modify the audio sink:
// Example approach using AudioTrack
AudioTrack audioTrack = new AudioTrack.Builder()
.setAudioAttributes(new AudioAttributes.Builder()
.setUsage(AudioAttributes.USAGE_MEDIA) // Instead of VOICE_COMMUNICATION
.setContentType(AudioAttributes.CONTENT_TYPE_MUSIC)
.build())
.setAudioFormat(/* your format */)
.setTransferMode(AudioTrack.MODE_STREAM)
.build();
Implement a virtual audio device in WebRTC:
public class VirtualAudioDevice implements AudioDeviceModule {
private MediaProjection mediaProjection;
private AudioRecord audioRecord;
private AudioTrack audioTrack;
private boolean isCapturing;
private boolean isPlaying;
// Buffer for audio routing
private final LinkedBlockingQueue<ByteBuffer> audioBuffer = new LinkedBlockingQueue<>();
@Override
public int init() {
// Initialize virtual device
AudioFormat format = new AudioFormat.Builder()
.setSampleRate(48000)
.setEncoding(AudioFormat.ENCODING_PCM_16BIT)
.setChannelMask(AudioFormat.CHANNEL_OUT_STEREO)
.build();
// Create a virtual output device that other apps can capture
audioTrack = new AudioTrack.Builder()
.setAudioAttributes(new AudioAttributes.Builder()
.setUsage(AudioAttributes.USAGE_MEDIA)
.setContentType(AudioAttributes.CONTENT_TYPE_MUSIC)
.setFlags(AudioAttributes.FLAG_LOW_LATENCY)
.build())
.setAudioFormat(format)
.setTransferMode(AudioTrack.MODE_STREAM)
.setPerformanceMode(AudioTrack.PERFORMANCE_MODE_LOW_LATENCY)
.build();
return 0;
}
@Override
public boolean startPlayout() {
if (isPlaying) {
return true;
}
audioTrack.play();
isPlaying = true;
// Start audio routing thread
new Thread(() -> {
while (isPlaying) {
try {
ByteBuffer buffer = audioBuffer.take();
if (buffer != null) {
byte[] audio = new byte[buffer.remaining()];
buffer.get(audio);
audioTrack.write(audio, 0, audio.length);
}
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
break;
}
}
}).start();
return true;
}
@Override
public boolean stopPlayout() {
isPlaying = false;
if (audioTrack != null) {
audioTrack.stop();
audioTrack.flush();
}
return true;
}
// Method to receive audio data from WebRTC
public void onWebRTCAudioFrame(ByteBuffer audioData) {
if (isPlaying) {
try {
audioBuffer.put(audioData);
} catch (InterruptedException e) {
Thread.currentThread().interrupt();
}
}
}
// Implement other required methods...
@Override
public void release() {
stopPlayout();
if (audioTrack != null) {
audioTrack.release();
audioTrack = null;
}
}
}
Other alternatives are:
Modify your WebRTC build to use the virtual audio device implementation provided above. This creates a virtual audio output that should be captured as internal audio.
In your WebRTC initialization:
VirtualAudioDevice audioDevice = new VirtualAudioDevice();
PeerConnectionFactory.initialize(
PeerConnectionFactory.InitializationOptions.builder(context)
.setAudioDeviceModule(audioDevice)
.createInitializationOptions()
);
Route the WebRTC audio through this device:
// In your WebRTC audio track callback
@Override
public void onWebRTCAudioFrame(AudioTrack.Buffer buffer) {
audioDevice.onWebRTCAudioFrame(buffer.data);
}
Some key considerations to note: