I'm trying to create a Camera2 CameraCaptureSession that is capable of four outputs:
SurfaceView, up to 1080p)ImageReader, up to 8k photos)MediaRecorder/MediaCodec, up to 4k videos)ImageReader, up to 4k video frames)Unfortunately Camera2 does not support attaching all of those four outputs (Surfaces) at the same time, so I'm going to have to make a compromise.
The compromise that seemed most logical to me was to combine the two video capture pipelines into one, so that the Frame Processing output (#4, ImageReader) redirects the frames into the Video Capture output (#3, MediaRecorder).
How do I write the Images from the ImageReader:
val imageReader = ImageReader.newInstance(4000, 2256, ImageFormat.YUV_420_888, 3)
imageReader.setOnImageAvailableListener({ reader ->
val image = reader.acquireNextImage() ?: return@setOnImageAvailableListener
callback.onVideoFrameCaptured(image)
}, queue.handler)
val captureSession = device.createCaptureSession(.., imageReader.surface)
..into the Surface from the MediaRecorder?
val surface = MediaCodec.createPersistentInputSurface()
val recorder = MediaRecorder(context)
..
recorder.setInputSurface(surface)
I'm thinking that I might need an OpenGL pipeline here with a pass-through shader - but I don't know how I get from the ImageReader's Image to an OpenGL texture, so any help here would be appreciated.
What I tried: I looked into the HardwareBuffer APIs, specifically
auto clientBuffer = eglGetNativeClientBufferANDROID(hardwareBuffer);
...
auto image = eglCreateImageKHR(display,
EGL_NO_CONTEXT,
EGL_NATIVE_BUFFER_ANDROID,
clientBuffer,
attribs);
...
glEGLImageTargetTexture2DOES(GR_GL_TEXTURE_EXTERNAL, image);
And I think this might work, but it requires API Level 28. So I still need a solution for API Level 23 and above. The image.getPlanes() function returns me three ByteBuffers for the YUV data, not sure how I can create an OpenGL texture from there though..
I (kinda) figured it out! I found the ImageWriter API, which is exactly what I was about to rebuild from scratch - a pass-through pipeline from an Image to a Surface.
So now I stream Camera Frames into the ImageReader, call the Frame Processor with the Image, then pass the Image through to the MediaRecorder using the ImageWriter as a middle-man :)
val size = config.getOutputSizes(ImageFormat.PRIVATE).max()
// Video Recorder Surface. We need to stream Frames here if we are recording.
val surface = recordingSession.surface
val imageWriter = ImageWriter.newInstance(surface,
VIDEO_OUTPUT_BUFFER_SIZE)
// Image Reader Surface. We stream Frames here for Frame Processor or Recording.
val flags = HardwareBuffer.USAGE_GPU_SAMPLED_IMAGE or HardwareBuffer.USAGE_VIDEO_ENCODE
val imageReader = ImageReader.newInstance(size.width,
size.height,
ImageFormat.PRIVATE,
VIDEO_OUTPUT_BUFFER_SIZE,
flags)
imageReader.setOnImageAvailableListener({ reader ->
val image = reader.acquireNextImage() ?: return
image.timestamp = System.nanoTime()
// Call JS Frame Processor
frameProcessor?.call(image)
// If recording, write to Video File
if (isRecording) {
imageWriter.queueInputImage(image)
}
image.close()
}, CameraQueues.videoQueue)
// Camera only streams frames into one single Surface
cameraSession.configure(.., imageReader.surface)
My only problem now is that the resulting video recording sometimes has ~1 second long hickups after around ~3 seconds of recording, I have no idea why. Maybe I should use MediaCodec instead of MediaRecorder. Maybe I should use a different ImageFormat. Maybe I should investigate the resulting .mp4 file to see what's wrong. Maybe I should fix the Image timestamps. I don't know.
Also, logcat gets spammed with this:
2023-08-17 11:38:17.977 3780-3899 GraphicBufferSource com.mrousavy.camera.example W released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.021 3780-3899 GraphicBufferSource com.mrousavy.camera.example W released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.050 3780-3899 GraphicBufferSource com.mrousavy.camera.example W released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.082 3780-3899 GraphicBufferSource com.mrousavy.camera.example W released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.113 3780-3899 GraphicBufferSource com.mrousavy.camera.example W released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.146 3780-3899 GraphicBufferSource com.mrousavy.camera.example W released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
2023-08-17 11:38:18.179 3780-3899 GraphicBufferSource com.mrousavy.camera.example W released unpopulated slots: [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63]
But hey - it records a video. This is a good start.