I am trying to encode an mp4 video from an array of NalUnits where each unit is a frame represented by byte[] that is saved from an rtp packet from an rtsp stream. I am encoding 451 frames at a 30fps where the first frame data is the spp and pps frame combined. This is my current configuration:
Width: 720
Height: 480
Bitrate: 10_000_000
Fps: 30
Colorformat: COLOR_FormatYUV420Flexible
Key I Frame Interval: 1
Here is my method where I encode the configure the encoder:
public boolean createMp4Video(FrameQueue.Frame[] frames, File dir) {
try {
// Set up MediaMuxer
Date date = new Date();
File file = new File(dir, date + "-recording.mp4");
this.muxer = new MediaMuxer(file.getAbsolutePath(), MediaMuxer.OutputFormat.MUXER_OUTPUT_MPEG_4);
// Set up MediaCodec
mMediaFormat = MediaFormat.createVideoFormat(codecName, width, height);
mMediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, colorFormat);
mMediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, bitrate);
mMediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, framerate);
mMediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);
codec = MediaCodec.createEncoderByType(codecName);
codec.configure(mMediaFormat, null, null, MediaCodec.CONFIGURE_FLAG_ENCODE);
codec.start();
MediaCodec.BufferInfo videoBufferInfo = new MediaCodec.BufferInfo();
// Process frames
for (int i = 0; i < frames.length; i++) {
if (i == frames.length - 1) { // if last frame, add end of stream flag
muxFrame(frames[i].getData(), i, true);
} else {
muxFrame(frames[i].getData()), i, false);
}
// Stop and release resources
muxer.stop();
muxer.release();
codec.stop();
codec.release();
listener.onMp4VideoCreated();
return true;
} catch (IOException e) {
return false;
}
}
I loop through all the frames using a simple for loop and call the following function to method:
private void muxFrame(byte[] frameData, int encodeIndex, boolean isVideoEOS) {
long presentationTimeUs = 1000000 / framerate * encodeIndex
int inputBufferIndex = codec.dequeueInputBuffer(-1);
if (inputBufferIndex >= 0) {
ByteBuffer inputBuffer = codec.getInputBuffer(inputBufferIndex);
inputBuffer.put(frameData);
codec.queueInputBuffer(inputBufferIndex, 0, frameData.length, presentationTimeUs, 0);
}
// Create a MediaCodec.BufferInfo to hold the frame information
MediaCodec.BufferInfo muxerOutputBufferInfo = new MediaCodec.BufferInfo();
int outputBufferIndex = codec.dequeueOutputBuffer(muxerOutputBufferInfo, 1000);
switch (outputBufferIndex) {
case MediaCodec.INFO_OUTPUT_FORMAT_CHANGED:
MediaFormat newFormat = codec.getOutputFormat();
mVideoTrack = muxer.addTrack(newFormat);
muxer.start();
break;
case MediaCodec.INFO_TRY_AGAIN_LATER:
break;
default:
// Set the size, presentation time, and flags of the muxerOutputBufferInfo
muxerOutputBufferInfo.size = frameData.length;
muxerOutputBufferInfo.offset = 0;
muxerOutputBufferInfo.flags = isVideoEOS ? MediaCodec.BUFFER_FLAG_END_OF_STREAM : MediaCodec.BUFFER_FLAG_KEY_FRAME;
muxerOutputBufferInfo.presentationTimeUs = presentationTimeUs;
// Write the frame data to the muxer
muxer.writeSampleData(mVideoTrack, ByteBuffer.wrap(frameData), muxerOutputBufferInfo);
codec.releaseOutputBuffer(outputBufferIndex, false);
break;
}
}
I am able to save the video and play it but all the frames are corrupted and green as such https://dropmefiles.com/rn6I0
Here is my log when I am encoding:
I/OMXClient: IOmx service obtained
W/OMXUtils: do not know color format 0x7f000200 = 2130706944
W/OMXUtils: do not know color format 0x7f000789 = 2130708361
I/ACodec: [OMX.MTK.VIDEO.ENCODER.AVC] using color format 0x7f420888 in place of 0x7f420888
I/ACodec_vilte: set I frame rate
I/ACodec_vilte: set framerate
I/ACodec_vilte: setMTKParameters, width: 720
I/ACodec_vilte: setMTKParameters, height: 480
I/ACodec: setupAVCEncoderParameters with [profile: Baseline] [level: Level41]
I/ACodec: [OMX.MTK.VIDEO.ENCODER.AVC] cannot encode color aspects. Ignoring.
I/ACodec: [OMX.MTK.VIDEO.ENCODER.AVC] cannot encode HDR static metadata. Ignoring.
I/ACodec: setupVideoEncoder succeeded
I/ACodec_vilte: set I frame rate
I/ACodec_vilte: set framerate
I/ACodec_vilte: setMTKParameters, width: 720
I/ACodec_vilte: setMTKParameters, height: 480
I/general: [33] Configuring video encoder - 450 frames
I/general: [33] Video encoding in progress
D/MPEG4Writer: start+ 797
I/MPEG4Writer: limits: 4294967295/0 bytes/us, bit rate: -1 bps and the estimated moov size 3191 bytes
D/MPEG4Writer: start+ 2403
D/MPEG4Writer: start- 2481
D/MPEG4Writer: start- 964
I/MPEG4Writer: setStartTimestampUs: 166665
I/MPEG4Writer: Earliest track starting time: 166665
D/MPEG4Writer: Video mStartTimestampUs=166665us
D/MPEG4Writer: reset+ 1081
D/MPEG4Writer: Video track stopping. Stop source
D/MPEG4Writer: Video track source stopping
D/MPEG4Writer: Video track source stopped
I/MPEG4Writer: Received total/0-length (439/0) buffers and encoded 439 frames. - Video
D/MPEG4Writer: Video track stopped. Stop source
D/MPEG4Writer: Stopping writer thread
D/MPEG4Writer: 0 chunks are written in the last batch
D/MPEG4Writer: Writer thread stopped
I/MPEG4Writer: Ajust the moov start time from 166665 us -> 166665 us
I/MPEG4Writer: The mp4 file will not be streamable.
D/MPEG4Writer: reset- 1187
D/MPEG4Writer: reset+ 1081
D/MPEG4Writer: Video track stopping. Stop source
I/general: [33] MP4 video created successfully
Why does the video have corrupted frames? From a look at the shared bytes:
1) Every frame is a Key-frame
I wondered why your MP4's section for listing keyframes, the stss atom, had every single frame listed as a type "Key-frame" (why no P-frames and B-frames?) but then I see that your code has this command:
mMediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);
Since you are muxing (copy/pasting existing media bytes), that media has its own interval which should be written into the output MP4... A Muxing process does not need to be given a new keyframe interval.
solution:
Test a possible fix of disabling (or comment //) that line,
With same // disabling of the bitrate setting.
2) Your first frame is not a keyframe
The first video frame that you send to the muxer is not a keyframe.
It is marked as a key-frame in the MP4 metadata (at the stss atom) but it's actual bytes are of type P-frame. This means: After start-code 0, 0, 0, 1 your NAL header is found to be hex: 42, or decimal: 65). Be aware because a decimal 65 is not same as hex 65 and so it can be confusing, if you forget to convert, and think that decimal 65 means this NALU is a keyframe.
solution:
Make sure you send a NAL unit with decimals: 0, 0, 0, 1, 101 (or in hex: 00, 00, 00, 01, 65).
The 101 marks it as a keyframe NALU (header byte value is: 0x65 or as decimal value: 101).
3) You have wrong SPS and PPS
Once again the problem is a wrong SPS and PPS...
Inside the MP4, the SPS/PPS data goes into the avcC section (which holds the "AVC configuration", and AVC is just an alternate name for H264).
If I manually overwrite a part of the avcC section by copy/pasting over the bytes from the raw NALUs then your corrupt MP4 shows a perfect picture (not messy colours).
solution:
option A: Try to set a MediaFormat with the SPS and PPS included.
This means putting an entry for Codec-Specfic Data according to these keys:
Format CSD buffer #0 CSD buffer #1 CSD buffer #2
H.264 AVC SPS (Sequence Parameter Sets) PPS (Picture Parameter Sets) Not Used
option B: In a worst case scenario you might have to manually fix the bytes after the MP4 file is created by your Android code (eg: after the file creation code finishes, run another function which finds and overwrites some circa 25 bytes of existing SPS & PPS inside the newly created MP4). Hopefully not needed as a solution.
I cannot test Android code, please check if this example code creates a working MP4:
(1) Converting between Decimals into Hex (since bytes are written as hex)
//# check Byte as Decimal
int temp_int = (NALU_SPS[4] & 0xFF); //where NALU_SPS is an Array.
System.out.println("NALU Decimal: " + temp_int );
//# check Byte as Hex
String temp_str = Integer.toString( temp_int, 16 );
temp_str = (temp_str.length() < 2) ? ("0"+temp_str) : temp_str;
System.out.println("NALU Byte: 0x" + temp_str );
Use the above to convert your Array values when double-checking their byte value, since tutorials and H264 Specification will mention the byte values in a Hex format.
A byte example like 0xFF is hex FF and is decimal 255.
(2) Try this MediaFormat setup.
//# Set up MediaCodec
mMediaFormat = MediaFormat.createVideoFormat(codecName, width, height);
mMediaFormat.setInteger(MediaFormat.KEY_FRAME_RATE, framerate);
//# Your SPS/PPS in an Array. Can be: byte[] ...or Else: int[]
byte[] NALU_SPS = { 0x00, 0x00, 0x00, 0x01, 0x67, 0x42, 0x00, 0x29, 0xE2, 0x90, 0x16, 0x87, 0xB6, 0x06, 0xAC, 0x18, 0x04, 0x1B, 0x87, 0x89, 0x11, 0x50 };
byte[] NALU_PPS = { 0x00, 0x00, 0x00, 0x01, 0x68, 0xCE, 0x3C, 0x80 };
//# send SPS/PPS to the Muxer
mMediaFormat.setByteBuffer( "csd-0", ByteBuffer.wrap( NALU_SPS ) );
mMediaFormat.setByteBuffer( "csd-1", ByteBuffer.wrap( NALU_PPS ) );
//# This might not be needed (since Muxer could get from SPS/PPS)
mMediaFormat.setInteger(MediaFormat.KEY_COLOR_FORMAT, colorFormat);
//# test with these disabled
//mMediaFormat.setInteger(MediaFormat.KEY_BIT_RATE, bitrate);
//mMediaFormat.setInteger(MediaFormat.KEY_I_FRAME_INTERVAL, 1);