androidjavacviplimagecamera2

Android JavaCV + Camera2

trying to record video from camera by using javaCV, 

  // recoder settings:
  private int imageWidth  = 320;  
  private int imageHeight = 240;
  private int frameRate   = 30;

  recorder = new FFmpegFrameRecorder(ffmpeg_link, imageWidth, imageHeight, 1);
  recorder.setFormat("mp4");
  recorder.setFrameRate(frameRate);

  // frame settings:
  IplImage yuvIplimage = null;
  yuvIplimage = IplImage.create(320, 320, IPL_DEPTH_16U, 1); //32 not supported

  //image reader:
  private ImageReader mImageReader;
  mImageReader = ImageReader.newInstance(320, 320, ImageFormat.YUV_420_888, 10);  
  mImageReader.setOnImageAvailableListener(
                    mOnImageAvailableListener, mBackgroundHandler);

private final ImageReader.OnImageAvailableListener mOnImageAvailableListener
        = new ImageReader.OnImageAvailableListener() {

    @Override  
    public void onImageAvailable(ImageReader reader) {
        Image image = reader.acquireNextImage();// acquireLatestImage(); - also tried
        if (image == null)
            return; 

        final ByteBuffer buffer = image.getPlanes()[0].getBuffer();
        byte[] bytes = new byte[buffer.remaining()];
        buffer.get(bytes, 0, bytes.length); 


        if (yuvIplimage != null ) {
            // OPTION 1
            yuvIplimage.getByteBuffer().put(convertYUV420ToNV21(image)); 
            // OPTION 2
            //yuvIplimage.getByteBuffer().put(decodeYUV420SP(bytes,320,320));
            try {
                if (started)  {
                recorder.record(yuvIplimage);
                }
            } catch (Exception e) {
                e.printStackTrace();
            }
        }  

        image.close();
    }
};

Option 1 is to decode image to NV21 by using code:

 private byte[] convertYUV420ToNV21(Image imgYUV420) {
    byte[] rez;

    ByteBuffer buffer0 = imgYUV420.getPlanes()[0].getBuffer();
    ByteBuffer buffer2 = imgYUV420.getPlanes()[2].getBuffer();
    int buffer0_size = buffer0.remaining();
    int buffer2_size = buffer2.remaining();
    rez = new byte[buffer0_size + buffer2_size]; 

    buffer0.get(rez, 0, buffer0_size);  
    buffer2.get(rez, buffer0_size, buffer2_size);

    return rez;
}

enter image description here

option 2 is to convert to rgb as if i understand corect:

public byte[] decodeYUV420SP( byte[] yuv420sp, int width, int height) {
    final int frameSize = width * height;
    byte rgb[]=new byte[width*height];
    for (int j = 0, yp = 0; j < height; j++) {
        int uvp = frameSize + (j >> 1) * width, u = 0, v = 0;
        for (int i = 0; i < width; i++, yp++) {
            int y = (0xff & ((int) yuv420sp[yp])) - 16;
            if (y < 0) y = 0;
            if ((i & 1) == 0) {
                v = (0xff & yuv420sp[uvp++]) - 128;
                u = (0xff & yuv420sp[uvp++]) - 128;
            }
            int y1192 = 1192 * y;
            int r = (y1192 + 1634 * v);
            int g = (y1192 - 833 * v - 400 * u);
            int b = (y1192 + 2066 * u);
            if (r < 0) r = 0; else if (r > 262143) r = 262143;
            if (g < 0) g = 0; else if (g > 262143) g = 262143;
            if (b < 0) b = 0; else if (b > 262143) b = 262143;
            rgb[yp] = (byte) (0xff000000 | ((r << 6) & 0xff0000)
                    | ((g >> 2) & 0xff00) | ((b >> 10) & 0xff));
        }
    }
    return rgb;   }

enter image description here

it looks incorect also. wisch is the correct way to conert camera2 image to IplImage? and is it possible to do it on fly?

Solution

  • If the recorder requires NV21, then converting the image to that instead of RGB is likely the fastest option.

    But why don't you just use android.media.MediaRecorder? It's much more efficient and can use the hardware encoders.

    But if you need to stick with ffmpeg, your first option is incorrect for many devices. In addition, make sure you remove that buffer.get call earlier - it'll make the rest of the reads from plane 0 not work right, which may be your current problem. Once you read plane 0 once, .remaining() will return 0.

    The YUV image has 3 planes, and unless you've checked that the underlying format is actually NV21, you shouldn't blindly assume that., or assume that the row stride is equal to width. To be safe, you need to look at both row and pixel strides when copying the three planes into the semiplanar byte[].