I am trying to convert a DepthFrame
object that I have obtained from the Intel Realsense D455 camera to an OpenCV Mat
object in Java. I can get the the target depth of a pixel using DepthFrame.getDistance(x,y)
but I am trying to get the whole matrix so that I can get the distance values in meters, similar to the sample code in their Github repo, which is in C++.
I can convert any color image obtained from the camera stream (VideoFrame
or colored DepthFrame
) to a Mat
since they are 8 bits per pixel using the following function:
public static Mat VideoFrame2Mat(final VideoFrame frame) {
Mat frameMat = new Mat(frame.getHeight(), frame.getWidth(), CV_8UC3);
final int bufferSize = (int)(frameMat.total() * frameMat.elemSize());
byte[] dataBuffer = new byte[bufferSize];
frame.getData(dataBuffer);
ByteBuffer.wrap(dataBuffer).order(ByteOrder.LITTLE_ENDIAN).asReadOnlyBuffer().get(dataBuffer);
frameMat.put(0,0, dataBuffer);
return frameMat;
}
However, the un-colorized DepthFrame
values are 16 bits per pixel, and the above code gives an error when the CV_8UC1
is substituted with CV_16UC1
. The error arises because in the Java wrapper of the OpenCV function Mat.put(row, col, data[])
, there is a type check that allows only 8 bit Mat
s to be processed:
// javadoc:Mat::put(row,col,data)
public int put(int row, int col, byte[] data) {
int t = type();
if (data == null || data.length % CvType.channels(t) != 0)
throw new UnsupportedOperationException(
"Provided data element number (" +
(data == null ? 0 : data.length) +
") should be multiple of the Mat channels count (" +
CvType.channels(t) + ")");
if (CvType.depth(t) == CvType.CV_8U || CvType.depth(t) == CvType.CV_8S) {
return nPutB(nativeObj, row, col, data.length, data);
}
throw new UnsupportedOperationException("Mat data type is not compatible: " + t);
}
Therefore I tried to use the constructor of Mat
that accepts the array and wrote the following method:
public static Mat DepthFrame2Mat(final DepthFrame frame) {
byte[] dataBuffer = new byte[frame.getDataSize()];
frame.getData(dataBuffer);
ByteBuffer buffer = ByteBuffer.wrap(dataBuffer).order(ByteOrder.LITTLE_ENDIAN).asReadOnlyBuffer().get(dataBuffer);
Log.d(TAG, String.format("DepthFrame2Mat: w: %s h: %s capacity: %s remaining %s framedatasize: %s databufferlen: %s framedepth: %s type: %s " ,
frame.getWidth(), frame.getHeight(), buffer.capacity(), buffer.remaining(), frame.getDataSize(), dataBuffer.length, frame.getBitsPerPixel(), frame.getProfile().getFormat()));
return new Mat(frame.getHeight(), frame.getWidth(), CV_16UC1, buffer);
}
But now I keep getting the error E/cv::error(): OpenCV(4.5.5) Error: Assertion failed (total() == 0 || data != NULL) in Mat, file /build/master_pack-android/opencv/modules/core/src/matrix.cpp, line 428
Using the log
command seen in the function, I am checking if the data
is empty or null
, but it is not. Moreover, the DepthFrame
bit depth and type seems to be correct, too:
D/CvHelpers: DepthFrame2Mat: w: 640 h: 480 capacity: 614400 remaining 0 framedatasize: 614400 databufferlen: 614400 framedepth: 16 type: Z16
What could be the reason of this error? Is there a better way to handle this conversion?
Note: I have checked the SO questions such as this and examples on the web, however, all of them are in C++. I don't want to add JNI support just for creating a Mat
.
Even though not directly an OpenCV API solution, converting the byte array to short array in Java seems to work:
public static Mat DepthFrame2Mat(final DepthFrame frame) {
Mat frameMat = new Mat(frame.getHeight(), frame.getWidth(), CV_16UC1);
final int bufferSize = (int)(frameMat.total() * frameMat.elemSize());
byte[] dataBuffer = new byte[bufferSize];
short[] s = new short[dataBuffer.length / 2];
frame.getData(dataBuffer);
ByteBuffer.wrap(dataBuffer).order(ByteOrder.LITTLE_ENDIAN).asShortBuffer().get(s);
frameMat.put(0,0, s);
return frameMat;
}