javafxyuvpixelformat

Render YUV in JavaFX

I need to render a yuyv422 stream in JavaFX with minimum latency. If I convert it to RGB, I can use an ImageView with a WritableImage with a PixelFormat instance, and it works, but the RGB conversion consumes a lot of CPU, specially with high resolutions. I saw this exact feature request

https://bugs.openjdk.java.net/browse/JDK-8091933

but seems it will not be implemented in Java 9. And if it does, I wonder if it won't introduce latency or demand too much CPU. Is there another way using JavaFX?

Solution

  • In General:
    Image processing is always expensive, this is why Vectorization or Hardware Acceleration is used for these tasks. Simple looping through an Image with just one thread is already really slow, especially in java. On top of that people tend to use Color objects for color modifications which is tremendously slow.

    Pure Java:
    If you want to keep your code in pure Java. You should check which internal format is used for the WriteableImage by calling: 

    myImage.getPixelWriter().getPixelFormat().getType()

    If the internal format isn't RGB adapt your color conversion to the given format to avoid double conversion. Additionally make sure that your code is optimized as much as possible:
    -Don't use any objects except arrays
    -Minimize the use of local variables
    You can also try to multithread the conversion process via parallel loops.

    JNI:
    Moving away from Java opens up a lot of possibilities. There are several platform independent libraries for converting YUV to RGB and back:

    OpenCV:
    Easy to use and coming already with an java API:

    byte[] myYuvImage = null; //your image here
byte[] myRgbImage = new byte[width * height * 3]; //the output image
Mat yuvMat = new Mat(height, width, CvType.CV_8UC2); //YUV422 should be 2 channel
Mat rgbMat = new Mat(height, width, CvType.CV_8UC3);
yuvMat.put(0,0, myYuvImage);
Imgproc.cvtColor(yuvMat, rgbMat, Imgproc.COLOR_YUV2RGB_Y422);
rgbMat.get(0, 0, myRgbImage);

    Intel IPP:
    Only available via JNI. You would use ippiRGBToYUV422_8u_C3C2R see RGBToYUV422 for more information.

    SwScale as part of FFmpeg:
    Only available via JNI. See this answer and adapt the example.

    My personal experience is that IPP offers by far the best performance even on AMD machines. However the license it comes with may be free but it prohibits decompiling which might be an not compatible with LGPL libraries.