For performance reasons, I ditched Python-Opnecv/FFmpeg solution and moved on to Java.
But to my surprise, I am not able to find any better and complete solution as we have in Python. I tried using vlcj but again it just gives more of a command line kind of interface. I am not able to find any callback kind of mechanism for reading and analyzing all the frames.
I also tried using Java Sockets but wasn't able to do anything more than establishing a connection with Ip Camera streaming h264 video over RTSP.
Note: It will be running in a server environment so we don't want to show any frame, we just need run certain other operations on frames.
Please guide me in the right direction.
If you want to get access to the video frame buffer while media is playing you have a couple of options.
I'l assume you are using vlcj 4.x+, which is current at time of writing.
First, you can use an EmbeddedMediaPlayer with a CallbackVideoSurface.
You can use the MediaPlayerFactory to create your video surface.
When you create your video surface, it requires a RenderCallback implementation that you provide.
Create the embedded media player as normal, and invoke mediaPlayer.setVideoSurface() to set your video surface.
It is this render callback implementation class that will be called back by VLC with raw video frame data in the form of a ByteBuffer backed by native memory. You can then do your analysis on the data in this byte buffer.
The second approach is to look instead at the CallbackMediaPlayerComponent class - this class aims to make it very easy for you to get an out-of-the-box working media player and provides a way for you to plug in only the bits you want to customise. In this case you plug in your render callback implementation to do your analysis.
There are examples in the vlcj source code at the github project page that show all of this. One of the examples processes this buffer to dynamically convert the video to greyscale, but obviously you can do anything you want with the frame data.
The method is named "onDisplay()" but you do not have to actually display the video anywhere if you're only interested in performing some analysis.
This is the extent of what vlcj can provide if you want to access the video frame data.