wpflibvlcip-cameralibvlcsharph.265

LibVlcSharp SetVideoCallbacks(Lock, null, Display) not getting called when playing stream from IP camera


I am trying to develop an application that streams videos from IP cameras and also processes some frames from those IP cameras. I have set the Video callbacks using SetVideoCallbacks(Lock, null, Display) but the Lock and Display methods are not getting called.

I tried to do this with a mp4 video and that worked correctly. But when I try to do this with my IP camera I can see the stream in the VideoView but the Lock and Display methods are not getting called. Earlier I thought that this could be because the frames are getting decoded using the GPU but the Task Manager shows that the GPU usage is pretty negligible when I run the application, but the CPU usage is pretty high(~40%). I have also tried setting the EnableHardwareDecoding to both true and false. I believe that there is some format error. Maybe the camera rtsp stream is not getting decoded to I420 format. My camera stream is H.265 codec but also supports H.264 codec. Also I know that the fact that I can see the video feed is itself fishy because if the Lock function was getting called then I wouldn't see the video (unless I copy the frames). Please help me to solve this issue or guide me if there is some better way. Thanks.

public partial class MainWindow : Window
{
    private Camera _camera;
    private LibVLC _libVLC;
    private LibVLCSharp.Shared.MediaPlayer _mediaPlayer;

    public const uint width = 1280;
    public const uint height = 720;
    public const uint pitch = width*2;//because of I420 which uses 12 bits per pixel
    public MemoryMappedFile? CurrentMappedFile;
    public MemoryMappedViewAccessor? CurrentMappedViewAccessor;
    public ConcurrentQueue<(MemoryMappedFile? file, MemoryMappedViewAccessor? accessor)> FramesToProcess = new ConcurrentQueue<(MemoryMappedFile? file, MemoryMappedViewAccessor? accessor)>();
    public int FrameCounter = 0;
    public CancellationTokenSource cancellationTokenSource = new CancellationTokenSource();

    public MainWindow()
    {
        try
        {
            InitializeComponent();
            Core.Initialize();
            _camera = new Camera("Camera", "192.168.1.188", 554, "admin", "123456", "stream");
            _libVLC = new LibVLC("--verbose=2");
            _mediaPlayer = new LibVLCSharp.Shared.MediaPlayer(_libVLC) { EnableHardwareDecoding = true};

            //_libVLC.Log += (sender, e) => Debug.WriteLine($"[{e.Level}] {e.Module}:{e.Message}");
            var media = new Media(_libVLC, _camera.GetStreamUrl(), FromType.FromLocation);

            media.AddOption(":no-audio");
            _mediaPlayer.Media = media;

            _mediaPlayer.SetVideoFormat("I420", width, height, pitch);
            _mediaPlayer.SetVideoCallbacks(Lock, null, Display);
            _mediaPlayer.Stopped += (s, e) => cancellationTokenSource.CancelAfter(1);
            
            VideoView.MediaPlayer = _mediaPlayer;
            VideoView.MediaPlayer.Play();
            this.Closed += MainWindow_Closed;
            Task.Run(() => ProcessFrames(cancellationTokenSource.Token), cancellationTokenSource.Token);
        }
        catch(Exception ex) {
            Application.Current.Dispatcher.Invoke(() => MessageBox.Show(ex.Message));
            Debug.WriteLine(ex.Message);
        }
    }


    private IntPtr Lock(IntPtr opaque, IntPtr planes)
    {
        //Debug.WriteLine("Lock is called");

        Dispatcher.Invoke(() =>
        {
            lockTextBox.Text = $"Lock method called {FrameCounter}";
        });

        
        CurrentMappedFile = MemoryMappedFile.CreateNew(null, pitch * height);
        CurrentMappedViewAccessor = CurrentMappedFile.CreateViewAccessor();

        
        Marshal.WriteIntPtr(planes, CurrentMappedViewAccessor.SafeMemoryMappedViewHandle.DangerousGetHandle());

        return IntPtr.Zero;
    }

    private void Display(IntPtr opaque, IntPtr picture)
    {
        //Debug.WriteLine($"Debug is called with framecounter= {FrameCounter}");
        Dispatcher.Invoke(() =>
        {
            displayTextBox.Text = $"Display method called {FrameCounter}";
        });

        if (FrameCounter  == 5)
        {
            //Enqueued frames to be disposed in the processing thread
            FramesToProcess.Enqueue((CurrentMappedFile, CurrentMappedViewAccessor));
            CurrentMappedFile = null;
            CurrentMappedViewAccessor = null;
            FrameCounter = 0;//reset counter
        }
        else
        {
            if (CurrentMappedViewAccessor != null && CurrentMappedFile != null)
            {
                CurrentMappedViewAccessor.Dispose();
                CurrentMappedFile.Dispose();
            }
            
            CurrentMappedFile = null;
            CurrentMappedViewAccessor = null;
            FrameCounter++;
        }
    }
}

Solution

  • You can't use both a VideoView and video callbacks at the same time in the current libvlc API, so I think there's a conflict there.

    Your options are limited : either open two streams, or do the display yourself in the callback (slower, harder and more CPU intensive than directly displaying in a VideoView. LibVLC 4 will have a new callback to get directX textures directly but it's still in preview)