uwpmediacapture

How to fix memory leak in GetPreviewFrameAsync


I have code based on this sample; https://github.com/microsoft/Windows-universal-samples/tree/master/Samples/CameraGetPreviewFrame. Every time the line;

await _mediaCapture.GetPreviewFrameAsync(_currentVideoFrame);

is hit we seem to leak memory. What am I not tidying up? I have also tried creating the template frame each time and disposing and nulling it on each loop - this doesn't seem to work either.

I have gone back to the original sample from Microsoft and it seems to leak too. Here is my code;

await Task.Run(async () =>
{
    try
    {
        var videoEncodingProperties = 
            _mediaCapture.VideoDeviceController.GetMediaStreamProperties
                (MediaStreamType.VideoPreview) as VideoEncodingProperties;

        Debug.Assert(videoEncodingProperties != null, nameof(videoEncodingProperties) + " != null");

        _currentVideoFrame = new VideoFrame(BitmapPixelFormat.Gray8,
            (int) videoEncodingProperties.Width,
            (int) videoEncodingProperties.Height);

        TimeSpan? lastFrameTime = null;

        while (_mediaCapture.CameraStreamState == CameraStreamState.Streaming)
        {
            token.ThrowIfCancellationRequested();

            await _mediaCapture.GetPreviewFrameAsync(_currentVideoFrame);

            if (!lastFrameTime.HasValue ||
                lastFrameTime != _currentVideoFrame.RelativeTime)
            {
                await CoreApplication.MainView.CoreWindow.Dispatcher.RunAsync
                    (CoreDispatcherPriority.Normal,
                    () =>
                    {
                        try
                        {
                            Debug.Assert(_currentVideoFrame != null,
                                        $"{nameof(_currentVideoFrame)} != null");

                            var bitmap = _currentVideoFrame.SoftwareBitmap.AsBitmap();

                            float focalLength = _cameraOptions == CameraOptions.Front ? AppSettings.FrontCameraFocalLength : AppSettings.RearCameraFocalLength;

                            _frameProcessor.ProcessFrame(bitmap, focalLength);
                        }
                        catch (Exception ex)
                        {
                            Debug.WriteLine($"Exception: {ex}");
                        }
                    });

                lastFrameTime = _currentVideoFrame.RelativeTime;
            }
        }
    }
    catch (Exception ex)
    {
        Debug.WriteLine($"Exception: {ex}");
    }
},
token);

This should simply get frames and put them through the _frameProcessor.ProcessFrame() call, but even when that does nothing (and I cut out everything except the GetPreviewFrameAsync) it leaks.

To repeat the problem, download the sample from; https://github.com/microsoft/Windows-universal-samples/tree/master/Samples/CameraGetPreviewFrame. Run the sample in the debugger with diagnostic tools (Debug->Windows->Show Diagnostic tools) remotely to a Surface Pro 4 (i5-6300U @2.4GHz) under Windows 10 v 1903 (18362.175). Turn on show frame check box and watch the memory as you press the GetPreviewFrameAsync button. The memory looks as follows where each uptick is me pressing the button;

Memory Leak


Solution

  • Using the MediaFrameReader API works around this bug well in our code, and is perhaps slightly more responsive. Microsoft have now added a note to the GetPreviewFrameAsync documentation page to point to this.

    This works for us;

    ...
    private MediaFrameReader _mediaFrameReader;
    ...
    
    private async Task InitializeCameraAsync()
    {
        if (_mediaCapture == null)
        {
             _mediaCapture = new MediaCapture();
             var frameSourceGroups = await MediaFrameSourceGroup.FindAllAsync();
             var selectedGroup = frameSourceGroups.FirstOrDefault(x => x.Id.Equals(_camera.UwpDeviceInfo.Id));
    
             try
             {
                 var mediaInitSettings = new MediaCaptureInitializationSettings
                 {
                     SourceGroup = selectedGroup,
                     VideoDeviceId = _camera.UwpDeviceInfo.Id,
                     AudioDeviceId = string.Empty,
                     StreamingCaptureMode = StreamingCaptureMode.Video,
                     MemoryPreference = MediaCaptureMemoryPreference.Cpu
                 };
    
                 await _mediaCapture.InitializeAsync(mediaInitSettings);
    
                 _isInitialized = true;
             }
             catch (UnauthorizedAccessException)
             {
    ...
             }
             catch (Exception ex)
             {
    ...
             }
    ...
    
             // Set-up for frameProcessing
             var sourceInfo = selectedGroup?.SourceInfos.FirstOrDefault(info =>
                 info.SourceKind == MediaFrameSourceKind.Color);
    ...                   
             var colorFrameSource = _mediaCapture.FrameSources[sourceInfo.Id];
             var preferredFormat = colorFrameSource.SupportedFormats
                 .OrderByDescending(x => x.VideoFormat.Width)
                 .FirstOrDefault(x => x.VideoFormat.Width <= 1920 &&
                      x.Subtype.Equals(MediaEncodingSubtypes.Nv12, StringComparison.OrdinalIgnoreCase));
    
             await colorFrameSource.SetFormatAsync(preferredFormat);
    
             _mediaFrameReader = await _mediaCapture.CreateFrameReaderAsync(colorFrameSource);
         }
    ...
    }
    
    ...
    
    private void _mediaFrameReader_FrameArrived(MediaFrameReader sender, MediaFrameArrivedEventArgs args)
    {
    ...
            var mediaFrameReference = sender.TryAcquireLatestFrame();
            var videoMediaFrame = mediaFrameReference?.VideoMediaFrame;
            var softwareBitmap = videoMediaFrame?.SoftwareBitmap;
    
            if (softwareBitmap != null && _frameProcessor != null)
            {
                if (_mediaCapture.CameraStreamState == CameraStreamState.Streaming)
                {
    ...
                    _frameProcessor.ProcessFrame(SoftwareBitmap.Convert(softwareBitmap, 
                            BitmapPixelFormat.Gray8).AsBitmap(), _camera);
    ...
                }
            }
    ...
        }
    }