iosswiftdeep-copypoolcmsamplebuffer

Pulling data from a CMSampleBuffer in order to create a deep copy

I am trying to create a copy of a CMSampleBuffer as returned by captureOutput in a AVCaptureVideoDataOutputSampleBufferDelegate.

Since the CMSampleBuffers come from a preallocated pool of (15) buffers, if I attach a reference to them they cannot be recollected. This causes all remaining frames to be dropped.

To maintain optimal performance, some sample buffers directly reference pools of memory that may need to be reused by the device system and other capture inputs. This is frequently the case for uncompressed device native capture where memory blocks are copied as little as possible. If multiple sample buffers reference such pools of memory for too long, inputs will no longer be able to copy new samples into memory and those samples will be dropped.

If your application is causing samples to be dropped by retaining the provided CMSampleBufferRef objects for too long, but it needs access to the sample data for a long period of time, consider copying the data into a new buffer and then releasing the sample buffer (if it was previously retained) so that the memory it references can be reused.

Obviously I must copy the CMSampleBuffer but CMSampleBufferCreateCopy() will only create a shallow copy. Thus I conclude that I must use CMSampleBufferCreate(). I filled in the 12! parameters that the constructor needs but ran into the problem that my CMSampleBuffers do not contain a blockBuffer (not entirely sure what that is but it seems important).

This question has been asked several times but not answered.

Deep Copy of CMImageBuffer or CVImageBuffer and Create a copy of CMSampleBuffer in Swift 2.0

One possible answer is "I finally figured out how to use this to create a deep clone. All the copy methods reused the data in the heap which kept would lock the AVCaptureSession. So I had to pull the data out into a NSMutableData object and then created a new sample buffer." credit to Rob on SO. However, I do not know how to do this correcly.

If you are interested, this is the output of print(sampleBuffer). There is no mention of blockBuffer, aka CMSampleBufferGetDataBuffer returns nil. There is a imageBuffer, but creating a "copy" using CMSampleBufferCreateForImageBuffer does not seem to free the CMSampleBuffer either.

EDIT: Since this question has been posted I have been trying even more ways of copying the memory.

I did the same thing that user Kametrixom tried. This is my attempt at the same idea, to first copy the CVPixelBuffer then use CMSampleBufferCreateForImageBuffer to create the final sample buffer. However this results in one of two error:

You can see that both Kametrixom and I did use CMSampleBufferGetFormatDescription(sampleBuffer) to try to copy the source buffer's format description. Thus, I'm not sure why the format of the given media does not match the given format description.

Solution

  • Alright, I think I finally got it. I created a helper extension to make a full copy of a CVPixelBuffer:

    extension CVPixelBuffer {
    func copy() -> CVPixelBuffer {
        precondition(CFGetTypeID(self) == CVPixelBufferGetTypeID(), "copy() cannot be called on a non-CVPixelBuffer")

        var _copy : CVPixelBuffer?
        CVPixelBufferCreate(
            nil,
            CVPixelBufferGetWidth(self),
            CVPixelBufferGetHeight(self),
            CVPixelBufferGetPixelFormatType(self),
            CVBufferGetAttachments(self, kCVAttachmentMode_ShouldPropagate)?.takeUnretainedValue(),
            &_copy)

        guard let copy = _copy else { fatalError() }

        CVPixelBufferLockBaseAddress(self, kCVPixelBufferLock_ReadOnly)
        CVPixelBufferLockBaseAddress(copy, 0)

        for plane in 0..<CVPixelBufferGetPlaneCount(self) {
            let dest = CVPixelBufferGetBaseAddressOfPlane(copy, plane)
            let source = CVPixelBufferGetBaseAddressOfPlane(self, plane)
            let height = CVPixelBufferGetHeightOfPlane(self, plane)
            let bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(self, plane)

            memcpy(dest, source, height * bytesPerRow)
        }

        CVPixelBufferUnlockBaseAddress(copy, 0)
        CVPixelBufferUnlockBaseAddress(self, kCVPixelBufferLock_ReadOnly)

        return copy
    }
}

    Now you can use this in your didOutputSampleBuffer method:

    guard let pixelBuffer = CMSampleBufferGetImageBuffer(sampleBuffer) else { return }

let copy = pixelBuffer.copy()

toProcess.append(copy)

    But be aware, one such pixelBuffer takes up about 3MB of memory (1080p), which means that in 100 frames you got already about 300MB, which is about the point at which the iPhone says STAHP (and crashes).

    Note that you don't actually want to copy the CMSampleBuffer since it only really contains a CVPixelBuffer because it's an image.