swiftcore-imageopentoktokboxcvpixelbuffer

TokBox: 'consumeFrame' crashes when using a modified pixel buffer


I'm trying to modify the pixel buffer from live video feed from AVFoundation to stream through OpenTok's API. But whenever I try to do so and feed it through OpenTok's consumeFrame, it crashes.

I am doing this so I can apply different live video effects (filters, stickers, etc).. I have tried converting CGImage->CVPixelBuffer with different methods but nothing works.

func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
    if !capturing || videoCaptureConsumer == nil {
        return
    }

    guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
        else {
            print("Error acquiring sample buffer")
            return
    }

    guard let videoInput = videoInput
        else {
            print("Capturer does not have a valid input")
            return
    }

    let time = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
    videoFrame.clearPlanes()

    videoFrame.timestamp = time
    let height = UInt32(CVPixelBufferGetHeight(imageBuffer))
    let width = UInt32(CVPixelBufferGetWidth(imageBuffer))

    if width != captureWidth || height != captureHeight {
        updateCaptureFormat(width: width, height: height)
    }

    // This is where I convert CVImageBuffer->CIImage, modify it, turn it into CGImage, then CGImage->CVPixelBuffer
    guard let finalImage = makeBigEyes(imageBuffer) else { return }

    CVPixelBufferLockBaseAddress(finalImage, CVPixelBufferLockFlags(rawValue: 0))

    videoFrame.format?.estimatedCaptureDelay = 10
    videoFrame.orientation = .left

    videoFrame.clearPlanes()
    videoFrame.planes?.addPointer(CVPixelBufferGetBaseAddress(finalImage))

    delegate?.finishPreparingFrame(videoFrame)

    videoCaptureConsumer!.consumeFrame(videoFrame)

    CVPixelBufferUnlockBaseAddress(finalImage, CVPixelBufferLockFlags(rawValue: 0))
}

And here is my CGImage->CVPixelBuffer method:

    func buffer(from image: UIImage) -> CVPixelBuffer? {
    let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue, kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary
    var pixelBuffer : CVPixelBuffer?
    let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(image.size.width), Int(image.size.height), kCVPixelFormatType_32ARGB, attrs, &pixelBuffer)
    guard (status == kCVReturnSuccess) else {
        return nil
    }

    CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
    let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer!)

    let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
    let context = CGContext(data: pixelData, width: Int(image.size.width), height: Int(image.size.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!), space: rgbColorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue)

    context?.translateBy(x: 0, y: image.size.height)
    context?.scaleBy(x: 1.0, y: -1.0)

    UIGraphicsPushContext(context!)
    image.draw(in: CGRect(x: 0, y: 0, width: image.size.width, height: image.size.height))
    UIGraphicsPopContext()
    CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))

    return pixelBuffer
}

I get this error on the first frame:

* Terminating app due to uncaught exception 'NSRangeException', reason: '* -[NSConcretePointerArray pointerAtIndex:]: attempt to access pointer at index 1 beyond bounds 1'

If you made it this far, thank you for reading. I've been stuck on this issue a while now, so any kind of pointer would be greatly appreciated. Thanks.


Solution

  • Since you are converting the camera(NV12) frame to RGB, you need to set pixelFormat to OTPixelFormatARGB on videoFrame.format