I'm trying to modify the pixel buffer from live video feed from AVFoundation
to stream through OpenTok's API
. But whenever I try to do so and feed it through OpenTok's consumeFrame
, it crashes.
I am doing this so I can apply different live video effects (filters, stickers, etc).. I have tried converting CGImage->CVPixelBuffer
with different methods but nothing works.
func captureOutput(_ output: AVCaptureOutput, didOutput sampleBuffer: CMSampleBuffer, from connection: AVCaptureConnection) {
if !capturing || videoCaptureConsumer == nil {
return
}
guard let imageBuffer = CMSampleBufferGetImageBuffer(sampleBuffer)
else {
print("Error acquiring sample buffer")
return
}
guard let videoInput = videoInput
else {
print("Capturer does not have a valid input")
return
}
let time = CMSampleBufferGetPresentationTimeStamp(sampleBuffer)
videoFrame.clearPlanes()
videoFrame.timestamp = time
let height = UInt32(CVPixelBufferGetHeight(imageBuffer))
let width = UInt32(CVPixelBufferGetWidth(imageBuffer))
if width != captureWidth || height != captureHeight {
updateCaptureFormat(width: width, height: height)
}
// This is where I convert CVImageBuffer->CIImage, modify it, turn it into CGImage, then CGImage->CVPixelBuffer
guard let finalImage = makeBigEyes(imageBuffer) else { return }
CVPixelBufferLockBaseAddress(finalImage, CVPixelBufferLockFlags(rawValue: 0))
videoFrame.format?.estimatedCaptureDelay = 10
videoFrame.orientation = .left
videoFrame.clearPlanes()
videoFrame.planes?.addPointer(CVPixelBufferGetBaseAddress(finalImage))
delegate?.finishPreparingFrame(videoFrame)
videoCaptureConsumer!.consumeFrame(videoFrame)
CVPixelBufferUnlockBaseAddress(finalImage, CVPixelBufferLockFlags(rawValue: 0))
}
And here is my CGImage->CVPixelBuffer
method:
func buffer(from image: UIImage) -> CVPixelBuffer? {
let attrs = [kCVPixelBufferCGImageCompatibilityKey: kCFBooleanTrue, kCVPixelBufferCGBitmapContextCompatibilityKey: kCFBooleanTrue] as CFDictionary
var pixelBuffer : CVPixelBuffer?
let status = CVPixelBufferCreate(kCFAllocatorDefault, Int(image.size.width), Int(image.size.height), kCVPixelFormatType_32ARGB, attrs, &pixelBuffer)
guard (status == kCVReturnSuccess) else {
return nil
}
CVPixelBufferLockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
let pixelData = CVPixelBufferGetBaseAddress(pixelBuffer!)
let rgbColorSpace = CGColorSpaceCreateDeviceRGB()
let context = CGContext(data: pixelData, width: Int(image.size.width), height: Int(image.size.height), bitsPerComponent: 8, bytesPerRow: CVPixelBufferGetBytesPerRow(pixelBuffer!), space: rgbColorSpace, bitmapInfo: CGImageAlphaInfo.noneSkipFirst.rawValue)
context?.translateBy(x: 0, y: image.size.height)
context?.scaleBy(x: 1.0, y: -1.0)
UIGraphicsPushContext(context!)
image.draw(in: CGRect(x: 0, y: 0, width: image.size.width, height: image.size.height))
UIGraphicsPopContext()
CVPixelBufferUnlockBaseAddress(pixelBuffer!, CVPixelBufferLockFlags(rawValue: 0))
return pixelBuffer
}
I get this error on the first frame:
* Terminating app due to uncaught exception 'NSRangeException', reason: '* -[NSConcretePointerArray pointerAtIndex:]: attempt to access pointer at index 1 beyond bounds 1'
If you made it this far, thank you for reading. I've been stuck on this issue a while now, so any kind of pointer would be greatly appreciated. Thanks.
Since you are converting the camera(NV12) frame to RGB, you need to set pixelFormat
to OTPixelFormatARGB
on videoFrame.format