cocoaopenglavfoundationqtkit

How do I get an openGL texture out of a video using AVFoundation?


Building a 64bit native OSX (not iOS) ap targeting 10.7+. Somewhat new to dealing with video files in a Cocoa universe.

I would like to be able to open a video file and display the output inside an openGL render (IE, I would like to be able to efficiently access the framebuffer of a video and turn each frame into an opengl texture.)

Conceptually this seems straightforward, but I'm having a difficult time wading through various (old and deprecated)examples and options, all of which seem to have been recently deprecated in favor of AVFoundation. It's possible I missed something, but examples of using AVFoundation with OpenGL seem thin on the ground.

To clarify a bit further, this sample application (QTCoreVideo101 from Apple) does more or less exactly what I want, except that it is built around the deprecated QTKit will therefore not even compile in 64 bit.

I am reading through the AVFoundation docs right now, but I'm still not sure it make sense to try and get a glTexture out of AVFoundation or if I should be looking elsewhere.

UPDATE

This is the solution I ended up going with. "thisLayer.layerSource.videoPlayerOutput" is AVPlayerItemVideoOutput object.

if ([thisLayer.layerSource.videoPlayerOutput hasNewPixelBufferForItemTime:playerTime]){

    frameBuffer= [thisLayer.layerSource.videoPlayerOutput copyPixelBufferForItemTime:playerTime itemTimeForDisplay:NULL];

    CVReturn result= CVOpenGLTextureCacheCreateTextureFromImage(NULL,
                                           textureCache,
                                           frameBuffer,
                                           NULL,
                                           &textureRef);
    if(result == kCVReturnSuccess){
            // These appear to be GL_TEXTURE_RECTANGLE_ARB
            thisLayer.layerSource.vid_glTextureTarget=CVOpenGLTextureGetTarget(textureRef);
            thisLayer.layerSource.vid_glTexture=CVOpenGLTextureGetName(textureRef);
            thisLayer.layerSource.vid_glTextureSize=NSMakeSize(CVPixelBufferGetWidth(frameBuffer), CVPixelBufferGetHeight(frameBuffer));
            thisLayer.layerSource.vid_ciimage=[CIImage imageWithCVImageBuffer:frameBuffer];
            CFRelease(textureRef);
            CVOpenGLTextureCacheFlush(textureCache, 0);
    }else{
            NSLog(@"INTERNAL ERROR FAILED WITH CODE: %i",result);
    }
    CVBufferRelease(frameBuffer);
}

Solution

  • AVAssetReaderTrackOutput (added to your AVAssetReader) will output a CVPixelBufferRef, of which you can specify your preferred format to upload to OpenGL via glTexImage or glTexSubImage.