metalrender-to-texturemtkview

MTKView frequently displaying scrambled MTLTextures


I am working on an MTKView-backed paint program which can replay painting history via an array of MTLTextures that store keyframes. I am having an issue in which sometimes the content of these MTLTextures is scrambled.

As an example, say I want to store a section of the drawing below as a keyframe:

prior to keyframe creation

During playback, sometimes the drawing will display exactly as intended, but sometimes, it will display like this:

during keyframe playback

Note the distorted portion of the picture. (The undistorted portion constitutes a static background image that's not part of the keyframe in question)

I describe the way I Create individual MTLTextures from the MTKView's currentDrawable below. Because of color depth issues I won't go into, the process may seem a little round-about.

I first get a CGImage of the subsection of the screen that constitutes a keyframe.
I use that CGImage to create an MTLTexture tied to the MTKView's device. I store that MTLTexture into a MTLTextureStructure that stores the MTLTexture and the keyframe's bounding-box (which I'll need later) Lastly, I store in an array of MTLTextureStructures (keyframeMetalArray). During playback, when I hit a keyframe, I get it from this keyframeMetalArray.

The associated code is outlined below.

let keyframeCGImage = weakSelf!.canvasMetalViewPainting.mtlTextureToCGImage(bbox: keyframeBbox, copyMode: copyTextureMode.textureKeyframe) // convert from MetalTexture to CGImage

let keyframeMTLTexture = weakSelf!.canvasMetalViewPainting.CGImageToMTLTexture(cgImage: keyframeCGImage)

let keyframeMTLTextureStruc = mtlTextureStructure(texture: keyframeMTLTexture, bbox: keyframeBbox, strokeType: brushTypeMode.brush)

weakSelf!.keyframeMetalArray.append(keyframeMTLTextureStruc)

Without providing specifics about how each conversion is happening, I wonder if, from an architecture design point, I'm overlooking something that is corrupting my data stored in the keyframeMetalArray. It may be unwise to try to store these MTLTextures in volatile arrays, but I don't know that for a fact. I just figured using MTLTextures would be the quickest way to update content.

By the way, when I swap out arrays of keyframes to arrays of UIImage.pngData, I have no display issues, but it's a lot slower. On the plus side, it tells me that the initial capture from currentDrawable to keyframeCGImage is working just fine.

Any thoughts would be appreciated.

p.s. adding a bit of detail based on the feedback:

mtlTextureToCGImage:

func mtlTextureToCGImage(bbox: CGRect, copyMode: copyTextureMode) -> CGImage {

    let kciOptions = [convertFromCIContextOption(CIContextOption.outputPremultiplied): true,
                      convertFromCIContextOption(CIContextOption.useSoftwareRenderer): false] as [String : Any]
    let bboxStrokeScaledFlippedY = CGRect(x: (bbox.origin.x * self.viewContentScaleFactor), y: ((self.viewBounds.height - bbox.origin.y - bbox.height) * self.viewContentScaleFactor), width: (bbox.width * self.viewContentScaleFactor), height: (bbox.height * self.viewContentScaleFactor))

let strokeCIImage = CIImage(mtlTexture: metalDrawableTextureKeyframe,
                                  options: convertToOptionalCIImageOptionDictionary(kciOptions))!.oriented(CGImagePropertyOrientation.downMirrored)
      let imageCropCG = cicontext.createCGImage(strokeCIImage, from: bboxStrokeScaledFlippedY, format: CIFormat.RGBA8, colorSpace: colorSpaceGenericRGBLinear)

      cicontext.clearCaches()

      return imageCropCG!

} // end of func mtlTextureToCGImage(bbox: CGRect)

CGImageToMTLTexture:

func CGImageToMTLTexture (cgImage: CGImage) -> MTLTexture {

    // Note that we forego the more direct method of creating stampTexture:
    //let stampTexture = try! MTKTextureLoader(device: self.device!).newTexture(cgImage: strokeUIImage.cgImage!, options: nil)
    // because  MTKTextureLoader seems to be doing additional processing which messes with the resulting texture/colorspace

    let width = Int(cgImage.width)
    let height = Int(cgImage.height)


    let bytesPerPixel = 4

    let rowBytes = width * bytesPerPixel
    //
    let texDescriptor = MTLTextureDescriptor.texture2DDescriptor(pixelFormat: .rgba8Unorm,
                                                                 width: width,
                                                                 height: height,
                                                                 mipmapped: false)
    texDescriptor.usage = MTLTextureUsage(rawValue: MTLTextureUsage.shaderRead.rawValue)
    texDescriptor.storageMode = .shared
    guard let stampTexture = device!.makeTexture(descriptor: texDescriptor) else {
      return brushTextureSquare // return SOMETHING

    }

    let dstData: CFData = (cgImage.dataProvider!.data)!
    let pixelData = CFDataGetBytePtr(dstData)

    let region = MTLRegionMake2D(0, 0, width, height)

    print ("[MetalViewPainting]: w= \(width) | h= \(height)  region = \(region.size)")

    stampTexture.replace(region: region, mipmapLevel: 0, withBytes: pixelData!, bytesPerRow: Int(rowBytes))

    return stampTexture

  } // end of func CGImageToMTLTexture (cgImage: CGImage)

Solution

  • The type of distortion looks like a bytes-per-row alignment issue between CGImage and MTLTexture. You're probably only seeing this issue when your image is a certain size that falls outside of the bytes-per-row alignment requirement of your MTLDevice. If you really need to store the texture as a CGImage, ensure that you are using the bytesPerRow value of the CGImage when copying back to the texture.