I’m trying to use Core ML model inference to render screenshots continuously. I’ve read about asynchronous prediction at WWDC, and it seems like it could improve performance. I can do it well in sync mode, but when I switch to asynchronous, things get a bit wonky.
My workflow is like this: I take a screenshot, get the copy from the screenshot, run the Core ML model asynchronously, set the RenderData, and wait for the MTKView call “draw()”. After I get the screenshot copy, I create a Task to keep the screen capture going.
Since I’m using asynchronous prediction, I need to make sure that I’m using resources in a thread-safe way. I’m using an actor RenderSemaphore to do that. But even though I update the setRenderData() function or renderAsync() that checks the frame timestamp, it still shows not sequential behavior sometimes when the refresh rate is 120 FPS (60 FPS might be okay).
I’ve attached my code below. Can you help me figure out where I went wrong? Thanks a bunch!
actor RenderSemaphore {
private var currentBufferIndex = 0
private var writeBufferIndex = 0
private var frameBuffers: [FrameData?]
private var lastReadTimeStamp: Double = 0.0
private var lastWriteTimeStamp: Double = 0.0
private var isRendering = false
private var lastCommittedTimeStamp: Double = 0
private var activeTaskCount = 0
private let maxTasks = 3
init(count: Int = 3) {
frameBuffers = Array(repeating: nil, count: count)
}
func beginRender(timeStamp: Double) async -> Bool {
if isRendering { return false }
isRendering = true
return true
}
func endRender() {
isRendering = false
}
func setBuffers(surface: IOSurface, texture: MTLTexture, depth: MTLBuffer, timeStamp: Double) -> Bool {
guard writeBufferIndex < frameBuffers.count else { return false }
guard timeStamp > lastWriteTimeStamp else { return false }
let frameData = FrameData(
surface: surface,
texture: texture,
depthBuffer: depth,
timeStamp: timeStamp
)
frameBuffers[writeBufferIndex] = frameData
lastWriteTimeStamp = timeStamp
writeBufferIndex = (writeBufferIndex + 1) % TotalBufferCount
return true
}
func getAvailableBuffer() -> FrameData? {
guard let frame = frameBuffers[currentBufferIndex] else { return nil }
if frame.timeStamp > lastReadTimeStamp {
lastReadTimeStamp = frame.timeStamp
currentBufferIndex = (currentBufferIndex + 1) % TotalBufferCount
return frame
}
return nil
}
// for setRenderDataNormalized
func tryEnterTask() -> Bool {
guard activeTaskCount < maxTasks else { return false }
activeTaskCount += 1
return true
}
func exitTask() {
activeTaskCount -= 1
}
func commitFrame(timeStamp: Double,
commandBuffer: MTLCommandBuffer,
drawable: MTLDrawable) async -> Bool {
guard timeStamp > lastCommittedTimeStamp else {
print("Drop frame at commit: \(timeStamp) <= \(lastCommittedTimeStamp)")
return false
}
commandBuffer.present(drawable)
commandBuffer.commit()
lastCommittedTimeStamp = timeStamp
return true
}
}
func setRenderData(surface: IOSurface, renderData: [Float], timeStamp: Double) async {
guard await renderSemaphore.tryEnterTask() else {
print("drop frame")
return
}
let result: MTLTexture!
let _textureName = "RenderData"
var px: Unmanaged<CVPixelBuffer>?
let status = CVPixelBufferCreateWithIOSurface(kCFAllocatorDefault,
surface,
nil,
&px)
guard status == kCVReturnSuccess, let screenImage = px?.takeRetainedValue() else {
await renderSemaphore.exitTask()
return }
// use texture cache
var texture: CVMetalTexture? = nil
let width = CVPixelBufferGetWidthOfPlane(screenImage, 0)
let height = CVPixelBufferGetHeightOfPlane(screenImage, 0)
let result2 = CVMetalTextureCacheCreateTextureFromImage(
kCFAllocatorDefault,
self._textureCache!,
screenImage,
nil,
MTLPixelFormat.bgra8Unorm,
width,
height,
0, &texture)
CVMetalTextureCacheFlush(self._textureCache!, 0)
if result2 == kCVReturnSuccess, let cvTexture = texture {
let mtlTexture = CVMetalTextureGetTexture(cvTexture)
result = mtlTexture
result?.label = "\(timeStamp)"// <--- Updated 20241118
} else {
await renderSemaphore.exitTask()
return }
let buffer = self.device.makeBuffer(bytes: renderData, length: renderData.count*MemoryLayout<Float>.stride)!
let ret = await renderSemaphore.setBuffers(
surface: surface,
texture: result,
depth: buffer,
timeStamp: timeStamp
)
if !ret {
print("drop frame")
}
await renderSemaphore.exitTask()
}
func draw(in view: MTKView) {
Task {
await self.renderAsync(view)
}
}
var lastTimeStamp: Double = 0
func renderAsync(_ view: MTKView) async {
guard await renderSemaphore.beginRender(timeStamp: frame.timeStamp) else {
return
}
guard let frame = await renderSemaphore.getAvailableBuffer() else {
await renderSemaphore.endRender()
return
}
if frame.timeStamp <= lastTimeStamp {
print("Skipping out-of-order frame: \(frame.timeStamp) <= \(lastTimeStamp)")
await renderSemaphore.endRender()
return
}
lastTimeStamp = frame.timeStamp
// Updated 20241118
print(lastTimeStamp)
print(frame.texture.label!)
guard let commandBuffer = _commandQueue.makeCommandBuffer(),
let renderPassDescriptor = await view.currentRenderPassDescriptor,
let renderEncoder = commandBuffer.makeRenderCommandEncoder(descriptor: renderPassDescriptor) else {
await renderSemaphore.endRender()
return
}
renderEncoder.label = "new render encoder"
let (modelMatrix, leftView, rightView, projectMatrix) = self.updateMatricesAsync()
var (sceneConstants, modelConstants) = self.getRenderConstants(
renderPassDescriptor: renderPassDescriptor,
projectMatrix: projectMatrix,
modelMatrix: modelMatrix
)
// Vertex Shader
renderEncoder.setVertexBuffer(self._vertexBufferDepth[currentBufferIndex], offset: 0, index: 0)
renderEncoder.setVertexBytes(&sceneConstants, length: SceneConstants.stride, index: 1)
renderEncoder.setVertexBytes(&modelConstants, length: ModelConstants.stride, index: 2)
renderEncoder.setVertexBuffer(frame.depthBuffer, offset: 0, index: 3)
renderEncoder.setFragmentSamplerState(samplerState, index: 0)
renderEncoder.setFragmentBytes(&material, length: Material.stride, index: 1)
renderEncoder.setFragmentTexture(frame.texture, index: 0)
sceneConstants.viewport = 0
renderEncoder.setViewport(viewport)
renderEncoder.setScissorRects(scissors)
sceneConstants.viewMatrix = viewMatrix
renderEncoder.setVertexBytes(&sceneConstants, length: SceneConstants.stride, index: 1)
renderEncoder.setRenderPipelineState(self._renderPipelineState)
renderEncoder.drawIndexedPrimitives(type: .triangleStrip,
indexCount: self._indexBufferDepth.length/MemoryLayout<UInt32>.size,
indexType: .uint32,
indexBuffer: self._indexBufferDepth,
indexBufferOffset: 0)
renderEncoder.endEncoding()
commandBuffer.addCompletedHandler() { [weak self] _ in
self?.countFPS()
}
// commit frame in actor
let success = await renderSemaphore.commitFrame(
timeStamp: frame.timeStamp,
commandBuffer: commandBuffer,
drawable: view.currentDrawable!
)
await renderSemaphore.endRender()
}
I have updated the code that utilizes all the actor features, which can only be executed with a single thread.
20241118 updated
I found when rendering, the timeStamp is not matched its frame's texture label that stores its timestamp, as above code marked with "20241118 updated". I don't know why the codes as below 2 lines is not matched. They should be the same!
print(lastTimeStamp)
print(frame.texture.label!)
Please help me!
20241121 updated
I think the result of print() shows all frames being sorted. Even though I use a sorting method to insert the frames into the array, the result sometimes includes duplicate frames or older frames.
I suspect this happens due to one of the following reasons:
The frame's address is updated or overwritten by another frame. The commit action is concurrent or not synchronized. However, I believe these two issues shouldn’t occur.
MultiThreaded rendering with actor - Apple Developer Forum
Replied by Apple DTS Engineer Quinn, said that I should draw(in:) frames with sync.
I can't get how to sync get the data that have to access using async before, and the Apple Engineer Quinn answered it. I think everyone here can have a look that Quinn replied.
But I'll still give the bounty to Ostoura. Thank you for your answer.