I have SpriteKit nodes on which I apply Core Image filters using SKEffectNode. The sprites are images added by the user on the scene, and can be of any size. Some of the filters change the size of the rendered sprites. When that size exceeds the limit allowed by Metal, the app crashes with an error like this:
-[MTLTextureDescriptorInternal validateWithDevice:]:1357: failed assertion `Texture Descriptor Validation
MTLTextureDescriptor has width (9050) greater than the maximum allowed size of 8192.
MTLTextureDescriptor has height (9050) greater than the maximum allowed size of 8192.
How can I prevent any image processing I make in real-time from exceeding Metal's limits?
Here's the SpriteKit code:
var mySprite: SKSpriteNode!
var myEffectNode: SKEffectNode!
var sliderInput: Double = 0
override func didMove(to view: SKView) {
// Sprite from an image
// The user picks the image, it could be of any size
mySprite = SKSpriteNode(imageNamed: "myImage")
mySprite.name = "mySprite"
myEffectNode = SKEffectNode()
myEffectNode.addChild(mySprite)
addChild(myEffectNode)
}
// Called by SwiftUI
// The filter changes dynamically in real-time
func updateEffects() {
myEffectNode.filter = CIFilter(name: "CIZoomBlur", parameters: [
"inputAmount": sliderInput
])
}
Desired pseudo code:
func carefullyUpdateEffects() {
// Compute the image processing
// Get the texture size limit on this device
// Check if the expected output of the processing exceeds the limit
// If no, proceed and render
// If yes, handle case
}
I have made progress and would like to share it.
A Core Image filter may return a result that, when rendered, exceeds Metal's size limit per texture.
Example: A CIFilter(name: "CIZoomBlur", parameters: ["inputAmount": 140]) on an image of 1024x1024 produces an image of 17325*17325. When that filter returns its result to SpriteKit, the renderer crashes.
How to get the filter's expected output size before it is sent to the renderer?
Subclass CIFilter and do the check inside that custom class. There, we can override the outputImage property, which is of type CIImage. A CIImage is a Core Image object that represents an image but is not rendered until explicitly asked to. Therefore, we can check its extent.size before the output is sent to the renderer.
The custom class below is a working solution that prevents SpriteKit's renderer from crashing if the applied filter exceeds the renderer's limits. It is based on this answer, which was written to chain filters on the same SpriteKit SKEffectNode. X birds with one stone!
import Core Image
class ChainFilters: CIFilter {
let chainedFilters: [CIFilter]
let metalSizeLimit: CGFloat = 8000 /// TBD. Get the texture limit of the device's GPU family, and substract some safety margin
@objc dynamic var inputImage: CIImage?
init(filters: [CIFilter?]) {
/// The array of filters can contain a nil if the CIFilter inside it is given a wrong name or parameter
/// `compactMap { $0 }` filter out any `nil` values from the array
self.chainedFilters = filters.compactMap { $0 }
super.init()
}
/// Override `outputImage` to:
/// - Chain multiple filters
/// - Check the output result of each filter before it is passed on
override var outputImage: CIImage? {
get {
let imageKey = "inputImage"
var workingImage = self.inputImage
for filter in chainedFilters {
assert(filter.inputKeys.contains(imageKey))
filter.setValue(workingImage, forKey: imageKey)
guard let result = filter.outputImage else {
assertionFailure("Filter failed: \(filter.name)")
return nil
}
/// Start Metal limit test
/// We check the `extent` property of the working image, which is a `CIImage`
/// A CIImage is an object that represents an image but is not rendered until explicitly asked to
if (result.extent.size.width > metalSizeLimit || result.extent.size.height > metalSizeLimit) {
print("Input size = \(workingImage?.extent.size ?? CGSize.zero)")
print("Output size = \(result.extent.size)")
return workingImage
}
/// End Metal limit test
workingImage = result
}
/// Here the output image is passed on, ultimately to be rendered in SpriteKit or elsewhere
return workingImage
}
}
required init?(coder aDecoder: NSCoder) { fatalError("init(coder:) has not been implemented") }
}
Example usage in SpriteKit:
myEffectNode.filter = ChainFilters(filters: [
CIFilter(name: "CIZoomBlur", parameters: ["inputAmount": 140]), // will not render if the result exceeds the limit
CIFilter(name: "CIPixellate", parameters: ["inputScale": 8])
])
Improve the output checking inside the CIFilter subclass. For example, when the limit is exceeded, return a result with the value that did not make the filter exceed the limit, instead of returning the base unfiltered input image.