I am developing a shader to render an image with a film frame effect. I've crafted the following shader:
extern "C" {
namespace coreimage {
float4 reassemble(sampler source, sampler frame, float imgWidth, float perFrameSize, float epsilon, destination dest) {
float2 coord = dest.coord();
float4 srcExtent = source.extent();
float4 pixel = float4(0, 1, 0, 1);
float sideFrameSize = epsilon + (imgWidth - srcExtent.z)/2 - perFrameSize;
float offset = sideFrameSize + perFrameSize - epsilon;
if (coord.x < sideFrameSize) {
pixel = source.sample(source.transform(float2(coord.x + srcExtent.x + srcExtent.z - sideFrameSize, coord.y)));
}
else if (coord.x >= offset && coord.x <= srcExtent.z + offset) {
pixel = source.sample(source.transform(float2(coord.x + srcExtent.x - offset, coord.y)));
}
else if (coord.x >= srcExtent.z + offset + perFrameSize - epsilon) {
pixel = source.sample(source.transform(float2(coord.x + srcExtent.x - srcExtent.z - offset - perFrameSize + epsilon, coord.y)));
}
float4 framePixel = frame.sample(frame.transform(coord));
return mix(pixel, framePixel, framePixel.a);
}
}
}
It operates as expected, and produce the following image.
The issue arises when attempting to integrate cropping into a pipeline. When I add the crop before calling a kernel function, everything functions as expected. However, if I crop the output image from the kernel function, it appears that the source image is first cropped, and then the shader runs on the cropped image.
if let source = self.kernel.apply(extent: frameImage.extent,
roiCallback: { _, rect -> CGRect in
return rect
},
arguments: [scaled.transformed(by: .init(translationX: 0, y: perFrameWidth - vOffset)), frameImage, image.extent.width, perFrameWidth, hOffset]) {
return source.cropped(to: frameImage.extent.insetBy(dx: 80, dy: 200))
}
The resulting image generated by the code above.
While this could be an optimization, it's rather unexpected. How can I modify this behavior to ensure that the output image from the shader is cropped, rather than the original? Your insights would be greatly appreciated.
I would like to explain the issue in more detail. The first argument of the shader is the image, and it's important to note that I don't crop it directly. Instead, the cropping occurs on the output image. Strangely, the image I pass to the shader seems to be cropped as well. What's more, I try to crop after the calling shader, and this is strange that the shader receives a cropped image.
As you guessed correctly, Core Image performs an optimization here: It concatenates the different filters and operations you perform on an image into as few processing operations as possible. Specifically for cropping, it tries to only process the pixels that will be visible in the final result. This is outlined in the (old and outdated) Core Image Programming Guide.
To do that, it analyzes the filter graph from the back to the front to figure out what part of an input image is required to compute the desired output image. This is called the region of interest (ROI) and it usually depends on the region of the output that should be produced (domain of definition, DOD). This is also described here.
As a filter developer, it's your responsibility to tell Core Image what region of the input image (ROI) your kernel required to read in order to produce the given output region. This is what the roiCallback parameter in the kernel call is used for: For the given input index (in case your kernel takes multiple input images) and the given output region (DOD), you have to return the ROI for that input image.
In your previous code, you told Core Image that your kernel performs a 1:1 mapping, i.e., that for a given output region you only need to read the same region from the input image. That means Core Image also only loaded that region from the input and passed it to your kernel.
However, your kernel actually samples the input image at locations different from the output coord, so the provided input region was not enough.
If you instead write your roiCallback as follows, you are telling Core Image that you always require the whole input image, regardless of what region of the output to produce. This way, your kernel always has all pixels of the input image available for sampling.
let inputExtent = frameImage.extent // best do that outside of the block to not capture the input image inside the ROI callback
let roiCallback = { _, _ in
return inputExtent
}
Please note that this approach is also not ideal, since Core Image now can't optimize anymore. It always needs to load the whole image and potentially process it with any previous filters in the pipeline, regardless of how small the actual output would be. Ideally, you would figure out what ROI your filter really needs for a given DOD and return that in your roiCallback.