CIContext options for HDR display

I have two CIContexts configured with the following options:

let options1:[CIContextOption:Any] = [CIContextOption.cacheIntermediates: false, CIContextOption.outputColorSpace: NSNull(), CIContextOption.workingColorSpace: NSNull()];

let options2:[CIContextOption:Any] = [CIContextOption.cacheIntermediates: false];

And an MTKView with CAMetalLayer configured with HDR output.

    metalLayer = self.layer as? CAMetalLayer
    
    metalLayer?.wantsExtendedDynamicRangeContent = true
    metalLayer.colorspace = CGColorSpace(name: CGColorSpace.itur_2100_HLG)
    
    colorPixelFormat = .bgr10a2Unorm

The two context options produce different outputs when input is in BT.2020 pixel buffers. But I believe the outputs shouldn't be different. Because the first option simply disables color management. The second one performs intermediate buffer calculations in sRGB extended linear color space and then converts those buffers to BT.2020 color space in the output. Is the output different because of loss of information in the intermediate step?

Finally, which of the options is correct in displaying HDR sample buffers?

Solution

Let me answer your last question first: Yes, this is a valid and good way to set up the MTKView for displaying HDR content. But, as you pointed out, you also need to tell the MTLView to tone-map the EDR values to the display's brightness range by setting metalLayer.edrMetadata = .hlg.

Alternatively, I found that the following setup also works for displaying EDR content:

metalLayer.wantsExtendedDynamicRangeContent = true
metalLayer.colorspace = CGColorSpace(name: CGColorSpace.extendedLinearSRGB)
colorPixelFormat = .rgba16Float

It seems the view knows how to tone-map extended linear EDR values to its display automatically in this case – no need for setting edrMetadata.

The rest I would like to split into two topics:

Working Color Space

You are correct: When setting CIContextOption.workingColorSpace: NSNull(), you effectively disable Core Image's automatic color management. However, I would only advise doing that when the following is the case:

You know what the color space of the input is,
you don't apply any built-in filters (as they assume their input to be in CI's default working color space), and
you configured the output to be in the same color space as the input.

So in your case, if you never want to apply filters in your pipeline, you are probably ok disabling the color matching by setting the working space to NSNull(). This would even save a bit of processing time, though not very much, as color space conversion is rather inexpensive.

In most scenarios, I would recommend not touching the workingColorSpace option and letting CI decide what to use.

Output Color Space

Unfortunately, the CIContextOption.outputColorSpace is very misleading. It doesn't seem to be used in almost all use cases – at least I haven't seen any effects so far.

That's because usually you either

pass the output color space to a render API of CIContext directly, like in createCGImage(_ image:, from fromRect:, format:, colorSpace:),
or the render destination defines the color space.

And the second point is the crux here. I believe you are rendering into the view using the CIRenderDestination API, probably defined like this:

let destination = CIRenderDestination(width: Int(drawableSize.width),
                                      height: Int(drawableSize.height),
                                      pixelFormat: self.colorPixelFormat,
                                      commandBuffer: commandBuffer,
                                      mtlTextureProvider: { () -> MTLTexture in
                                          return currentDrawable.texture
                                      })

The problem here is that a CIRenderDestination can't fully infer the color space of the target on its own (it usually defaults to sRGB). That's why you have to set it explicitly.

And here you know that the view is expecting the drawable texture to contain pixel data in itur_2100_HLG. So either set that directly, or simply set it to the view's color space:

destination.colorSpace = metalLayer.colorspace

Then Core Image will know to which color space to convert to when rendering into the destination.

With this change, there should now be no more difference between color management turned off (CIContextOption.workingColorSpace: NSNull()) and on.