scenekitmetal

Sample a texture in a SCNGeometry geometry modifier


I am trying to get a RGB value in a SceneKit geometry modifier and use that value for positioning my geometry. Here is what I have in my shader.

#include <metal_stdlib>
#include <simd/simd.h>
#include <metal_texture>

#pragma arguments
texture2d terrainRGB;

#pragma transparent
#pragma body

constexpr sampler textureSampler(coord::normalized, address::clamp_to_edge, filter::nearest, mip_filter::nearest);
float4 elevation = terrainRGB.sample(textureSampler, _geometry.texcoords[kSCNTexcoordCount-1]);

float r = elevation.r * 256;
float g = elevation.g * 256;
float b = elevation.b * 256;

float rgbExpression = (r * 256.0 * 256.0) + (g * 256.0) + b;
float e = -10000.0 + (rgbExpression * 0.1);

_geometry.position.z = e;

Then I set up my geometry like this.

let geo = geo = SCNGeometry(...)
let texture = SCNMaterialProperty(contents: img)
geo.setValue(texture, forKeyPath: "terrainRGB")

If I encode my elevations into the geometry normals use the normals instead of a texture to get the elevations this code works. But I don't want to use the normals because that changes how light sources work with my scene.

What I can't figure out is what data type I should pass to geo.setValue() to work with texture2d in my shader. Or is there something else I should be doing with a geometry modified to pass a texture and be able to sample it in the modifier.


Solution

  • It turns out there were three things that I needed to do to get this work.

    1. I needed to pass a MTLTexture to SCNMaterialProperty instead of a UIImage.
    2. I needed to set the filters on the SCNMaterialProperty to match my filters on the sampler in my geometry modifier. (I am not sure if other combinations would work, this is just what I had to do with my case.)
    3. I had to convert my image to CGColorSpace.genericRGBLinear. My image in the sRGB color space. I am not exactly sure why I had to do this. I have a vertex shader set up to do the exact same thing with RealityKit and it works just fine with the sRGB image.

    Here is what my final code to set the texture on my SCNGeometry looks like.

    if let cgImg = image.cgImage,
       let colorSpace = CGColorSpace(name: CGColorSpace.genericRGBLinear),
       let textureImg = cgImg.copy(colorSpace: colorSpace),
       let d = self.sceneDevice { //an MTLDevice 
        
        do {
            
            let textureLoader = MTKTextureLoader(device: d)
            let texture = try textureLoader.newTexture(cgImage: textureImg)
            let scnTexture = SCNMaterialProperty(contents: texture)
            scnTexture.minificationFilter = .nearest
            scnTexture.magnificationFilter = .nearest
            scnTexture.wrapS = .clamp
            scnTexture.wrapT = .clamp
            geo.setValue(scnTexture, forKeyPath: "terrainRGB")
            
        } catch {
            print("Error: \(error)")
        }
    }