Metal fragment shader -- are the color values already quantized to 8 bits?

I'm trying to implement some simple dithering in my Metal fragment shader, to get rid of banding across gradients. It is not working, and I am wondering if it is just a bug or if the color values passed to the shader are already quantized (if that's the right word) to 8 bits. In other words, the fragment shader deals with floating point values, but are those already at the discrete levels imposed by 8 bit rgb space?

Here is my shader and the matrix I'm using for dithering. I based this on these two articles/posts:

OpenGL gradient "banding" artifacts

http://www.anisopteragames.com/how-to-fix-color-banding-with-dithering/

var dither_pattern:[UInt8] = 
   [0, 32,  8, 40,  2, 34, 10, 42,   /* 8x8 Bayer ordered dithering  */
    48, 16, 56, 24, 50, 18, 58, 26,  /* pattern.  Each input pixel   */
    12, 44,  4, 36, 14, 46,  6, 38,  /* is scaled to the 0..63 range */
    60, 28, 52, 20, 62, 30, 54, 22,  /* before looking in this table */
    3, 35, 11, 43,  1, 33,  9, 41,   /* to determine the action.     */
    51, 19, 59, 27, 49, 17, 57, 25,
    15, 47,  7, 39, 13, 45,  5, 37,
    63, 31, 55, 23, 61, 29, 53, 21]

// this array is passed to the frag shader via a MTLBuffer

fragment float4 window_gradient_fragment(WindowGradientVertexOut interpolated [[stage_in]],
   const device unsigned char* pattern [[ buffer(0) ]]) {

    int x = (int)interpolated.position.x % 8;
    int y = (int)interpolated.position.y % 8;
    int val = pattern[x+y*8];

    float bayer = 255.0 * (float)val / 64.0;
    const float rgbByteMax = 255.0;
    float4 rgba = rgbByteMax*interpolated.color;
    float4 head = floor(rgba);
    float4 tail = rgba-head;
    float4 color = head+step(bayer,tail);

    return color/255.0;
}

I tested this by changing the matrix to a series of alternating 0,63 pairs, and up close it did produce faint vertical stripes every other pixel. But the overall banding stays the same, from what I can tell. This has me thinking that the banding is already "built-in" by the time it reaches the fragment shader. So applying the dither to it wouldn't really help, because the damage has already been done. I hope that's not true though ... it was my naive assumption that because the colors are floats, they would have full floating point precision and the conversion to 8 bit would happen after the frag shader.

Solution

Your logic seems wrong. tail is in [0.0,1.0) by construction. bayer is in [0.0,255.0), more or less. step() is comparing the two. For all values of val >= 1, bayer is >= 3.98, and thus greater than tail. So, it's not very "dithery"; only a val of 0 rounds the color down, all others round up.

I think you just want to not multiply by 255.0 when calculating bayer.