Context

versions in use

• kotlin: 2.0.20
• kotlinx-serialization: 2.0.20
• kotlinx-io: 0.5.4

I've been following the kotlinx-serialization Custom formats guide to implement a custom binary protocol and have run into a wall ...

The Decoder in question reads from a kotlinx-io Buffer and creates data class instances

The Issue

The decoder works for the most part except optional fields.

I'm trying to decode the Buffer into this type:

@Serializable
data class Foo(
    val one: Int,
    val two: Int,
    val three: Short,
    val four: Short = 10, // notice the default value
)

So the Buffer can sequentially contain either:

• 12 bytes / 4 fields

|int  |int  |short |short |
|bbbb |bbbb |bb    |bb    |

• or 10 bytes / 3 fields

|int  |int  |short |
|bbbb |bbbb |bb    |

The 12 bytes case is successfully decoded but the 10 bytes case fails with the following issue:

Buffer doesn't contain required number of bytes (size: 0, required: 2)
java.io.EOFException: Buffer doesn't contain required number of bytes (size: 0, required: 2)
    at kotlinx.io.Buffer.throwEof(Buffer.kt:163)
    at kotlinx.io.Buffer.readShort(Buffer.kt:103)
    at com.acme.corp.serde.BufferDecoder.decodeShort(BufferDecoder.kt:<line_number>)
    at kotlinx.serialization.encoding.AbstractDecoder.decodeShortElement(AbstractDecoder.kt:52)
    at com.acme.corp.serde.OptionalFieldTest$Foo$$serializer.deserialize(OptionalFieldTest.kt:<line_number>)

Testcase

class OptionalFieldTest {
    @Test
    fun `options fields default when not enough bytes are available`() {
        val buffer =
            Buffer().apply {
                writeInt(1)
                writeInt(2)
                writeShort(3)
                // notice missing writeShort() here
            }
        val decoder = BufferDecoder(buffer)
        val decoded = decoder.decodeSerializableValue(serializer(), null)
        println("decoded into value=$decoded")
    }
}

Decode Implementation

class BufferDecoder(
    private val buffer: Buffer
    private var elementsCount: Int = 0,
) : AbstractDecoder() {
    private var elementIndex = 0
    override val serializersModule: SerializersModule = EmptySerializersModule()

    override fun decodeBoolean(): Boolean = buffer.readByte().toInt() != 0

    override fun decodeByte(): Byte = buffer.readByte()

    override fun decodeShort(): Short = buffer.readShort()

    override fun decodeInt(): Int = buffer.readInt()

    override fun decodeLong(): Long = buffer.readLong()

    override fun decodeFloat(): Float = buffer.readFloat()

    override fun decodeDouble(): Double = buffer.readDouble()

    override fun decodeChar(): Char = buffer.readByte().toInt().toChar()

    override fun decodeString(): String = buffer.readString()

    override fun decodeEnum(enumDescriptor: SerialDescriptor): Int = decodeByte().toInt()

    override fun decodeSequentially(): Boolean = true

    override fun decodeElementIndex(descriptor: SerialDescriptor): Int =
        if (elementIndex == elementsCount) {
            CompositeDecoder.DECODE_DONE
        } else {
            elementIndex++
        }

    override fun beginStructure(descriptor: SerialDescriptor): CompositeDecoder =
        BufferDecoder(
            buffer = buffer,
            elementsCount = descriptor.elementsCount,
        )
}

Question

I'm not sure how to tell kotlinx-serialization to use the default value for the last/remaining fields if the Buffer has been exhuasted

You need to do a couple of things.

First, at the moment the decodeElementIndex function is not being called at all because decodeSequentially returns true. With this true, the decoder will be called for every expected field of Foo (ie four times) regardless of anything in decodeElementIndex, which is not called at all. (This is the sequential decoding protocol.)

So let's bring decodeElementIndex into play.

override fun decodeSequentially(): Boolean = false

Now as written in the question, decodeElementIndex will cause deserialization to stop when elementIndex equals elementsCount—ie when decoding function calls have been made for the full number of expected fields for Foo (namely four). But we also want it to stop when the buffer is empty, so let's modify it like this:

override fun decodeElementIndex(descriptor: SerialDescriptor): Int =
    when {
        buffer.exhausted() -> CompositeDecoder.DECODE_DONE
        elementIndex == descriptor.elementsCount -> CompositeDecoder.DECODE_DONE
        else -> elementIndex++
    }

Now the decoding should stop in the short buffer case and Foo will decode in both the 10 and 12 byte cases.

Note of course this does not consider error handling when a buffer is provided outside of these cases; you may need to consider an approach for this.