EventLoopFuture causing `Invalid state: unable to write message`

I'm using grpc-swift and the ClientInterceptor to add an auth token to requests. I have implemented the delegate method for send as follows

    override func send(_ part: GRPCClientRequestPart<Request>, promise: EventLoopPromise<()>?, context: ClientInterceptorContext<Request, Response>) {
        
        guard case .metadata(var headers) = part else {
            return context.send(part, promise: promise)
        }
        
        let p = context.eventLoop.makePromise(of: String.self)
        p.completeWithTask {
            "My Token" // In reality this is actually an async function like `await myActor.idToken`
        }
        p.futureResult.whenSuccess { token in
            print(context.eventLoop.inEventLoop) // This returns `true`
            headers.add(name: "Authorization", value: "Bearer \(token)")
            context.send(.metadata(headers), promise: promise)
        }
    }

I need to read the values from an actor which requires async/await to get the values hence the use of creating a promise and then p.completeWithTask.

The problem I am facing is when doing this and then calling the context.send(_,promise:) is that I receive an error back saying Invalid state: unable to write message, looking at the docs it says /// An invalid state was encountered. This is a serious implementation error., I'm clearly doing something very wrong here but for the life of me I am not sure what?

If this is used without promises/futures it all works fine. I'm not sure if I am using the p.completeWithTask & p.futureResult.whenSuccess correctly here?

Solution

I'm pretty sure you end up re-ordering messages. Usually, a gRPC request/response will be .metadata(...) followed by .message(...) and finally .end. And the order is important. If you disobey the correct order, you'll get into an illegal state.

So what could happen to you is the following:

you receive a .metadata(...) which you don't immediately forward because you asynchronously get the auth token
you receive the .message(...) which you immediately forward
(irrelevant because we're already in a bad state) you finish the async auth and forward the augmented .metadata(...)

See what's going on now? You might be forwarding .message before .metadata which is illegal and would get you into that state.

So how can you fix this? I think you need to manually buffer all further parts you receive until you managed to get the augmented .metadata(...) one out. That's not incredibly hard but also kinda annoying.

This issue is very similar to what you're doing btw: https://github.com/grpc/grpc-swift/issues/1181

Particularly this snippet from @glbrntt's response

You'll also have to make sure that within your interceptor you appropriately buffer any request parts to ensure that request parts are sent in the correct oder (i.e. you could receive a request message to send before the asynchronous code to fetch a token has completed).

I filed a number of bugs on gRPC Swift about this: