I am trying to build complex Spring Integration flows where parts of the flows need to handle error scenarios and provide a different branches in the flow that will skip some processing and continue the flow further down.
For the flow as a whole I have an error channel configured on the incoming gateway and errors always go to this channel. When I change the error channel message header in the middle of the flow it is ignored and the errored message is just sent to the gateway error channel. I believe it is because it just propagates the exception up the stack to the gateway and doesn't check the messages headers.
So it seems to me that the error channel header is essentially useless to a flow developer and really is just for internal Spring Integration framework purposes.
So my question is how is it possible to build complex flows where the error handling requirements change through the flows (sometimes dynamically configured based on runtime config data, or message flow data) if the error channel message header doesn't do what you would expect it to do in a messaging system?
Right, the gateway contract is that an error is handled over there when exception is thrown back to this gateway. That is not set into a header, though. Rather the part of the pattern which is a return address as a TemporaryReplyChannel. The ErrorMessage could be sent into that header as you would send a regular reply message. The exception from the downstream flow is thrown as it would be with regular Java program.
Now, if you'd like to handle errors in the middle of the flow, then you need to look into different mechanism. Think about it as a try..catch in Java. For that purpose Spring Integration suggests an ExpressionEvaluatingRequestHandlerAdvice: https://docs.spring.io/spring-integration/reference/handler-advice/classes.html#expression-advice.
The custom errorChannel header makes sense only if you shift flow work into a different thread, e.g. via an ExecutorChannel or QueueChannel. If you flow is direct, then no any error channels are involved and exception is handled as it would be with regular Java methods call. So, if you don't try..catch exception at specific point, it is going to be bubbled back to the original caller. In your case that one is a messaging gateway. And that's where an error channel configured on the gateway is involved.
In summery, you are right: the errorChannel has only limited usage in the framework, but that's fully similar if you would write regular Java application and have some thread shifting in the middle. If you don't, then exception comes back to the called.