[SOLVED] Apache Camel 4.7: disableStreamCache=true no longer working on http endpoint as expected

Apache Camel 4.7: disableStreamCache=true no longer working on http endpoint as expected

We upgraded from Apache Camel 3.16.0 to 4.7.0 (including upgrade Java 11 to 21). We must stream large HTTP responses without caching, but this no longer works in the newer version as expected. We have this route:

from(DIRECT_GETITEM)
   .process(collectionCheckingPdp)
   .setHeader(Exchange.HTTP_METHOD, constant(HTTP_GET))
   .removeHeader(Exchange.HTTP_URI)
   .process(new StorageManagerPathProcessor())
   .to("http://localhost:9099?disableStreamCache=true&headerFilterStrategy=#oldaHttpFilter");

In Camel version 3.16.0, the body of the resulting exchange message was of type org.apache.http.conn.EofSensorInputStream. But in Camel 4.7.0, the exchange message body is a byte[] array, while we would still expect a stream. What can we do to avoid reading the response in memory and obtain the same behavior we had in the old version?

Since we are using Spring Boot, I also tried this setting in application.properties, but without any effect:

camel.component.http.response-payload-streaming-threshold=-1

Solution

The Camel HTTP producer should set the message body to the response input stream when disableStreamCache=true, but a bug caused it to read the response body into memory instead. The bug was fixed in Apache Camel 4.8.0.

But that's not the whole story. Stream caching is a separate Camel architecture feature applied after the Camel HTTP producer emitted the message, and is enabled by default. If the message body is an input stream, Camel converts it to a stream cache. To disable stream caching for the entire Camel context, set application property:

camel.main.stream-caching-enabled=false