As you can see the Apache httpclient 3.x had an amazing method that you could use to reduce the amount of unnecessary connections and data that is send around between client and server:
client.getParams().setAuthenticationPreemptive(true);
http://hc.apache.org/httpclient-3.x/authentication.html#Preemptive_Authentication
What I want to know is, why did they remove it?
I know there are proper workarounds like using an interceptor or modifying the header, but I am wondering about the reasons behind that change: stability, performance, security, conformity? ...
Preemptive authentication as a simple boolean flag is a good example of how features got added to HC 3.x without long term considerations until the 3.x codeline became a completely unmanageable mess.
Prevention of accidental sensitive information disclosures was the main reason for changing the way preemptive authentication works in HC 4.x. HC 3.x made it exceptionally easy for people to submit their credentials to random sites in clear text without even realizing that.
As of 4.1 HttpClient employs a much saner strategy by default: authentication credentials get cached in the execution context after an explicit authentication challenge and a successful authentication. All subsequent requests in the same session are authenticated preemptively using cached authentication material. One can still force preemptive authentication of the initial request by pre-populating the auth cache if necessary. But at the very least that requires that the user explicitly provides auth material for a specific auth target.