So I seem to be having an issue related to the caching of DNS host name resolution when using PoolingHttpClientConnectionManager
I have an API that makes calls out to an outside service. This service uses Akamai edge caching so it's possible for the IP address to change. This api is under heavy load.
I have 100 connections max set for the pool size with an idle timeout thread closing idle connections after 30 seconds being idle. But with the heavy traffic, I don't think connections are ever idle. So it seems that if the IP address changes for this services host, my app never picks up that change since all connections in the pool are holding on to the old DNS resolved IP address.
I've tested this hypothesis locally by changing my /etc/hosts file on my mac (and flushing the OS cache). As long as there is steady traffic against my API app, it never picks up the change even with a TTL of 5 seconds set in java.security. I let it run for 20 minutes after changing the hostname to point to an invalid IP address that would cause calls to fail, and it never failed.
I stopped the JMeter traffic I was sending it, waiting about 30 seconds and hit it manually, and the call failed, telling me it picked up the change.
Any ideas? Do I need to stop using the connection pool? Or write my own DnsResolver (not sure how it should work)? How can I ensure that the connections in my connection pool will update if the DNS entry changes for this service?
So I somehow overlooked a constructor for PoolingHttpClientConnectionManager that takes a TimeToLive value. This seems to set an expiration time for each created connection. I tested it out and it definitely picks up the change in my local hosts file after this TTL value. I just don't know if this is the right way to solve my problem or not. If I don't hear about other options, I will assume this is the correct answer to my question.