How to use SSL with AppDynamics RabbitMQ Monitoring Plugin

I am trying to setup the rabbitmq machine agent for AppDynamics with a standalone RabbitMQ. https://www.appdynamics.com/community/exchange/extension/rabbitmq-monitoring-extension/

RabbitMQ 3.6.15
AppDynamics Controller 4.5.14.2417
AppD MachineAgent 4.5.14.2283
AppD Rabb

itMQ Monitoring Plugin 2.0.2

A curl on the RabbitMQ API works fine

curl -i -k -u user:pass https://127.0.0.1:15672/api/vhosts
HTTP/1.1 200 OK

Celery Flower is talking to rabbit fine with the following config options

--ca_certs=/opt/ca/cacert.pem --broker_api=https://user:pass@127.0.0.1:15672/api/

My rabbitMQ Monitoring plugin is configured like so in config.yml

servers:

   - host: "127.0.0.1"
     port: 15672
     useSSL: true
     username: "user"
     password: "pass"
     encryptedPassword: ""
     displayName : "RabbitMQ"

connection:
  socketTimeout: 10000
  connectTimeout: 10000

In my troubleshooting, I followed this guide to add /opt/ca/cacert.pem to a Java keystore. https://github.com/MichalHecko/SSLPoke

I initialize the machine agent as follows

java -Djavax.net.ssl.trustStore=/opt/trustStore.keystore -Djavax.net.ssl.trustStorePassword=password -jar /opt/machineagent-bundle-64bit-linux-4.5.14.2293/machineagent.jar

I am still getting the following error for every RabbitMQ api call by the monitor in machineagent-bundle-64bit-linux-4.5.14.2293/logs/machine-agent.log

[Monitor-Task-Thread6] 02 Oct 2019 20:02:37,671 DEBUG OptionalMetricsCollector - MetircsCollector Phaser arrived for RabbitMQ
[Monitor-Task-Thread4] 02 Oct 2019 20:02:37,671 DEBUG UrlBuilder - The url is initialized to https://127.0.0.1:15672/api/queues
[Monitor-Task-Thread4] 02 Oct 2019 20:02:37,672  INFO MetricsCollector - Fetching the RabbitMQ Stats for stat child: Queues from the URL {}https://127.0.0.1:15672/api/queues
[Monitor-Task-Thread4] 02 Oct 2019 20:02:37,672 DEBUG HttpClientUtils - Invoking the URL [https://127.0.0.1:15672/api/queues]
[Monitor-Task-Thread4] 02 Oct 2019 20:02:37,672 DEBUG Http4ClientBuilder - Added the BasicScheme to uri [https://127.0.0.1:15672]
[Monitor-Task-Thread4] 02 Oct 2019 20:02:37,679 ERROR HttpClientUtils - The response is null for the URL https://127.0.0.1e:15672/api/queues
[Monitor-Task-Thread4] 02 Oct 2019 20:02:37,679 ERROR HttpClientUtils - Exception while executing the request [https://127.0.0.1:15672/api/queues]
javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1946)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:316)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:310)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1639)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:223)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:1037)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:965)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1064)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1367)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1395)
        at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:1379)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(SSLConnectionSocketFactory.java:396)
        at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:355)
        at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142)
        at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373)
        at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:394)
        at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237)
        at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185)
        at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89)
        at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110)
        at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83)
        at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:108)
        at com.appdynamics.extensions.http.HttpClientUtils.getResponse(HttpClientUtils.java:119)
        at com.appdynamics.extensions.http.HttpClientUtils.getResponseAsJson(HttpClientUtils.java:68)
        at com.appdynamics.extensions.http.HttpClientUtils.getResponseAsJson(HttpClientUtils.java:45)
        at com.appdynamics.extensions.rabbitmq.metrics.MetricsCollector.run(MetricsCollector.java:103)
        at com.appdynamics.extensions.MonitorThreadPoolExecutor$2.run(MonitorThreadPoolExecutor.java:102)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)
Caused by: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:397)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:302)
        at sun.security.validator.Validator.validate(Validator.java:262)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:330)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:237)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:132)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1621)
        ... 27 more
Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
        at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141)
        at sun.security.provider.certpath.SunCertPathBuilder.engineBuild(SunCertPathBuilder.java:126)
        at java.security.cert.CertPathBuilder.build(CertPathBuilder.java:280)
        at sun.security.validator.PKIXValidator.doBuild(PKIXValidator.java:392)
        ... 33 more

What am I missing? Thank you!

Solution

The most important thing to understand about this error is the meaning of this line:

Caused by: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

SSL certificates work by establishing a certificate chain, or a hierarchy of trust. For example, if I go to https://www.google.com and look at their cert, this is what I see:

There is the google cert, which sits on their servers/CDN, then an intermediate cert which also sits on their servers/CDN, then a trusted root CA cert which is in the client keystore and is implicitly trusted. So when someone browses to google, b/c they have the root CA cert and have trusted it, the browser (client) will trust that the server is actually who they say they are and will establish a secure connection to the site.

So getting back to your error, whatever CA issued the server cert being used by RabbitMQ, the monitor is not recognizing it as trusted. To troubleshoot this error, here are the things to do:

Look at the server cert and ensure it can be validated. openssl works well for this; run: openssl s_client -connect 127.0.0.1:15672 -showcerts and look at the cert chain.
Validate the root CA is trusted by your java keystore. You can view these certs with the keystore tool: keytool -list -v -keystore </path/to/keystore> -storepass <pass>. Ensure the cert listed above is in the keystore.

There are a couple other gotchas to watch out for:

What keystore java is using is not always obvious. The jdk has a default keystore, and each app can use its own keystore, like you are doing above. Ensure you know what keystore is being used. Although it will add lots of logging, is can be helpful to add -Djavax.net.debug=all to the command line.
Beware adding individual server certs to the keystore. This will work, until the server cert expires. Much better to depend on trusted CA certs, which are generally maintained at the platform level. Adding individual certs is generally considered an anti-pattern.