Motivation

Elastic APM Java agent does not support R2DBC. They provide a plugin API, which

lets you add custom instrumentation to the agent, which the agent will automatically apply the same way as it applies the internally defined instrumentation.

Once you create the plugin, you pack it as a normal JAR file and you need to tell the APM agent in which folder to search for extra plugins, by setting the config-plugins-dir configuration parameter.

We want to deploy the plugin as a part of the fat Spring Boot Executable JAR.

What works

It seems that the config-plugins-dir is evaluated only as a path within the host system. When we manually place the plugin JAR into any system folder visible to the application, it is properly recognized.

What doesn't work

We didn't manage to set the config-plugins-dir parameter to force the java-apm-agent to search for the plugins wherever inside the application JAR file.

More background

There are multiple way of setting up the APM agent, among others:

• Manual setup with -javaagent flag
• Programmatic API setup to self-attach using call to ElasticApmAttacher.attach(); before SpringApplication.run().

Doubts

I'd understand that the first method using -javaagent might not see the inside of the JAR file, but why doesn't it work when it is attached from the Java application itself? I would expect that once inside the running application, I should see the JAR content, at least the classpath???

I guess it might be somehow related to similar sounding Stackoverflow questions, such as

• Specifying a javaagent within the jar file to be run
• Classpath including JAR within a JAR
• Reference Jar within main Jar to pass as javaagent to JVM
• How to access set of resource files within a JAR in JAvaagent
• How to Package a Java Agent into the Host Project?

however I am not able to put all of it together neither to understand why it doesn't work, nor how to make it working.

Question

Is there any way to achieve our primary purpose, so we may deploy the APM agent plugin inside the application JAR? E.g. any clever Maven trick to achieve it? Or something else?

Finally we came to a solution, but Maven changes were not enough, we also needed to modify the Dockerfile. Here is our solution:

Summary

1. Maven maven-dependency-plugin copies the plugin from artifact repository to target/apm-plugins/ folder.
2. Docker copies it from target/apm-plugins/ to /app/apm-plugins/.
3. For local development, the plugins_dir variable points to the local file location target/apm-plugins.
4. In Kubernetes, the plugins_dir variable is overwritten by system variable ELASTIC_APM_PLUGINS_DIR.

Details

The code was simplified and anonymized, so it is possible that I omitted some small things :)

1. Maven pom.xml

<dependency>
    <groupId>com.mycompany.myproject</groupId>
    <artifactId>apm-plugins</artifactId>
</dependency>
...
<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-dependency-plugin</artifactId>
    <executions>
        <execution>
            <id>copy</id>
            <phase>compile</phase>
            <goals>
                <goal>copy</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <artifactItems>
            <artifactItem>
                <groupId>com.mycompany.myproject</groupId>
                <artifactId>apm-plugins</artifactId>
                <type>jar</type>
                <overWrite>true</overWrite>
                <outputDirectory>${project.build.directory}/apm-plugins</outputDirectory>
            </artifactItem>
        </artifactItems>
        <overWriteReleases>true</overWriteReleases>
        <overWriteSnapshots>true</overWriteSnapshots>
    </configuration>
</plugin>

2. Docker Dockerfile

COPY target/apm-plugins/*.jar /app/apm-plugins/

3. Local configuration elasticapm.properties:

This ensures that the plugin is found also when the application is built on in the local environment.

# https://www.elastic.co/guide/en/apm/agent/java/current/plugin-api.html
# A folder that contains external agent plugins.
# We have implemented our own plugin to support reactive R2DBC. More frameworks support can be added there later.
# Overridden by environment variable ELASTIC_APM_PLUGINS_DIR.
plugins_dir=target/apm-plugins

# This is to suppress the "Unexpected error during automatic discovery process for cloud provider" warning after start of application
cloud_provider=NONE

4. Kubernetes

values.yaml

env:
  ELASTIC_APM_PLUGINS_DIR: apm-plugins

deployment.yaml - this ensures that all from values.yaml under env is set as a system variable:

spec:
  template:
    spec:
      containers:
        - name: xxxx
          env:
            {{- range $key, $value := .Values.env }}
            - name: {{ $key }}
              value: {{ $value | quote }}
            {{- end }}