amazon-web-services amazon-elastic-beanstalk spring-boot-maven-plugin paketo

How to solve Memory issues with Paketo buildpack used to build a spring-boot app?

I am building Docker image with the spring-boot-maven-plugin that is deployed to AWS BeanStalk. I use the plugin through the 2.4.3 spring boot starter dependency) However, when the container is started, I get the error below. I am a bit new in the buildpack stuff, but tried to solve it by playing with the Buildpack env variables as described on the website. But it had absolutely no effect on the values shown in the error log below. I found this github issue but not sure if it's relevant and how to use it.

I am using AWS Micro instance that has 1G total RAM, it performs a rolling update, so at the time of starting the new image, the other is also running till the new one started with success, so to start the container could as well be that only 300MB is available, however, during normal run it has more available.

Why do I need this memory calculation? Can't I just disable it? When I build a Docker image of the app.jar and deploy it to aws beanstalk, it works well without any memory settings:

docker build . --build-arg JAR_FILE=./target/app.jar -t $APPLICATION_NAME

But I would love to use the image build through the spring-boot-maven plugin. Please some advice on how to solve this?

<plugin>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-maven-plugin</artifactId>
    <configuration>
        <image>
            <name>${image.name}</name>
            <env>
                <tag>${project.version}</tag>
                <!--BPE_APPEND_JAVA_TOOL_OPTIONS>-XX:MaxDirectMemorySize=1M</BPE_APPEND_JAVA_TOOL_OPTIONS-->
                <BPE_JAVA_TOOL_OPTIONS>-Xms1024m -Xmx3048m</BPE_JAVA_TOOL_OPTIONS>
            </env>
        </image>
    </configuration>
</plugin>

The AWS Beanstalk error during deployment:

Tue May 18 2021 18:07:14 GMT+0000 (UTC)   INFO   Successfully built aws_beanstalk/staging-app
Tue May 18 2021 18:07:22 GMT+0000 (UTC)   ERROR   Docker container quit unexpectedly after launch: 0M, -Xss1M * 250 threads
[31;1mERROR: [0mfailed to launch: exec.d: failed to execute exec.d file at path '/layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/memory-calculator': exit status 1. Check snapshot logs for details.
Tue May 18 2021 18:07:24 GMT+0000 (UTC)   ERROR   [Instance: i-0dc33dcb517e89ef9] Command failed on instance. Return code: 1 Output: (TRUNCATED)...pectedly after launch: 0M, -Xss1M * 250 threads
[31;1mERROR: [0mfailed to launch: exec.d: failed to execute exec.d file at path '/layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/memory-calculator': exit status 1. Check snapshot logs for details. 
Hook /opt/elasticbeanstalk/hooks/appdeploy/enact/00run.sh failed. For more detail, check /var/log/eb-activity.log using console or EB CLI.
Tue May 18 2021 18:07:24 GMT+0000 (UTC)   INFO   Command execution completed on all instances. Summary: [Successful: 0, Failed: 1].
Tue May 18 2021 18:07:24 GMT+0000 (UTC)   ERROR   Unsuccessful command execution on instance id(s) 'i-0dc33dcb517e89ef9'. Aborting the operation.
Tue May 18 2021 18:07:24 GMT+0000 (UTC)   ERROR   Failed to deploy application.
Tue May 18 2021 18:07:24 GMT+0000 (UTC)   ERROR   During an aborted deployment, some instances may have deployed the new application version. To ensure all instances are running the same version, re-deploy the appropriate application version.
##[error]Error: Error deploy application version to Elastic Beanstalk

The Docker error log downloaded in AWS Beanstalk:

Docker container quit unexpectedly on Tue May 18 18:07:21 UTC 2021:
Setting Active Processor Count to 1
Calculating JVM memory based on 274300K available memory
unable to calculate memory configuration
fixed memory regions require 662096K which is greater than 274300K available for allocation: -XX:MaxDirectMemorySize=10M, -XX:MaxMetaspaceSize=150096K, -XX:ReservedCodeCacheSize=240M, -Xss1M * 250 threads
[31;1mERROR: [0mfailed to launch: exec.d: failed to execute exec.d file at path '/layers/paketo-buildpacks_bellsoft-liberica/helper/exec.d/memory-calculator': exit status 1

Solution

OK, so here's what this is telling us:

Calculating JVM memory based on 274300K available memory

The memory calculator is detecting a maximum amount of memory available in the container as 274300KB, or about 274M.

fixed memory regions require 662096K which is greater than 274300K available for allocation: -XX:MaxDirectMemorySize=10M, -XX:MaxMetaspaceSize=150096K, -XX:ReservedCodeCacheSize=240M, -Xss1M * 250 threads

This message is saying that the memory calculator needs at least 662096KB or 662M in its present configuration.

It's also breaking down why it needs/wants that much:

10M for direct memory
150096K for metaspace
240M for reserved code cache
250M for threads (thread stack specifically)

That's not counting the heap which will require more (you seem to want at least 1G for the heap).

This leaves two possibilities:

The container is not provisioned large enough. You need to give it more memory.
The memory calculator is not correctly detecting the memory limit.

If you suspect #2, look at the following. The memory calculator selects it's max memory limit (i.e. the 274M in the example above) by looking in these places in this order.

Check the configured container memory limit by looking at /sys/fs/cgroup/memory/memory.limit_in_bytes inside the container.
Check the system's max available memory by looking at /proc/meminfo and the MemAvailable metric, again, from inside the container.
If all else fails, it'll end up with a 1G fallback.

If it's truly not working as described above, please open a bug and provide as much detail as you can.

Alternatively, you may tune the memory calculator. You can instruct it to give less memory to specific regions such that you reduce the total memory required to be less than the max available memory.

You can do that by setting the JVM memory flags in the JAVA_TOOL_OPTIONS env variable (you have BPE_JAVA_TOOL_OPTIONS which isn't right). See https://paketo.io/docs/buildpacks/language-family-buildpacks/java/#runtime-jvm-configuration.

For example, if you want to override the heap size then set -Xmx in JAVA_TOOL_OPTIONS to something custom. The memory calculator will see what you've set and adjust the remaining memory settings accordingly. Override as many as necessary.

To get things down to fit within 274M of RAM, you'd have to go really small. Something like -Xss256K -Xmx64M -XX:ReservedCodeCacheSize=64M -XX:MaxMetaspaceSize=64M. I didn't test to confirm, but this shows the idea of what you need to do. Reduce the memory settings such that the sum total all fits within your max memory limit for the container.

This also does not take into account if your application will actually be able to run within such small limits. If you go too small you may at some point see OutOfMemoryErrors or StackOverflowErrors, and your app will crash. You can also negatively impact performance by reducing code cache size too much since this is where the JIT stores byte code that it's optimized to native code. You could even cause GC issues, or degraded performance due to too much GC if the heap isn't sized right. In short, be very careful if you're going to do this.