javascalasbtaws-sdksbt-assembly

Missing AWS SDK execution interceptors in ExecutionInterceptorChain in Scala fat jar


I'm fairly new to scala / java / sbt and this is my first time debugging a discrepancy between code ran in 'scala' through sbt and the same code ran from a compiled jar. I have some Scala code which utilizes the AWS Java SDK to make some S3 requests. When I run my code in "Scala" using sbt or Intellij the code succeeds as intended. If I build a fat jar using sbt and run the same code in "Java" using the jar, the code fails. I debugged the failure and can attribute it to missing ExecutionInterceptors in the interceptor chain:

// Logging the code ran from 'scala'
2022-12-21 14:09:33,861 [main] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain -  Creating an interceptor chain that will apply interceptors in the following order: 
[software.amazon.awssdk.core.internal.interceptor.HttpChecksumRequiredInterceptor@5965844d, 
software.amazon.awssdk.core.internal.interceptor.SyncHttpChecksumInTrailerInterceptor@6d4a65c6, 
software.amazon.awssdk.core.internal.interceptor.HttpChecksumValidationInterceptor@aa004a0, 
software.amazon.awssdk.core.internal.interceptor.AsyncRequestBodyHttpChecksumTrailerInterceptor@4c98a6d5, 
software.amazon.awssdk.core.internal.interceptor.HttpChecksumInHeaderInterceptor@392a04e7, 
software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@7f02251, 
software.amazon.awssdk.awscore.eventstream.EventStreamInitialRequestInterceptor@dffa30b, 
software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor@4d8126f, 
software.amazon.awssdk.services.s3.internal.handlers.EnableChunkedEncodingInterceptor@6d3c232f, 
software.amazon.awssdk.services.s3.internal.handlers.DisableDoubleUrlEncodingInterceptor@6b587673, 
software.amazon.awssdk.services.s3.internal.handlers.EnableTrailingChecksumInterceptor@1bcf67e8, 
software.amazon.awssdk.services.s3.internal.handlers.CreateMultipartUploadRequestInterceptor@5f404594, 
software.amazon.awssdk.services.s3.internal.handlers.GetObjectInterceptor@53692008, 
software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor@7b2a3ff8, 
software.amazon.awssdk.transfer.s3.internal.ApplyUserAgentInterceptor@1bbae752, 
software.amazon.awssdk.services.s3.internal.handlers.AsyncChecksumValidationInterceptor@460b6d54, 
software.amazon.awssdk.services.s3.internal.handlers.EndpointAddressInterceptor@5cf87cfd, 
software.amazon.awssdk.services.s3.internal.handlers.SignerOverrideInterceptor@76075d65, 
software.amazon.awssdk.services.s3.internal.handlers.GetBucketPolicyInterceptor@3a4ba480, 
software.amazon.awssdk.services.s3.internal.handlers.PutObjectInterceptor@27b71f50, 
software.amazon.awssdk.services.s3.internal.handlers.SyncChecksumValidationInterceptor@383790cf, 
software.amazon.awssdk.services.s3.internal.handlers.DecodeUrlEncodedResponseInterceptor@74971ed9, 
software.amazon.awssdk.services.s3.internal.handlers.CreateBucketInterceptor@131fcb6f, 
software.amazon.awssdk.services.s3.internal.handlers.CopySourceInterceptor@ccd1bc3]
// Logging the same code ran from 'java'
2022-12-21 14:15:06,281 [main] DEBUG software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain -  Creating an interceptor chain that will apply interceptors in the following order: 
[software.amazon.awssdk.core.internal.interceptor.HttpChecksumRequiredInterceptor@147e2ae7, 
software.amazon.awssdk.core.internal.interceptor.SyncHttpChecksumInTrailerInterceptor@448c8166, 
oftware.amazon.awssdk.core.internal.interceptor.HttpChecksumValidationInterceptor@4470fbd6, 
software.amazon.awssdk.core.internal.interceptor.AsyncRequestBodyHttpChecksumTrailerInterceptor@15d49048, 
software.amazon.awssdk.core.internal.interceptor.HttpChecksumInHeaderInterceptor@7098b907, 
software.amazon.awssdk.awscore.interceptor.HelpfulUnknownHostExceptionInterceptor@503f91c3, 
software.amazon.awssdk.awscore.eventstream.EventStreamInitialRequestInterceptor@13526e59, 
software.amazon.awssdk.awscore.interceptor.TraceIdExecutionInterceptor@2235eaab, 
software.amazon.awssdk.transfer.s3.internal.ApplyUserAgentInterceptor@17503f6b]

All of the interceptors of the type software.amazon.awssdk.services.s3.internal.handlers are missing from the code ran in the jar.

I'm not sure if this is directly related. My build.sbt has the following merge strategies:

  assembly / assemblyMergeStrategy := {
    // Don't discard Geotools META-INF service files
    case PathList("META-INF", xs @ _*) =>
      (xs map { _.toLowerCase }) match {
        case ("manifest.mf" :: Nil) | ("index.list" :: Nil) | ("dependencies" :: Nil) =>
          MergeStrategy.discard
        case ps @ (x :: xs) if ps.last.endsWith(".sf") || ps.last.endsWith(".dsa") =>
          MergeStrategy.discard
        case "plexus" :: xs =>
          MergeStrategy.discard
        case "services" :: xs =>
          MergeStrategy.filterDistinctLines
        case ("spring.schemas" :: Nil) | ("spring.handlers" :: Nil) =>
          MergeStrategy.filterDistinctLines
        case _ => MergeStrategy.discard
      }
    case PathList("reference.conf", xs @ _*) => MergeStrategy.concat
    case PathList("version.conf", xs @ _*)   => MergeStrategy.concat
    case x                                   => MergeStrategy.first
  },

When debugging compiling my jar with sbt assembly I noticed this line: [debug] Merging 'software/amazon/awssdk/services/s3/execution.interceptors' with strategy 'first' Is this related to why i'm missing the interceptors? or might there be some other cause for missing these handlers, such as a missing dependency?


Solution

  • I solved this question, the issue was that the aws s3 interceptors were getting merged out, I added this line to the merge strategy and it solved the problem:

    case PathList(
            "software",
            "amazon",
            "awssdk",
            "services",
            "s3",
            "execution.interceptors",
            xs @ _*
            ) =>
          MergeStrategy.concat