amazon-s3amazon-ec2s3fs

s3fs: How to mount S3 buckets on AWS EC2 instances behind proxy, using IMDS v2


We are having some trouble to mount an AWS S3 bucket (using s3fs v1.90) into an AWS EC2 instance which:

The HTTP response code returned by the curl lib is "417 - Expectation Failed" (more details below). I found some hints on the www that the 417 error might relate to our proxy config, see:
HTTP POST Returns Error: 417 "Expectation Failed."
https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z0000019LuWSAU

This makes me believe that our NO_PROXY config is not being picked up by s3fs, but I'm really not sure...

Anyway, this is what we've tried to do in order to mount the bucket:

sudo s3fs SOME_BUCKET ./mnt-s3/ -o iam_role=SOME_ROLE,url=https://s3.eu-central-1.amazonaws.com,endpoint=eu-central-1,allow_other,uid=1000,gid=1000,mp_umask=007,use_cache=/tmp/s3foldercache,dbglevel=debug -f

This is the output:


2021-09-08T12:36:27.681Z [INF] curl.cpp:CheckIAMCredentialUpdate(1826): IAM Access Token refreshing...
2021-09-08T12:36:27.681Z [INF]       curl.cpp:GetIAMCredentials(3068): [IAM role=SOME_ROLE]
2021-09-08T12:36:27.681Z [DBG] curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 31
2021-09-08T12:36:27.681Z [DBG] curl.cpp:RequestPerform(2509): connecting to URL http://169.254.169.254/latest/api/token
2021-09-08T12:36:27.682Z [ERR] curl.cpp:RequestPerform(2622): HTTP response code 417, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
         "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>417 - Expectation Failed</title>
 </head>
 <body>
  <h1>417 - Expectation Failed</h1>
 </body>
</html>
2021-09-08T12:36:27.682Z [ERR] curl.cpp:GetIAMCredentials(3105): AWS IMDSv2 token retrieval failed: -5
2021-09-08T12:36:27.682Z [DBG] curl.cpp:RequestPerform(2509): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/SOME_ROLE
2021-09-08T12:36:27.684Z [ERR] curl.cpp:RequestPerform(2622): HTTP response code 401, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
        "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
 <head>
  <title>401 - Unauthorized</title>
 </head>
 <body>
  <h1>401 - Unauthorized</h1>
 </body>
</html>
2021-09-08T12:36:27.684Z [ERR] curl.cpp:CheckIAMCredentialUpdate(1830): IAM Access Token refresh failed
2021-09-08T12:36:27.684Z [DBG] curl_handlerpool.cpp:ReturnHandler(103): Return handler to pool
2021-09-08T12:36:27.684Z [INF] curl_handlerpool.cpp:ReturnHandler(110): Pool full: destroy the oldest handler
2021-09-08T12:36:27.685Z [CRT] s3fs.cpp:s3fs_check_service(3520): Failed to check IAM role name(SOME_ROLE).
2021-09-08T12:36:27.685Z [ERR] s3fs.cpp:s3fs_exit_fuseloop(3372): Exiting FUSE event loop due to errors

When running curl directly, though, we do receive a valid IMDS v2 token:

$ curl -v -X PUT -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" http://169.254.169.254/latest/api/token 

*   Trying 169.254.169.254...
* TCP_NODELAY set
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> PUT /latest/api/token HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.58.0
> Accept: */*
> X-aws-ec2-metadata-token-ttl-seconds: 21600
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Length: 56
< Content-Type: text/plain
< Date: Wed, 08 Sep 2021 13:14:02 GMT
< X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
< Connection: close
< Server: EC2ws
<
* Closing connection 0
SOME_TOKEN

Finally, this is our proxy config (defined by environment variables):

$ echo $HTTP_PROXY                                                                                                                                  
<SOME_HOST>:<SOME_PORT>

$ echo $NO_PROXY                                                                                                                                  
169.254.169.254,*.eu-central-1.amazonaws.com

So, my best guess is that s3fs might be ignoring the NO_PROXY variable, trying to use our proxy when asking local IP 169.254.169.254 for a new token.


Solution

  • A fix for this is being worked on in https://github.com/s3fs-fuse/s3fs-fuse/pull/1766