We are having some trouble to mount an AWS S3 bucket (using s3fs v1.90) into an AWS EC2 instance which:
The HTTP response code returned by the curl lib is "417 - Expectation Failed" (more details below). I found some hints on the www that the 417 error might relate to our proxy config, see:
HTTP POST Returns Error: 417 "Expectation Failed."
https://knowledge.ni.com/KnowledgeArticleDetails?id=kA00Z0000019LuWSAU
This makes me believe that our NO_PROXY config is not being picked up by s3fs, but I'm really not sure...
Anyway, this is what we've tried to do in order to mount the bucket:
sudo s3fs SOME_BUCKET ./mnt-s3/ -o iam_role=SOME_ROLE,url=https://s3.eu-central-1.amazonaws.com,endpoint=eu-central-1,allow_other,uid=1000,gid=1000,mp_umask=007,use_cache=/tmp/s3foldercache,dbglevel=debug -f
This is the output:
2021-09-08T12:36:27.681Z [INF] curl.cpp:CheckIAMCredentialUpdate(1826): IAM Access Token refreshing...
2021-09-08T12:36:27.681Z [INF] curl.cpp:GetIAMCredentials(3068): [IAM role=SOME_ROLE]
2021-09-08T12:36:27.681Z [DBG] curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 31
2021-09-08T12:36:27.681Z [DBG] curl.cpp:RequestPerform(2509): connecting to URL http://169.254.169.254/latest/api/token
2021-09-08T12:36:27.682Z [ERR] curl.cpp:RequestPerform(2622): HTTP response code 417, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>417 - Expectation Failed</title>
</head>
<body>
<h1>417 - Expectation Failed</h1>
</body>
</html>
2021-09-08T12:36:27.682Z [ERR] curl.cpp:GetIAMCredentials(3105): AWS IMDSv2 token retrieval failed: -5
2021-09-08T12:36:27.682Z [DBG] curl.cpp:RequestPerform(2509): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/SOME_ROLE
2021-09-08T12:36:27.684Z [ERR] curl.cpp:RequestPerform(2622): HTTP response code 401, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>401 - Unauthorized</title>
</head>
<body>
<h1>401 - Unauthorized</h1>
</body>
</html>
2021-09-08T12:36:27.684Z [ERR] curl.cpp:CheckIAMCredentialUpdate(1830): IAM Access Token refresh failed
2021-09-08T12:36:27.684Z [DBG] curl_handlerpool.cpp:ReturnHandler(103): Return handler to pool
2021-09-08T12:36:27.684Z [INF] curl_handlerpool.cpp:ReturnHandler(110): Pool full: destroy the oldest handler
2021-09-08T12:36:27.685Z [CRT] s3fs.cpp:s3fs_check_service(3520): Failed to check IAM role name(SOME_ROLE).
2021-09-08T12:36:27.685Z [ERR] s3fs.cpp:s3fs_exit_fuseloop(3372): Exiting FUSE event loop due to errors
When running curl directly, though, we do receive a valid IMDS v2 token:
$ curl -v -X PUT -H "X-aws-ec2-metadata-token-ttl-seconds: 21600" http://169.254.169.254/latest/api/token
* Trying 169.254.169.254...
* TCP_NODELAY set
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> PUT /latest/api/token HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.58.0
> Accept: */*
> X-aws-ec2-metadata-token-ttl-seconds: 21600
>
* HTTP 1.0, assume close after body
< HTTP/1.0 200 OK
< Content-Length: 56
< Content-Type: text/plain
< Date: Wed, 08 Sep 2021 13:14:02 GMT
< X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21600
< Connection: close
< Server: EC2ws
<
* Closing connection 0
SOME_TOKEN
Finally, this is our proxy config (defined by environment variables):
$ echo $HTTP_PROXY
<SOME_HOST>:<SOME_PORT>
$ echo $NO_PROXY
169.254.169.254,*.eu-central-1.amazonaws.com
So, my best guess is that s3fs might be ignoring the NO_PROXY variable, trying to use our proxy when asking local IP 169.254.169.254 for a new token.
A fix for this is being worked on in https://github.com/s3fs-fuse/s3fs-fuse/pull/1766