We are unable to mount a S3 bucket on an AWS EC2 instance that requires IMDS v2 (IMDS v1 works fine, though!). We're using s3fs-fuse v1.90 which is supposed to support IMDS v2 (on Ubuntu 18.04).
This is how we've been trying to mount our S3 bucket:
sudo s3fs <bucket> <mount-point> -o iam_role=auto,url=https://s3.eu-central-1.amazonaws.com,endpoint=eu-central-1,allow_other,uid=1000,gid=1000,mp_umask=007,use_cache=/tmp/s3foldercache,dbglevel=debug -f
However, we receive this error message:
2021-09-02T12:40:53.157Z [INF] curl.cpp:LoadIAMRoleFromMetaData(2919): Get IAM Role name
2021-09-02T12:40:53.157Z [DBG] curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 31
2021-09-02T12:40:53.158Z [DBG] curl.cpp:RequestPerform(2283): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/
2021-09-02T12:40:53.159Z [ERR] curl.cpp:RequestPerform(2394): HTTP response code 401, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>401 - Unauthorized</title>
</head>
<body>
<h1>401 - Unauthorized</h1>
</body>
</html>
2021-09-02T12:40:53.160Z [CRT] s3fs.cpp:s3fs_init(3395): could not load IAM role name from meta data.
2021-09-02T12:40:53.160Z [ERR] s3fs.cpp:s3fs_exit_fuseloop(3372): Exiting FUSE event loop due to errors
Again, when we allow IMDS v1 on our EC2 instance, the mount command works fine.
How do we make sure s3fs will use IMDS v2?
EDIT:
Many thanks to @Noah for contributing a fix for this to s3fs. I've pulled the latest commit but am seeing a new error message now. Apparently IMDSv2 token retrieval fails, regardless of whether I apply iam_role=auto or iam_role=some_profile (removing -f wouldn't change anything either):
2021-09-06T05:43:58.857Z [INF] s3fs.cpp:s3fs_init(3382): init v1.90(commit:9d1552a) with OpenSSL
2021-09-06T05:43:58.857Z [INF] curl.cpp:LoadIAMRoleFromMetaData(3181): Get IAM Role name
2021-09-06T05:43:58.857Z [DBG] curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 31
2021-09-06T05:43:58.857Z [DBG] curl.cpp:RequestPerform(2509): connecting to URL http://169.254.169.254/latest/api/token
2021-09-06T05:43:58.889Z [ERR] curl.cpp:RequestPerform(2622): HTTP response code 417, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>417 - Expectation Failed</title>
</head>
<body>
<h1>417 - Expectation Failed</h1>
</body>
</html>
2021-09-06T05:43:58.890Z [ERR] curl.cpp:LoadIAMRoleFromMetaData(3212): AWS IMDSv2 token retrieval failed: -5
2021-09-06T05:43:58.890Z [DBG] curl.cpp:RequestPerform(2509): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/
2021-09-06T05:43:58.892Z [ERR] curl.cpp:RequestPerform(2622): HTTP response code 401, returning EIO. Body Text: <?xml version="1.0" encoding="iso-8859-1"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<title>401 - Unauthorized</title>
</head>
<body>
<h1>401 - Unauthorized</h1>
</body>
</html>
2021-09-06T05:43:58.892Z [CRT] s3fs.cpp:s3fs_init(3395): could not load IAM role name from meta data.
2021-09-06T05:43:58.892Z [ERR] s3fs.cpp:s3fs_exit_fuseloop(3372): Exiting FUSE event loop due to errors
Does it work if you specify the IAM role explicitly?
It looks like there may be an issue with retrieval of the IAM role information (needed with iam_role=auto) when tokens are required.
Additionally, when testing this, I had issues with when the -f option is used, so also try removing that.
Below are logs that show the proper token retrieval when mounting a bucket with the following command. Bucket access via the mount point works as expected.
$ sudo s3fs <REDACTED> /mnt -o iam_role=s3fs-testing,allow_other,uid=1000,gid=1000,mp_umask=007,use_cache=/tmp/s3foldercache,dbglevel=debug
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: init v1.89(commit:unknown) with GnuTLS(gcrypt)
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: check services.
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: IAM Access Token refreshing...
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: [IAM role=s3fs-testing]
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: curl_handlerpool.cpp:GetHandler(81): Get handler from pool: rest = 31
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: curl.cpp:RequestPerform(2254): connecting to URL http://169.254.169.254/latest/api/token
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: HTTP response code 200
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: Setting AWS IMDSv2 API token to <REDACTED>
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: curl.cpp:RequestPerform(2254): connecting to URL http://169.254.169.254/latest/meta-data/iam/security-credentials/s3fs-testing
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: HTTP response code 200
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: IAM credential response = "{
"Code" : "Success",
"LastUpdated" : "2021-09-02T15:36:20Z",
"Type" : "AWS-HMAC",
"AccessKeyId" : "REDACTED",
"SecretAccessKey" : "REDACTED",
"Token" : "REDACTED",
"Expiration" : "2021-09-02T22:10:48Z"
}"
Sep 02 16:06:46 ip-10-0-0-77 s3fs[600]: IAM Access Token refreshed