I am trying to create keys in apache OZone using AWS S3 API for .NET.
The key I am trying to create must be inside a bucket called "test" that I created using AWS S3 CLI.
My code:
static async Task WriteFile()
{
AmazonS3Config config = new AmazonS3Config();
config.ServiceURL = "http://myApacheOzoneEndpoint:8744"; // This port is mapped from a docker container to (not the original endpoint port for Ozone)
AWSCredentials credentials = new BasicAWSCredentials("testuser/scm@EXAMPLE.COM", "c261b6ecabf7d37d5f9ded654b1c724adac9bd9f13e247a235e567e8296d2999"); // Credentials must be set but can be random since Ozone doesn't use authentication
AmazonS3Client client = new AmazonS3Client(credentials, config);
using (FileStream fs = File.OpenRead(@"C:\Users\me\path.to.file\image.jpg"))
{
string responseBody = "";
try
{
PutObjectRequest request = new PutObjectRequest
{
BucketName = "test",
Key = "deleteme.jpg",
InputStream = fs
};
PutObjectResponse response = await client.PutObjectAsync(request);
Console.WriteLine($"Result: {response.HttpStatusCode.ToString()}");
}
catch (AmazonS3Exception e)
{
Console.WriteLine("Error encountered ***. Message:'{0}' when writing an object", e.Message);
}
catch (Exception e)
{
Console.WriteLine("Unknown encountered on server. Message:'{0}' when writing an object", e.Message);
}
}
}
This code is returning a 200 code error (OK), and if I call GetObjectAsync from the API I get an HTML with metadata as result (But can't read the file content by the moment)
Then I go to Apache Ozone and execute these commands in within the ozone shell:
bash-4.2$ ozone s3 path test
Volume name for S3Bucket is : s3c89e813c80ffcea9543004d57b2a1239
Ozone FileSystem Uri is : o3fs://test.s3c89e813c80ffcea9543004d57b2a1239
bash-4.2$ ozone sh bucket list /s3c89e813c80ffcea9543004d57b2a1239
[ {
"volumeName" : "s3c89e813c80ffcea9543004d57b2a1239",
"bucketName" : "test",
"createdOn" : "Wed, 01 Apr 2020 08:06:40 GMT",
"acls" : null,
"versioning" : "DISABLED",
"storageType" : "DISK",
"encryptionKeyName" : "N/A"
}, {
"volumeName" : "s3c89e813c80ffcea9543004d57b2a1239",
"bucketName" : "deleteme.jpg",
"createdOn" : "Tue, 31 Mar 2020 10:57:26 GMT",
"acls" : null,
"versioning" : "DISABLED",
"storageType" : "DISK",
"encryptionKeyName" : "N/A"
} ]
bash-4.2$ ozone sh key list /s3c89e813c80ffcea9543004d57b2a1239/test => This command returns only keys that have been put from AWS S3 CLI
As you can see, the object is created as a new bucket inside the Ozone volume used for S3, instead of being created as a new key below test
bucket. If I try to put keys from AWS S3 Cli it works as expected
What is happening?
FYI I made the same example from Java API and the same issue occurs!
Thanks
Ozone s3-gateway uses by default the path-style addressing while updated sdk libraries use the virtual-hosted addressing. The quickest solution would be to switch to path-style:
// AmazonS3Config config = new AmazonS3Config();
config.ForcePathStyle = true;
Alternatively, as mentioned on the docs, you could enable the virtual-hosted schema in ozone.
Please notice that the path-style is going to be deprecated in aws s3.