[SOLVED] Copying Data from AWS bucket to Ceph Bucket

Copying Data from AWS bucket to Ceph Bucket

I have a ceph object bucket and an AWS bucket. I want to copy data from the AWS bucket to the ceph bucket without copying data to any intermediate system local system. Is there a way to do that, since there will be requirement of separate endpoint, separate keys.

Solution

I have added information of both bucket in hdfs-client.xml Now I am using Hadoop distcp to move the data from one bucket to other.

This helped.

Configuring different S3 buckets with Per-Bucket Configuration
Different S3 buckets can be accessed with different S3A client configurations. This allows for different endpoints, data read and write strategies, as well as login details.

All fs.s3a options other than a small set of unmodifiable values (currently fs.s3a.impl) can be set on a per bucket basis.
The bucket specific option is set by replacing the fs.s3a. prefix on an option with fs.s3a.bucket.BUCKETNAME., where BUCKETNAME is the name of the bucket.
When connecting to a bucket, all options explicitly set will override the base fs.s3a. values.
As an example, a configuration could have a base configuration to use the IAM role information available when deployed in Amazon EC2.

<property>
  <name>fs.s3a.aws.credentials.provider</name>
  <value>com.amazonaws.auth.InstanceProfileCredentialsProvider</value>
</property>
This will become the default authentication mechanism for S3A buckets.

A bucket s3a://nightly/ used for nightly data can then be given a session key:

<property>
  <name>fs.s3a.bucket.nightly.access.key</name>
  <value>AKAACCESSKEY-2</value>
</property>

<property>
  <name>fs.s3a.bucket.nightly.secret.key</name>
  <value>SESSIONSECRETKEY</value>
</property>

<property>
  <name>fs.s3a.bucket.nightly.session.token</name>
  <value>Short-lived-session-token</value>
</property>

<property>
  <name>fs.s3a.bucket.nightly.aws.credentials.provider</name>
  <value>org.apache.hadoop.fs.s3a.TemporaryAWSCredentialsProvider</value>
</property>
Finally, the public s3a://landsat-pds/ bucket can be accessed anonymously:

<property>
  <name>fs.s3a.bucket.landsat-pds.aws.credentials.provider</name>
  <value>org.apache.hadoop.fs.s3a.AnonymousAWSCredentialsProvider</value>
</property>

ref : Refrence Link