hadoopamazon-s3hdfss3distcp

Hadoop distcp to S3a with hidden key pair

How can I hide ACCESS_ID_KEY and SECRET_ACCESS_KEY for access to Amazon S3?

I know about adding it to core-site.xml, but maybe there is different solutions. Becouse with this approach every user from cluster will run distcp with same keys. Maybe there is some solution, like to store it in some property file for each cluster user?

Thanks.

Solution

  • Please see my HCC post on using the Hadoop Credential API for this use case.