amazon-web-servicesapache-pigemr

Pass AWS credentials to PigStorage function


How can I pass AWS credentials (aws_access_key and aws_secret_key) to PIG PigStorage function?

Thanks


Solution

  • I wasn't very explicit, and did gave an example of my use case, sorry. I needed that because I needed to use two different AWS access_keys, and using something like s3n://access:secret@bucket did not solve. I solved this changing the PigStorage function , storing in hdfs the results, and on the cleanUpWithSucess method invoke a method that uploads the hdfs files to s3 with credentials. In this way I can pass the credentials to the PigStorageFunction when it is used to store, of course I also changed the constructor of the PigStorage to receive these arguments.