hadoopamazon-web-servicesamazon-s3mapreduceelastic-map-reduce

Writing to a file in S3 from jar on EMR on AWS


Is there any way in which I can write to a file from my Java jar to an S3 folder where my reduce files would be written ? I have tried something like:

    FileSystem fs = FileSystem.get(conf);
    FSDataOutputStream FS = fs.create(new Path("S3 folder output path"+"//Result.txt"));        

    PrintWriter writer  = new PrintWriter(FS);
    writer.write(averageDelay.toString());
    writer.close();
    FS.close();

Here Result.txt is the new file which I would want to write.


Solution

  • Answering my own question:-

    I found my mistake.I should be passing the URI of S3 folder path to the fileSystem Object like below:-

     FileSystem fileSystem = FileSystem.get(URI.create(otherArgs[1]),conf);
     FSDataOutputStream fsDataOutputStream = fileSystem.create(new Path(otherArgs[1]+"//Result.txt"));      
     PrintWriter writer  = new PrintWriter(fsDataOutputStream);
     writer.write("\n Average Delay:"+averageDelay);
     writer.close();
     fsDataOutputStream.close();