hadoopmapreducemultipleoutputs

hadoop MultipleOutputs to absolute path , but file is already being created by other attempt

I use MultipleOutputs to output data to some absolute paths, instead of a path relative to OutputPath.

Then, i get the error:

Error: org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.hdfs.protocol.AlreadyBeingCreatedException): Failed to create file [/test/convert.bak/326/201505110030/326-m-00035] for [DFSClient_attempt_1425611626220_29142_m_000035_1001_-370311306_1] on client [192.168.7.146], because this file is already being created by [DFSClient_attempt_1425611626220_29142_m_000035_1000_-53988495_1] on [192.168.7.149] at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.recoverLeaseInternal(FSNamesystem.java:2320) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInternal(FSNamesystem.java:2083) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFileInt(FSNamesystem.java:2012) at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startFile(FSNamesystem.java:1963) at

Solution

  • https://issues.apache.org/jira/browse/MAPREDUCE-6357

    Output files must in ${mapred.output.dir} 。

    The design and implementation dosn't support outputing data to files out of ${mapred.output.dir}.