hivehbasehiveqlapache-phoenixapache-tez

Why am I getting negative allocated mappers in Tez job? Vertex failure?

I'm trying to use the PhoenixStorageHandler as documented here, and populate it with the following query in beeline shell:

insert into table pheonix_table select * from hive_table;

I get the following breakdown of the mappers in the Tez session:

...
INFO  : Map 1: 0(+50)/50 
INFO  : Map 1: 0(+50)/50 
INFO  : Map 1: 0(+50,-2)/50 
INFO  : Map 1: 0(+50,-3)/50
...

before the session crashes with a very long error message (422 lines) about vertex failure:

Error: Error while processing statement: FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.tez.TezTask. Vertex failed, vertexName=Map 1, vertexId=vertex_1499857429667_0084_2_00, diagnostics=[Task failed, taskId=task_1499857429667_0084_2_00_000007, diagnostics=[TaskAttempt 0 failed, info=[Error: Failure while running task:java.lang.RuntimeException: java.lang.RuntimeException: Map operator initialization failed [.........] Vertex did not succeed due to OWN_TASK_FAILURE, failedTasks:1 killedTasks:49, Vertex vertex_1499857429667_0084_2_00 [Map 1] killed/failed due to:OWN_TASK_FAILURE]DAG did not succeed due to VERTEX_FAILURE. failedVertices:1 killedVertices:0 (state=08S01,code=2)

What is this error referring to? Why are there 'negative mappers'?

Solution

  • Negative number indicates the number of failed or killed attempts. The format is:

    finished(running,-failed or killed)/total

    You can see details about why some mapper has failed in job tracker logs.

    See also this answer: https://stackoverflow.com/a/39144600/2700344