I came across a problem and thought of a question I did not find a good answer to. And that is, how can I purposely make an AWS EMR step fail? I have a Spark Scala script which is added as a Spark step with some command line arguments and the output of the script is written into S3.
But if something goes wrong reading and handling the command line arguments, then the logic of the script is skipped and the script ends. But for EMR it is normal behaviour, it does not know that an if block was not entered.
And after a "failed" run, the step status is still changed to "Completed" and it seems it was successful without the results being written into S3.
I want to finish the step so it would be in "Failed" status.
I can do that by throwing an exception and then I can see the corresponding exception with my message in EMR Step error logs. But is there a better way? I would like to handle all my exceptions myself, manually.
And in addition, can I use AWS SDK to somehow programmatically find out the reason of the step's failure?
Return a non-zero value from your program