bashshellshdatabricksdatabricks-notebook

Handle error message from the cat command

I am using Databricks Notebook to execute Shell script using the %sh and I want to capture error message into err variable and then print the error message if the cat command fails (for eg, purposefully fail if the header_file directory doesn't exist). In the below code, if the cat command fails, it is coming inside the if statement and printing the error code correctly, but it is not showing the error from the err variable. The error message is showing as Error msg : and not showing why cat command failed.

err=$(cat /dbfs/mnt/devl/header_file/*.csv /dbfs/mnt/devl/data_files/*.csv > /dbfs/mnt/devl/output.txt 2>&1)
RC=$?
if [ $RC -ne 0 ]; then
  echo "Error code : $RC"
  echo "Error msg : $err"
fi

I checked if the issue is with the cat command and tried to replicate using ls command. In ls command, it is displaying the error message correctly as Error msg : ls: cannot access '/dbfs/mnt/devl/header_file/*.csv': No such file or directory. Is this due to some limitation with the cat command?

err=$(ls /dbfs/mnt/devl/header_file/*.csv 2>&1)
RC=$?
if [ $RC -ne 0 ]; then
  echo "Error code : $RC"
  echo "Error msg : $err"
fi

Note : My question is not related how to handle file/folder not found. The intention of this question is to display the actual error message that says why cat command failed for any kind of failure from the cat command.

Solution

  • You can resolve the issue by changing the places of STDOUT and STDERR redirects:

    err=$(cat /dbfs/mnt/devl/header_file/*.csv /dbfs/mnt/devl/data_files/*.csv 2>&1 > /dbfs/mnt/devl/output.txt)

    This will send the STDERR to the original STDOUT handler (in your case this is a err variable) and redirect the original STDOUT handler to file.