bash warning : delimited by end-of-file (wanted `!')

I have a K8S CronJob that uploads files to a remote SFTP server. To ensure that the Job does not run to completion when an upload error occurs I added error handling logic. The logic however is giving an error as below:

Starting-BACKUP
2023-09-27T07:39:01.384+0000    reading password from standard input
Enter password for mongo user:
/configmap/backup.sh: line 18: warning: here-document at line 9 delimited by end-of-file (wanted `!')
/configmap/backup.sh: line 19: syntax error: unexpected end of file
Database backup was successful

The error I am trying to handle occurs when the SFTP server is full on capacity. In such cases, the job should ideally not run to completion and the associated container should fail with exit code 1. When an upload is unsuccessful because of out-of-space issues the Job 's log output looks as below:

Starting-BACKUP
2023-09-26T13:25:01.406+0000    reading password from standard input
Enter password for mongo user:
Database backup was successful
Warning: Permanently added '[ftp.mycompany.com]:2020,[10.XX.XXX.XXX]:xxxx' (RSA) to the list of known hosts.     
Connected to ftp.mycompany.com.
Changing to: /oasys
sftp>   put labs-instance_20230926132501.gzip
Uploading labs-instance_20230926132501.gzip to /oasys/labs-instance_20230926132501.gzip
remote open("/oasys/labs-instance_20230926132501.gzip"): Failure
sftp>   bye
BackUp file labs-instance_20230926132501.gzip was successfully transfered
BackUp file labs-instance_20230926132501.gzip was deleted

Currently the job runs to completion despite the actual upload being unsuccessful:

NAME                       COMPLETIONS   DURATION   AGE
labs-backupjob-28262274    1/1           15s        7m42s

The pod created by the job also has a final status of Completed:

NAME                                   READY   STATUS      RESTARTS         AGE
labs-backupjob-28262286--1-s6g72       0/1     Completed   0                38s

The CronJob and bash script are given below :

 apiVersion: v1
    kind: ConfigMap
    metadata:
      name: backup-config
data:
  backup.sh: |-
    #!/bin/bash 
    set -e  # Exit on error

    echo "Starting-BACKUP"
    TIMESTAMP=$(date +"%Y%m%d%H%M%S")
    BACKUP_FILE_NAME="labs-instance_$TIMESTAMP.gzip"
    echo $MONGODB_ROOT_PASSWORD | mongodump --uri="mongodb://mongodb-prom:27017" --username="root" --authenticationDatabase="admin" --gzip --archive="$BACKUP_FILE_NAME" --quiet 
    echo "Database backup was successful"
    if ! sshpass -e sftp -o StrictHostKeyChecking=no -P XXXX oasys@mycompany.com:/oasys << !
    put $BACKUP_FILE_NAME
    bye
    !; then
      echo "Error uploading file to SFTP server"
      exit 1 
    fi
    echo "BackUp file $BACKUP_FILE_NAME was successfully transfered"
    rm $BACKUP_FILE_NAME
    echo "BackUp file $BACKUP_FILE_NAME was deleted"    

---
apiVersion: batch/v1
kind: CronJob
metadata:
  name: labs-backupjob
spec:
  schedule: "*/3 * * * *"
  concurrencyPolicy: Forbid
  failedJobsHistoryLimit: 1
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: mongodb-backup
            image: registry.mycompany.com/oasys/devops/docker-images/mongosh:6.0
            imagePullPolicy: Always            
            command: ["/configmap/backup.sh"]
            env:
            - name: MONGODB_ROOT_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: mongodb-labs
                  key: mongodb-root-password
            - name: SSHPASS
              valueFrom:
                secretKeyRef:
                  name: sftp-password
                  key: sftp-backup-password
            volumeMounts:
                - name: backup-volume
                  mountPath: /configmap
          volumes:
            - name: backup-volume
              configMap:
                name: backup-config    
                defaultMode: 0777
          restartPolicy: Never
          imagePullSecrets:
            - name: regcred
      backoffLimit: 0

What am I missing?

Solution

This is how I got it working - I added sftp option -oBatchMode=no -b -. Only then will sftp exit with status 1 if some sftp command goes wrong. Otherwise the exit status is always 0 . The modified code is as below :

apiVersion: v1
kind: ConfigMap
metadata:
  name: backup-config
data:
  backup.sh: |-
    #!/bin/bash 
    set -e  # Exit on error
echo "Starting-BACKUP"
TIMESTAMP=$(date +"%Y%m%d%H%M%S")
BACKUP_FILE_NAME="labs-instance_$TIMESTAMP.gzip"
echo $MONGODB_ROOT_PASSWORD | mongodump --uri="mongodb://mongodb-prom:27017" --username="root" --authenticationDatabase="admin" --gzip --archive="$BACKUP_FILE_NAME" --quiet 
echo "Database backup was successful"
sshpass -e sftp -o StrictHostKeyChecking=no -oBatchMode=no -b - -P XXXX oasys@mycompany.com:/oasys << !
  put $BACKUP_FILE_NAME
  bye
!
echo "BackUp file $BACKUP_FILE_NAME was successfully transfered"
rm $BACKUP_FILE_NAME
echo "BackUp file $BACKUP_FILE_NAME was deleted"