I have a backup plan:
{
"BackupPlan": {
"BackupPlanName": "prometheus_dev",
"Rules": [
{
"RuleName": "prometheus_dev",
"TargetBackupVaultName": "prometheus_eu_central_1_dev",
"ScheduleExpression": "cron(15 * ? * * *)",
"StartWindowMinutes": 60,
"CompletionWindowMinutes": 180,
"Lifecycle": {
"DeleteAfterDays": 7
},
"RuleId": "ffcd7e8c-9b14-4e2b-89f0-d8cbe5b5ae25",
"CopyActions": [
{
"Lifecycle": {
"DeleteAfterDays": 7
},
"DestinationBackupVaultArn": "arn:aws:backup:eu-west-1:614797193252:backup-vault:prometheus_backup_eu_west_1_dev"
}
],
"EnableContinuousBackup": false
}
]
},
"BackupPlanId": "830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
"BackupPlanArn": "arn:aws:backup:eu-central-1:614797193252:backup-plan:830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
"VersionId": "YzUxMjhkODgtZTc0MC00NDA1LWEwYzktNmE4NDFhMDE5MTA4",
"CreationDate": "2023-10-13T13:42:22.048000+02:00",
"LastExecutionDate": "2023-10-20T11:16:48.451000+02:00"
}
The file system is protected:
{
"ResourceArn": "arn:aws:elasticfilesystem:eu-central-1:614797193252:file-system/fs-0318e5506f10caf1e",
"ResourceType": "EFS",
"LastBackupTime": "2023-10-20T11:22:48.159000+02:00",
"ResourceName": "prometheus_dev"
}
And backups are taken hourly:
// ...
{
"AccountId": "614797193252",
"BackupJobId": "06178992-5DF7-17BA-1E38-C969357B644A",
"BackupVaultName": "prometheus_eu_central_1_dev",
"BackupVaultArn": "arn:aws:backup:eu-central-1:614797193252:backup-vault:prometheus_eu_central_1_dev",
"RecoveryPointArn": "arn:aws:backup:eu-central-1:614797193252:recovery-point:58cb7957-5b0c-4038-a579-af78aadbc506",
"ResourceArn": "arn:aws:elasticfilesystem:eu-central-1:614797193252:file-system/fs-0318e5506f10caf1e",
"CreationDate": "2023-10-20T08:15:00+02:00",
"CompletionDate": "2023-10-20T08:25:07.003000+02:00",
"State": "COMPLETED",
"PercentDone": "100.0",
"BackupSizeInBytes": 0,
"IamRoleArn": "arn:aws:iam::614797193252:role/prometheus_backup_dev",
"CreatedBy": {
"BackupPlanId": "830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
"BackupPlanArn": "arn:aws:backup:eu-central-1:614797193252:backup-plan:830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
"BackupPlanVersion": "YzUxMjhkODgtZTc0MC00NDA1LWEwYzktNmE4NDFhMDE5MTA4",
"BackupRuleId": "ffcd7e8c-9b14-4e2b-89f0-d8cbe5b5ae25"
},
"StartBy": "2023-10-20T09:15:00+02:00",
"ResourceType": "EFS",
"IsParent": false,
"ResourceName": "prometheus_dev"
},
{
"AccountId": "614797193252",
"BackupJobId": "DE456E1F-1A49-4A36-507A-32646DD0AE85",
"BackupVaultName": "prometheus_eu_central_1_dev",
"BackupVaultArn": "arn:aws:backup:eu-central-1:614797193252:backup-vault:prometheus_eu_central_1_dev",
"RecoveryPointArn": "arn:aws:backup:eu-central-1:614797193252:recovery-point:998837d7-e0c8-4505-9d4f-ff19ca1f69c7",
"ResourceArn": "arn:aws:elasticfilesystem:eu-central-1:614797193252:file-system/fs-0318e5506f10caf1e",
"CreationDate": "2023-10-20T07:15:00+02:00",
"CompletionDate": "2023-10-20T07:25:52.381000+02:00",
"State": "COMPLETED",
"PercentDone": "100.0",
"BackupSizeInBytes": 0,
"IamRoleArn": "arn:aws:iam::614797193252:role/prometheus_backup_dev",
"CreatedBy": {
"BackupPlanId": "830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
"BackupPlanArn": "arn:aws:backup:eu-central-1:614797193252:backup-plan:830e8bb0-e3c6-4a96-8fe9-45a3ab97de40",
"BackupPlanVersion": "YzUxMjhkODgtZTc0MC00NDA1LWEwYzktNmE4NDFhMDE5MTA4",
"BackupRuleId": "ffcd7e8c-9b14-4e2b-89f0-d8cbe5b5ae25"
},
"StartBy": "2023-10-20T08:15:00+02:00",
"ResourceType": "EFS",
"IsParent": false,
"ResourceName": "prometheus_dev"
},
// ...
The problem is that ALL recovery points are empty, even if the file system is not:
{
// ...
"SizeInBytes": {
"Value": 67465216,
"Timestamp": "2023-10-20T10:41:44+02:00",
"ValueInIA": 0,
"ValueInStandard": 67465216
}
// ...
}
I've also tried to start a backup job:
aws backup start-backup-job \
--backup-vault-name prometheus_eu_central_1_dev \
--resource-arn arn:aws:elasticfilesystem:eu-central-1:614797193252:file-system/fs-0318e5506f10caf1e \
--iam-role-arn arn:aws:iam::614797193252:role/prometheus_backup_dev
Which produced the following response:
{
"BackupJobId": "7a010939-11cd-4d6f-bf2e-bbec0fc50452",
"RecoveryPointArn": "arn:aws:backup:eu-central-1:614797193252:recovery-point:ac338f60-9a4e-4c24-817e-2e0e15a72d03",
"CreationDate": "2023-10-20T11:22:48.159000+02:00",
"IsParent": false
}
But again the backup is empty:
{
"AccountId": "614797193252",
"BackupJobId": "7a010939-11cd-4d6f-bf2e-bbec0fc50452",
"BackupVaultName": "prometheus_eu_central_1_dev",
"BackupVaultArn": "arn:aws:backup:eu-central-1:614797193252:backup-vault:prometheus_eu_central_1_dev",
"RecoveryPointArn": "arn:aws:backup:eu-central-1:614797193252:recovery-point:ac338f60-9a4e-4c24-817e-2e0e15a72d03",
"ResourceArn": "arn:aws:elasticfilesystem:eu-central-1:614797193252:file-system/fs-0318e5506f10caf1e",
"CreationDate": "2023-10-20T11:22:48.159000+02:00",
"CompletionDate": "2023-10-20T11:22:55.105000+02:00",
"State": "COMPLETED",
"PercentDone": "100.0",
"BackupSizeInBytes": 0,
"IamRoleArn": "arn:aws:iam::614797193252:role/prometheus_backup_dev",
"ResourceType": "EFS",
"BytesTransferred": 0,
"StartBy": "2023-10-20T19:22:48.159000+02:00",
"IsParent": false,
"ResourceName": "prometheus_dev"
}
Why are these backups empty? How can i further debug this?
I see no errors anywhere, and the role used has the necessary permissions, since it's using AWS managed policy.
EDIT:
BackupSizeInBytes
as 0
, creating an instance from the recovery point provisions an instance with data as expected.EDIT 2:
Because AWS Backup perform incremental backups. The initial backup will be a full backup then the following, even the on-demand backup, will be an incremental backup.
AWS Backup performs incremental backups of EFS file systems. During the initial backup, a copy of the entire file system is made. During subsequent backups of that file system, only files and directories that have been changed, added, or removed are copied. https://docs.aws.amazon.com/efs/latest/ug/awsbackup.html#incremental-backup
"BackupSizeInBytes": 0,
This just mean that from the last recovery point/backup to the next, there wasn't any change.
I checked my backups as well and I got the same. But that's because I'm not actively using my EFS.
{
> aws backup describe-recovery-point --backup-vault-name myEfsVault --recovery-point-arn <recoveryPointArn>
...
"Status": "COMPLETED",
"CreationDate": "2023-10-20T07:00:00+02:00",
"CompletionDate": "2023-10-20T09:15:48.157000+02:00",
"BackupSizeInBytes": 0,
...
}
Here's what you can do to confirm your backup integrity:
Bonus
Here's an AWS recommendation for performing data recovery validation with AWS Backup.
https://aws.amazon.com/blogs/storage/automate-data-recovery-validation-with-aws-backup/
Backup testing
New Initial Backup
}
...
"Status": "COMPLETED",
"CreationDate": "2023-10-20T14:54:57.001000+02:00",
"CompletionDate": "2023-10-20T14:55:07.614000+02:00",
"BackupSizeInBytes": 1199258039,
...
}
Followup on-demand backup:
{
...
"Status": "COMPLETED",
"CreationDate": "2023-10-20T14:57:43.050000+02:00",
"CompletionDate": "2023-10-20T14:57:50.314000+02:00",
"BackupSizeInBytes": 0,
...
}
Restore test:
Restored EFS and Original EFS (Original names redacted):
Create Mount Targets (Select subnet, SG, AZ)
Mount restored FS and check contents:
[root@test-server ~]# mkdir /efs-restore
[root@test-server ~]# mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport,mountport=2049 10.10.10.113:/ /efs-restore
[root@test-server efs-restore]# df -h /efs-restore
Filesystem Size Used Avail Use% Mounted on
10.10.10.113:/ 8.0E 0 8.0E 0% /efs-restore
[root@test-server efs-restore]# ll /efs-restore
drwxrwxr-x 4 root root 6144 Aug 14 12:39 aws-backup-restore_2023-10-20T13-00-49-858648977Z
[root@test-server efs-restore]# ll /efs-restore/aws-backup-restore_2023-10-20T13-00-49-858648977Z/
drw--w---- 2 root root 6144 Oct 20 15:00 aws-backup-lost+found_2023-10-20T13-00-32-067742883Z
drwxrwxr-x 3 ec2-user ec2-user 6144 Aug 1 11:38 logs # <-- my application logs
[root@test-server efs-restore]# ll /efs-restore/aws-backup-restore_2023-10-20T13-00-49-858648977Z/logs/path/to/my/application/
total 100
drwxr-xr-x 3 ec2-user ec2-user 6144 Oct 12 14:55 ip-10-10-4-123
drwxr-xr-x 3 ec2-user ec2-user 6144 Oct 12 17:55 ip-10-10-4-127
drwxrwxr-x 3 ec2-user ec2-user 6144 Sep 11 16:50 ip-10-10-4-151
drwxr-xr-x 3 ec2-user ec2-user 6144 Oct 11 09:28 ip-10-10-4-160
Restore test 2:
Deleted the initial backup full backup to simulate the retention period. Then performed another restore, everything is still there.