I'm trying to use duplicity to create a rotating backup schedule: hourly backups for the last day, daily backups for the last week, weekly backups for the last month, monthly backups forever (to S3).
If there's a better way to accomplish this than what I'm about to describe, I'll take it. But I'll also take figuring out why this isn't working.
I've created a set of scripts: hourly
, hourly_clean
, daily
, etc...., and then put them in cron
12 * * * * root /etc/duplicity_backup/hourly
16 * * * * root /etc/duplicity_backup/hourly_clean
18 2 * * * root /etc/duplicity_backup/daily
22 2 * * * root /etc/duplicity_backup/daily_clean
24 3 * * 7 root /etc/duplicity_backup/weekly
28 3 * * 7 root /etc/duplicity_backup/weekly_clean
30 4 28 * * root /etc/duplicity_backup/monthly
Here are the hourly and daily scripts:
#!/usr/bin/ksh
#
# hourly
#
AWS_ACCESS_KEY_ID=<my_key_id>;
AWS_SECRET_ACCESS_KEY=<my_key>;
PASSPHRASE=foo;
export AWS_ACCESS_KEY_ID;
export AWS_SECRET_ACCESS_KEY;
export PASSPHRASE;
/usr/local/bin/duplicity --archive-dir /mnt/data/var/cache/duplicity \
--name webserver_vhosts --s3-region-name eu-central-1 \
--tempdir /mnt/data/var/tmp/ \
--full-if-older-than 1D \
--include **/wp-content \
--exclude **/wp-content/themes --exclude **/wp-content/plugins \
/mnt/data/www/vhosts/ boto3+s3://mirovoy-backup-webserver/hourly;
#!/usr/bin/ksh
#
# daily
#
AWS_ACCESS_KEY_ID=<my_key_id>;
AWS_SECRET_ACCESS_KEY=<my_key>;
PASSPHRASE=foo;
export AWS_ACCESS_KEY_ID;
export AWS_SECRET_ACCESS_KEY;
export PASSPHRASE;
/usr/local/bin/duplicity full --archive-dir /mnt/data/var/cache/duplicity \
--dry-run --name webserver_vhosts --s3-region-name eu-central-1 \
--tempdir /mnt/data/var/tmp/ \
--full-if-older-than 1W \
--s3-use-ia \
--include **/wp-content \
--exclude **/wp-content/themes --exclude **/wp-content/plugins \
/mnt/data/www/vhosts/ boto3+s3://mirovoy-backup-webserver/daily;
The hourly backups work fine, but the others, AFAICT, don't do anything.
If I try and run them manually, it gives a warning, but no errors:
$sudo ./daily
Warning, found signatures but no corresponding backup files
Sync would remove the following spurious local files:
duplicity-full-signatures.20210317T001201Z.sigtar.gz
duplicity-full-signatures.20210319T011202Z.sigtar.gz
duplicity-full.20210317T001201Z.manifest
duplicity-full.20210319T011202Z.manifest
duplicity-inc.20210317T001201Z.to.20210317T011201Z.manifest
duplicity-inc.20210317T011201Z.to.20210317T021201Z.manifest
[... lots more local files]
Last full backup date: none
--------------[ Backup Statistics ]--------------
StartTime 1616136196.38 (Fri Mar 19 06:43:16 2021)
EndTime 1616136207.36 (Fri Mar 19 06:43:27 2021)
ElapsedTime 10.98 (10.98 seconds)
SourceFiles 97158
SourceFileSize 1083141759 (1.01 GB)
NewFiles 54089
NewFileSize 1083141759 (1.01 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 54089
RawDeltaSize 0 (0 bytes)
TotalDestinationSizeChange 0 (0 bytes)
Errors 0
-------------------------------------------------
As I said, the hourly works fine, but the others aren't pushing anything to S3. Why are the daily/weekly/monthly backups not working? Alternatively, is there a better way to get duplicity to accomplish this?
The daily script has the option "--dry-run", which according to the duplicity man page has duplicity "Calculate what would be done, but do not perform any backend actions". The hourly script omits that option.