Duplicity

Using Swift backend, signatures and manifests are not segmented

Asked by Doug

Ubuntu 14.04
Duplicity 0.7.11

We are using duplicity to back up a large filesystem with many many small files. Currently the backup runs successfully but the signatures file is greater than 5G even compressed. This means swift rejects the upload as too big and the backup fails. When uploading the manifest or signature files, duplicity should use the swift upload -S to enable dynamic large objects support. A manual swift upload works and a subsequent collection-status is correct.

duplicity --verbosity Notice ${OPTIONAL_OPTIONS} \
                --full-if-older-than 5D \
                --num-retries 3 \
                --asynchronous-upload \
                --no-encryption \
                --max-blocksize 20480 \
                --volsize 1024 \
                ${EXCLUDE_ARCHIVE_DIR} ${SRC} "${DEST}"

It would be nice if the volsize was used for -S parameter on upload of the manifest and signature files.

Question information

Language:
English Edit question
Status:
Answered
For:
Duplicity Edit question
Assignee:
No assignee Edit question
Last query:
Last reply:
Revision history for this message
Doug (caniwi) said : 		#1

The duplicity swift error message is below:

Attempt 1 failed. ClientException: Object PUT failed: https://api.cloud.catalyst.net.nz:8443/v1/AUTH_a5f867bc7e244b51b7531097b55e7c00/cat-prod-seafile.catalyst.net.nz-seafile.catalyst.net.nz/duplicity-full-signatures.20170305T133159Z.sigtar.gz 413 Request Entity Too Large [first 60 chars of response] <html>^M
<head><title>413 Request Entity Too Large</title></h
Attempt 2 failed. ClientException: Object PUT failed: https://api.cloud.catalyst.net.nz:8443/v1/AUTH_a5f867bc7e244b51b7531097b55e7c00/cat-prod-seafile.catalyst.net.nz-seafile.catalyst.net.nz/duplicity-full-signatures.20170305T133159Z.sigtar.gz 413 Request Entity Too Large [first 60 chars of response] <html>^M
<head><title>413 Request Entity Too Large</title></h
Giving up after 3 attempts. ClientException: Object PUT failed: https://api.cloud.catalyst.net.nz:8443/v1/AUTH_a5f867bc7e244b51b7531097b55e7c00/cat-prod-seafile.catalyst.net.nz-seafile.catalyst.net.nz/duplicity-full-signatures.20170305T133159Z.sigtar.gz 413 Request Entity Too Large [first 60 chars of response] <html>^M
<head><title>413 Request Entity Too Large</title></h
Backup failed with rc = 50

Revision history for this message
Kenneth Loafman (kenneth-loafman) said : 		#2

See the linked bug report.

Here are a couple of workarounds, for now:

 - Increase --max-blocksize to 65536 or higher.
- Split the backup into a couple of runs.

Revision history for this message
Kenneth Loafman (kenneth-loafman) said : 		#3

See the linked bug report.

Here are a couple of workarounds, for now:

 - Increase --max-blocksize to 65536 or higher.
- Split the backup into a couple of runs.

Revision history for this message
Doug (caniwi) said : 		#4

Hi Ken,
Thanks for taking the time to reply. I think I am in a rock and a hard place. As I am stuck with millions of small files.

 # find . -type f |wc -l
1721980
# find . -size +10M -type f |wc -l
2169
# find . -size +5M -type f |wc -l
5204
# find . -size +1M -type f |wc -l
33583

So from my reading of the man page, setting the --max-blocksize is not really helping my cause. So that leaves me with the option of splitting the backup into a couple of runs which won't be straight forward for me either as the distribution of data can vary greatly in the sub directories.

Revision history for this message
Kenneth Loafman (kenneth-loafman) said : 		#5

If you truly have millions of files, then you will have to split your backup into manageable pieces for now. 2GB is a very small file size limitation, and I'm sure file metadata is eating up a significant portion of the sigtar file, maybe close to 50% if the file sizes are below the max-blocksize.

Revision history for this message
kiorky (kiorky) said : 		#6

@caniwi you may be pleased to test latest https://gitlab.com/duplicity/duplicity , it may be eventually fixed ^^

Revision history for this message
kiorky (kiorky) said : 		#7

specifically : https://gitlab.com/duplicity/duplicity/-/commit/ff1919877a095387294c0a7e225f9f65b2e24107

Can you help with this problem?

Provide an answer of your own, or ask Doug for more information if necessary.

To post a message you must log in.