Duplicity

practical limit on the number of incrementals?

Asked by Gábor Lipták

Is there a practical limit on the number of incrementals between full backups? Can full backups be taken a year apart (with incrementals between them)? Thanks

Question information

Language:
English Edit question
Status:
Solved
For:
Duplicity Edit question
Assignee:
No assignee Edit question
Solved by:
Kenneth Loafman
Solved:
Last query:
Last reply:
Revision history for this message
Kenneth Loafman (kenneth-loafman) said : 		#1

The practical limit is risk. You can increase the number of file
descriptors to handle long chains, but you'll have increasing overhead on
each link of the chain. Plus, any error in storage and the chain will
break.

On Wed, Oct 2, 2013 at 8:41 PM, gliptak <
<email address hidden>> wrote:

> New question #236750 on Duplicity:
> https://answers.launchpad.net/duplicity/+question/236750
>
> Is there a practical limit on the number of incrementals between full
> backups? Can full backups be taken a year apart (with incrementals between
> them)? Thanks
>
> --
> You received this question notification because you are a member of
> duplicity-team, which is an answer contact for Duplicity.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help : https://help.launchpad.net/ListHelp
>

Revision history for this message
Gábor Lipták (gliptak) said : 		#2

Kenneth, how I'm reading your answer is that duplicity might run out of filehandles when processing (reading and writing?) great number of incrementals.

So for 1024 handles, what would the risk be acceptable?

I'm trying to learn the "right rotation" using duplicity.

Would an approach of

daily 31
monthly 12
yearly all

be a good one?
(Similar as being described at http://scripts.dragon-it.co.uk/scripts.nsf/docs/batch-backup-with-rotation-and-emails!OpenDocument&ExpandSection=3&BaseTarget=East&AutoFramed and http://stackoverflow.com/questions/3481324/script-or-util-to-remove-old-backups )

Revision history for this message
Best Kenneth Loafman (kenneth-loafman) said : 		#3

You probably need to increase ulimit's num_files to 32768 or so if you are going to run a 31 day incremental chain. It's essentially 4 file handles per incremental file present for the backup. If you trust your storage that is. My experience is that you really should limit incremental chains to shortest possible. I do a full every Sunday and incrementals for the rest. I don't mess with monthly or yearly, and just keep six weeks of backups.

Revision history for this message
Gábor Lipták (gliptak) said : 		#4

Thank you Kenneth. I guess the answer to "trusting" the storage is to keep multiples with different backends.

Revision history for this message
Gábor Lipták (gliptak) said : 		#5

Thanks Kenneth Loafman, that solved my question.

Revision history for this message
edso (ed.so) said : 		#6

Ken,

isn't the ulimit issue solved with the patched gpg interface? doesn't it dispose unused file handlers now in time?

..ede/duply.net

Revision history for this message
Kenneth Loafman (kenneth-loafman) said : 		#7

edso,

Yes, it is fixed. Just playing it safe in case the patched gpg interface
was removed.

...Ken

On Mon, Oct 7, 2013 at 5:41 AM, edso
<email address hidden>wrote:

> Question #236750 on Duplicity changed:
> https://answers.launchpad.net/duplicity/+question/236750
>
> edso posted a new comment:
> Ken,
>
> isn't the ulimit issue solved with the patched gpg interface? doesn't it
> dispose unused file handlers now in time?
>
> ..ede/duply.net
>
> --
> You received this question notification because you are a member of
> duplicity-team, which is an answer contact for Duplicity.
>
> _______________________________________________
> Mailing list: https://launchpad.net/~duplicity-team
> Post to : <email address hidden>
> Unsubscribe : https://launchpad.net/~duplicity-team
> More help : https://help.launchpad.net/ListHelp
>