Déjà Dup

Déjà Dup can't restore the backup and Duplicity fails to recognize the most recent backup.

Asked by Marco

I have made a backup of my Home folder (approximately 260 GB) on Google Drive. At the time my OS was Ubuntu 18.04. Before reinstalling my OS, Duplicity correctly found 3 backups made in three different dates: 2020-04-29, 2020-05-04 and 2020-05-28.

I uninstalled Ubuntu and installed Debian. In Debian I have installed Déjà Dup and it could correctly find the latest backup (2020-05-28), but couldn't restore it all. The Debian experience didn't work out well for me and I have switched to Ubuntu 20.04.

When I try to restore the backup from the same GDrive account the program fails to recognize the folder (drive/backup) and creates an empty new one (so from my Drive account I see two folders with the same name, one empty and one not).

To go around this issue I had downloaded all of the files, put them into an hard drive and attempted to restore the backup from there.

Déjà Dup fails to find the backup in the folder that I specify.

Therefore I had tried with Duplicity, using the command line.

duplicity collection-status returns the following:

Last full backup date: Wed Apr 29 16:30:36 2020
Collection Status
-----------------
Connecting with backend: BackendWrapper
Archive dir: /home/user/.cache/duplicity/76bcf7cb13525876fd1ba4ac5713cb21

Found 1 secondary backup chain.
Secondary chain 1 of 1:
-------------------------
Chain start time: Wed Apr 29 16:30:36 2020
Chain end time: Wed Apr 29 16:30:36 2020
Number of contained backup sets: 1
Total number of contained volumes: 178
 Type of backup set: Time: Num volumes:
                Full Wed Apr 29 16:30:36 2020 178
-------------------------

No backup chains with active signatures found
No orphaned or incomplete backup sets found.

Which is incorrect since the volumes are way more than 178.

Performing a restore leads to the restoration of approximately 220MB of data.

My hypothesis is that there is something wrong with the name of the volumes. In fact, looking at the volumes in the folder, there are the following sets of archives (sorted by name):

* The first file is duplicity-full.20200429T143036Z.manifest.gpg

* Then there are 178 files with the following names: duplicity-full.20200429T143036Z.vol1.difftar.gpg to duplicity-full.20200429T143036Z.vol178.difftar.gpg.

* A file named duplicity-full-signatures.20200429T143036Z.sigt.gpg

* The original, and copies renamed from (1) to (5180) of the file: duplicity-inc.20200429T143036Z.to.20200429T1620.gpg

* duplicity-inc.20200429T162022Z.to.20200504T1257.gpg

* The original, and copies renamed from (1) to (6273)of the file: duplicity-inc.20200429T162022Z.to.20200504T1257.gpg

* The original, and copies renamed from (1) to (2)of the file: duplicity-inc.20200504T125757Z.to.20200528T0931.gpg

* The following three files: duplicity-new-signatures.20200429T143036Z.to.20.gpg ; duplicity-new-signatures.20200429T162022Z.to.20.gpg; duplicity-new-signatures.20200504T125757Z.to.20.gpg;

I can unencrypt all the files, potentially I could rename all the files if necessary, but I really need a good criterion to do so and, since I'm not that tech-savy, I would rather not to have to have to restore manually using rdiff as shown in this page: https://wiki.gnome.org/Apps/DejaDup/Help/Restore/WorstCase .

The files I have in my backup are very important and I need them ASAP, I kindly appreciate all your answers as soon as you can provide them to me.

Best regards.

Question information

Language:
English Edit question
Status:
Solved
For:
Déjà Dup Edit question
Assignee:
No assignee Edit question
Solved by:
Marco
Solved:
Last query:
Last reply:
Revision history for this message
Michael Terry (mterry) said : 		#1

Hello! OK so some of the issues with Deja Dup not seeing files on Google Drive is probably due to a Google API key transition.

Versions before 40.0 used GNOME's keys, which could see all files. Versions 40.0 and after used our own keys which can only see files it writes. It has some support for transitioning from old keys to new keys, as long as the GNOME keys are still available. But once that's gone (like if you reinstall), it can only see its own files.

But OK, you downloaded the files and that's a good move when things aren't working.

But I don't quite understand some of what you're talking about all the copies. You're saying you have 5180 copies of a single file duplicity-inc.20200429T143036Z.to.20200429T1620.gpg?

Revision history for this message
Marco (donansdon) said : 		#2

Hello Michael, thank you for your answer and sorry for the delay in my reply.

Thanks for sharing this detail with me, I had no idea that the keys would have changed during a reinstall.

Eventually I have found out what happened:

1) when I tried to download from my browser the whole backup folder (~270GB), after a while I could download around 80 zip folders containing the Déja Dup archives, each of a size of 2GB. Every time I asked GDrive to download the folder, only a part of it (around 60% each time).
2) to overcome this issue I thought that a smart way to do that would be to ask Google to download all of my data, which I could actually do and this time they were complete. When I had extracted everything I found out that the DD's archives' name were truncated after the n-th character, therefore what remained of the name was appended with the progressive numbering.
3) my solution to this problem was to download the backups folder from the Drive's browser (I could not install the proprietary software nor Drive backups or whatever is its name), extract it somewhere, and repeat. Each time the DD's archives were different and after around 10 rounds there were still around 20 archives that were freshly included (I always said 'skip all' when asked to replace the existing files. After that I run a simple python script that I have made to check which archive were missing, and download them singularly from the browser.

What is curious is that the restore point obtained in this way was older than expected (~2 months) and DD issued an error at the end of the restore saying something like "some archives were impossible to restore). Luckily I could repair the damage somehow with an older backup and now I am more or less ok with the state of the things.

I guess the root causes for this situation were both the change in keys and bigG technical issues with downloading the files.

I think it's fair to mark this problem as solved, I don't know if in this problem there might be some ideas to improve Déjà Dup.

Anyways thank you for making this tool.

Revision history for this message
Michael Terry (mterry) said : 		#3

Yikes, that sounds like a lot of work. Glad you got it sorted though!