Restore (list files) takes too long

Martin     Feb 17 5:16PM 2018 GUI

Hi,

I'm trying to restore a few files from my backup set, but when I click "List Files" the GUI just hangs until I give up and kill it (some 30 minutes later).

Is it supposed to take such a long time and I should just keep waiting, or is this a bug?

Martin


gchen    Feb 17 10:25PM 2018

You can try the 2.1.0 beta which has an optimized workflow for listing and restoring: https://duplicacy.com/issue?id=5708573640228864.

At least you'll have a progress indicator showing the number of files processed vs the total number of files.


Martin    Feb 18 10:11AM 2018

Thanks - I've tried the 2.1.0 beta. Definitely better having the indeterminate progress bar rather than just freezing.

I don't see any info on number of files processed though, and the operation still seems to never complete which makes restore impossible :(


gchen    Feb 18 11:14PM 2018

If you don't see the number of files, then it is probably still downloading the file list. Can you run the CLI version to see how long it takes to list the files?

cd /path/to/repository
duplicacy list -r revision -files

The duplicacy executable can be downloaded from https://github.com/gilbertchen/duplicacy/releases


Martin    Feb 23 4:59AM 2018

Running that command took about 10 minutes.

Piping to wc took less time (about 1 minute) and gave a count of 2248064.


Martin    Feb 27 9:33AM 2018

Any more thoughts on what the issue in the GUI could be?


gchen    Feb 27 11:45AM 2018

Can you remove the content of .duplicacy/cache and run the list command (with the -files option) to see how fast it is? The code in the GUI version to parse the CLI output is known to be inefficient -- I just wanted to know how much slower it is without the cache effect.


Martin    Mar 4 10:28AM 2018

Made a little difference - piping to wc was around 2 minutes whilst letting it print the output took 16 minutes.


gchen    Mar 5 10:01PM 2018

I plan to optimize the code that parses the CLI output in version 2.1.1.


Martin    Mar 6 10:24AM 2018

Thanks - let me know if you want me to test anything when you have it ready.


Martin    Apr 22 12:58PM 2018

Any update on when 2.1.1 will be released with the optimisations?


gchen    Apr 23 10:39AM 2018

Sorry I haven't started working on 2.1.1. I plan to find some time next month.


Martin    Jun 12 2:30AM 2018

Hey. Any update on this issue?


gchen    Jun 12 10:19PM 2018

I looked into the the code that parses the output from restore and realized that it is already as efficient as it can get. The bottleneck is in populating the file list, but fixing it is much harder than I thought. First, we should change the order files and folders are listed in the snapshot file, so that we don't need to download the entire snapshot file to get the top-level files and folders. Second, the tree list in wxWidgets doesn't provide a way to create the top-level items first and then expand others on demand.

Hopefully we'll be able to fix this issue in the new web-based GUI.