Charles Apr 19 9:33PM 2017 GUI
When backing up to Google Cloud Storage I keep receiving timeouts that stop the backup process.
I receive a dialog along the lines of ERROR Failed to upload chunk [somechunkhash]: Put https://storage.googleapis.com/[location]/[chunkHash] dial tcp [someIPAddress/notMine]:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
This past time I received 3 put messages like this and 1 head message. All in the same dialog.
The log does not reflect these messages though. The last line is simply Failed to find the path for chunk [someChunkHash]: Head https://storage.googleapis.com/[location]/[chunkHash]: dial tcp [someIPAddress/NotMine]
The file before the error was slightly larger if that makes any difference (I do not remember the sizes of the last file in the log before this particular occurrence). 5,512,787,968 bytes. I have gigabit upload speed. Let me know if you need anymore information.
gchen Apr 20 8:42AM 2017
I'm running some tests to reproduce it.
gchen Apr 20 8:56PM 2017
All my tests passed without errors.
This looks more like temporary network issues. Did these errors happen when the backup was starting, or in the middle after the backup had been running for a while?
Charles Apr 20 11:20PM 2017
I received this error on 2 different computers on 4 or 5 separate occasions now. I am using coldline storage. I have not had troubles with cloudberry using a similar setup. It happens after the backup has been running for a little while. The error interrupts the backup and it does not completely finish. I would say I've barely gotten through my larger backup, but the smaller one did eventually finish after several runs. If the exception could at least be handled in a way that the backup would continue to run, that would be nice. It does make me a little worried about the integrity of the upload though. Is there a way to verify already uploaded data?
Charles Apr 21 8:55AM 2017
It's not always a big file. This past time it was after a 39 MB file.
gchen Apr 21 9:45AM 2017
I didn't run the tests against the coldline storage. I'll try it now.
You can run this command to verify existing backups:
duplicacy check -files
Charles Apr 24 12:11PM 2017
Any luck reproducing this?
I have gigabit internet (up and down). If you are having troubles reproducing it, I wonder if it could be a race condition that is harder to observe. I could try limiting the upload speed, though would prefer not to limit it at all.
It could for instance be a connection time out since my pc may be spending more time generating packets than uploading, leaving the connection idle for too long.
Charles Apr 24 12:44PM 2017
Maybe not. Looking at the log files using a text editor, these lines are within a minute of each other.
2017-04-24 05:08:15.445 INFO UPLOAD_PROGRESS Uploaded chunk 109527 size 2638680, 46.39MB/s 1 day 05:50:44 8.6%
2017-04-24 05:08:36.468 ERROR UPLOAD_CHUNK Failed to upload the chunk 3dd5fb094eb7cc81c96b3adb9adb59d6967a253dc3a0aff22f355ab00050ac78: Put https://storage.googleapis.com/[redacted]/[redacted]/chunks/3dd5fb094eb7cc81c96b3adb9adb59d6967a253dc3a0aff22f355ab00050ac78: dial tcp 173.194.219.128:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
Charles Apr 24 1:05PM 2017
Perhaps it is still a rate issue though. Sort of like DOS prevention or something. To get an idea of how many requests I might be making per second, see the following 3 minutes worth of logs.
I also found this documentation about possible errors using Google Cloud Storage API and possible error 429--Too Many Requests https://cloud.google.com/storage/docs/json_api/v1/status-codes#429_Too_Many_Requests
It may be that the interoperability just doesn't return a pretty error message and cuts the connection.
Based on the following from that documentation
Trying to mutate a bucket's or object's metadata more than once per second.
Trying to compose objects at over 200 components per second.
It sounds to me like, if the name is not checked for with each upload (limit 2 per second), then it should be fine (because my 1 computer could never reach 200 puts per second). I may be interpreting this incorrectly though. I am not familiar with this stuff.
edit: Reading this again, I don't know that a check if file exists should count as a mutate/ modify. I do have a sneaky suspicion that I'm getting hit with some sort of rate limit though. As mentioned below, I will let you know if applying a rate limit helps.
Does passing in the "exclusive" flag skip the existing file check? I may just need to set this up using cli so I can pass that in if so. Limiting the speed would be the other option, but I couldn't really guarantee a number of API calls via bandwidth.
2017-04-21 20:33:00.037 INFO UPLOAD_PROGRESS Uploaded chunk 58966 size 1272683, 45.44MB/s 1 day 07:41:08 5.0%
2017-04-21 20:33:00.719 INFO UPLOAD_PROGRESS Uploaded chunk 58967 size 5919729, 45.45MB/s 1 day 07:41:05 5.0%
2017-04-21 20:33:01.098 INFO UPLOAD_PROGRESS Uploaded chunk 58968 size 10984396, 45.44MB/s 1 day 07:41:20 5.0%
2017-04-21 20:33:01.417 INFO UPLOAD_PROGRESS Uploaded chunk 58969 size 6616330, 45.44MB/s 1 day 07:41:17 5.0%
2017-04-21 20:33:01.740 INFO UPLOAD_PROGRESS Uploaded chunk 58970 size 4770930, 45.44MB/s 1 day 07:41:15 5.0%
2017-04-21 20:33:02.040 INFO UPLOAD_PROGRESS Uploaded chunk 58971 size 2209917, 45.43MB/s 1 day 07:41:33 5.0%
2017-04-21 20:33:02.341 INFO UPLOAD_PROGRESS Uploaded chunk 58972 size 1951206, 45.43MB/s 1 day 07:41:32 5.0%
2017-04-21 20:33:02.639 INFO UPLOAD_PROGRESS Uploaded chunk 58973 size 3481554, 45.43MB/s 1 day 07:41:31 5.0%
2017-04-21 20:33:03.072 INFO UPLOAD_PROGRESS Uploaded chunk 58974 size 14677418, 45.43MB/s 1 day 07:41:43 5.0%
2017-04-21 20:33:03.470 INFO UPLOAD_PROGRESS Uploaded chunk 58975 size 12931356, 45.43MB/s 1 day 07:41:38 5.0%
2017-04-21 20:33:03.470 INFO UPLOAD_PROGRESS Skipped chunk 58977 size 2425844, 45.43MB/s 1 day 07:41:37 5.0%
2017-04-21 20:33:03.591 INFO UPLOAD_PROGRESS Skipped chunk 58978 size 5970259, 45.43MB/s 1 day 07:41:35 5.0%
2017-04-21 20:33:03.625 INFO UPLOAD_PROGRESS Skipped chunk 58979 size 2089306, 45.43MB/s 1 day 07:41:34 5.0%
2017-04-21 20:33:03.817 INFO UPLOAD_PROGRESS Uploaded chunk 58976 size 9302021, 45.43MB/s 1 day 07:41:30 5.0%
2017-04-21 20:33:04.383 INFO UPLOAD_PROGRESS Uploaded chunk 58980 size 1207809, 45.43MB/s 1 day 07:41:48 5.0%
2017-04-21 20:33:04.383 INFO UPLOAD_PROGRESS Skipped chunk 58982 size 11185413, 45.43MB/s 1 day 07:41:44 5.0%
2017-04-21 20:33:04.537 INFO UPLOAD_PROGRESS Skipped chunk 58983 size 7455233, 45.43MB/s 1 day 07:41:40 5.0%
2017-04-21 20:33:04.715 INFO UPLOAD_PROGRESS Skipped chunk 58984 size 9700973, 45.43MB/s 1 day 07:41:36 5.0%
2017-04-21 20:33:04.770 INFO UPLOAD_PROGRESS Uploaded chunk 58981 size 3867690, 45.43MB/s 1 day 07:41:35 5.0%
2017-04-21 20:33:04.777 INFO UPLOAD_PROGRESS Skipped chunk 58985 size 3793858, 45.43MB/s 1 day 07:41:33 5.0%
2017-04-21 20:33:04.856 INFO UPLOAD_PROGRESS Skipped chunk 58986 size 4036820, 45.43MB/s 1 day 07:41:31 5.0%
2017-04-21 20:33:04.882 INFO UPLOAD_PROGRESS Skipped chunk 58987 size 1576452, 45.43MB/s 1 day 07:41:31 5.0%
2017-04-21 20:33:05.031 INFO UPLOAD_PROGRESS Skipped chunk 58989 size 4563305, 45.43MB/s 1 day 07:41:48 5.0%
2017-04-21 20:33:05.062 INFO UPLOAD_PROGRESS Skipped chunk 58990 size 2049545, 45.43MB/s 1 day 07:41:47 5.0%
2017-04-21 20:33:05.169 INFO UPLOAD_PROGRESS Skipped chunk 58991 size 5731868, 45.43MB/s 1 day 07:41:45 5.0%
2017-04-21 20:33:05.404 INFO UPLOAD_PROGRESS Uploaded chunk 58988 size 2269745, 45.43MB/s 1 day 07:41:44 5.0%
2017-04-21 20:33:05.777 INFO UPLOAD_PROGRESS Uploaded chunk 58992 size 3021416, 45.43MB/s 1 day 07:41:42 5.0%
2017-04-21 20:33:05.777 INFO UPLOAD_PROGRESS Skipped chunk 58994 size 9304856, 45.43MB/s 1 day 07:41:38 5.0%
2017-04-21 20:33:06.045 INFO UPLOAD_PROGRESS Uploaded chunk 58993 size 2440414, 45.42MB/s 1 day 07:41:56 5.0%
2017-04-21 20:33:06.379 INFO UPLOAD_PROGRESS Uploaded chunk 58995 size 7914152, 45.42MB/s 1 day 07:41:53 5.0%
2017-04-21 20:33:06.628 INFO UPLOAD_PROGRESS Uploaded chunk 58996 size 2440427, 45.43MB/s 1 day 07:41:52 5.0%
2017-04-21 20:33:06.920 INFO UPLOAD_PROGRESS Uploaded chunk 58997 size 2688368, 45.43MB/s 1 day 07:41:51 5.0%
2017-04-21 20:33:07.212 INFO UPLOAD_PROGRESS Uploaded chunk 58998 size 3971656, 45.42MB/s 1 day 07:42:08 5.0%
2017-04-21 20:33:07.459 INFO UPLOAD_PROGRESS Uploaded chunk 58999 size 2741504, 45.42MB/s 1 day 07:42:07 5.0%
2017-04-21 20:33:07.687 INFO UPLOAD_PROGRESS Uploaded chunk 59000 size 1207927, 45.42MB/s 1 day 07:42:07 5.0%
2017-04-21 20:33:08.036 INFO UPLOAD_PROGRESS Uploaded chunk 59001 size 11725219, 45.41MB/s 1 day 07:42:21 5.0%
2017-04-21 20:33:08.960 INFO UPLOAD_PROGRESS Uploaded chunk 59002 size 10897283, 45.42MB/s 1 day 07:42:16 5.0%
2017-04-21 20:33:09.206 INFO UPLOAD_PROGRESS Uploaded chunk 59003 size 1460328, 45.41MB/s 1 day 07:42:34 5.0%
2017-04-21 20:33:09.487 INFO UPLOAD_PROGRESS Uploaded chunk 59004 size 2801078, 45.41MB/s 1 day 07:42:33 5.0%
2017-04-21 20:33:09.862 INFO UPLOAD_PROGRESS Uploaded chunk 59005 size 3401958, 45.41MB/s 1 day 07:42:32 5.0%
2017-04-21 20:33:10.138 INFO UPLOAD_PROGRESS Uploaded chunk 59006 size 1706023, 45.40MB/s 1 day 07:42:50 5.0%
2017-04-21 20:33:10.436 INFO UPLOAD_PROGRESS Uploaded chunk 59007 size 1088772, 45.40MB/s 1 day 07:42:50 5.0%
2017-04-21 20:33:10.705 INFO UPLOAD_PROGRESS Uploaded chunk 59008 size 2108760, 45.40MB/s 1 day 07:42:49 5.0%
2017-04-21 20:33:11.037 INFO UPLOAD_PROGRESS Uploaded chunk 59009 size 4573017, 45.40MB/s 1 day 07:43:06 5.0%
2017-04-21 20:33:11.339 INFO UPLOAD_PROGRESS Uploaded chunk 59010 size 2583117, 45.40MB/s 1 day 07:43:05 5.0%
2017-04-21 20:33:11.677 INFO UPLOAD_PROGRESS Uploaded chunk 59011 size 2804437, 45.40MB/s 1 day 07:43:04 5.0%
2017-04-21 20:33:12.109 INFO UPLOAD_PROGRESS Uploaded chunk 59012 size 6914123, 45.39MB/s 1 day 07:43:20 5.0%
2017-04-21 20:33:12.446 INFO UPLOAD_PROGRESS Uploaded chunk 59013 size 1575288, 45.39MB/s 1 day 07:43:19 5.0%
2017-04-21 20:33:12.677 INFO UPLOAD_PROGRESS Uploaded chunk 59014 size 1377728, 45.39MB/s 1 day 07:43:18 5.0%
2017-04-21 20:33:13.132 INFO UPLOAD_PROGRESS Uploaded chunk 59015 size 1484582, 45.38MB/s 1 day 07:43:37 5.0%
2017-04-21 20:33:13.654 INFO UPLOAD_PROGRESS Uploaded chunk 59016 size 11566296, 45.38MB/s 1 day 07:43:32 5.0%
2017-04-21 20:33:14.141 INFO UPLOAD_PROGRESS Uploaded chunk 59017 size 9329107, 45.38MB/s 1 day 07:43:47 5.0%
2017-04-21 20:33:14.435 INFO UPLOAD_PROGRESS Uploaded chunk 59018 size 2417138, 45.38MB/s 1 day 07:43:46 5.0%
2017-04-21 20:33:14.889 INFO UPLOAD_PROGRESS Uploaded chunk 59019 size 13809365, 45.38MB/s 1 day 07:43:40 5.0%
2017-04-21 20:33:15.223 INFO UPLOAD_PROGRESS Uploaded chunk 59020 size 3774990, 45.37MB/s 1 day 07:43:58 5.0%
2017-04-21 20:33:15.575 INFO UPLOAD_PROGRESS Uploaded chunk 59021 size 3672965, 45.37MB/s 1 day 07:43:56 5.0%
2017-04-21 20:33:15.787 INFO UPLOAD_PROGRESS Uploaded chunk 59022 size 1091372, 45.38MB/s 1 day 07:43:56 5.0%
2017-04-21 20:33:15.787 INFO UPLOAD_PROGRESS Skipped chunk 59024 size 1210841, 45.38MB/s 1 day 07:43:55 5.0%
2017-04-21 20:33:15.808 INFO UPLOAD_PROGRESS Skipped chunk 59025 size 1518294, 45.38MB/s 1 day 07:43:54 5.0%
2017-04-21 20:33:16.098 INFO UPLOAD_PROGRESS Uploaded chunk 59023 size 3221036, 45.37MB/s 1 day 07:44:12 5.0%
2017-04-21 20:33:16.408 INFO UPLOAD_PROGRESS Uploaded chunk 59026 size 6954244, 45.37MB/s 1 day 07:44:09 5.0%
2017-04-21 20:33:16.706 INFO UPLOAD_PROGRESS Uploaded chunk 59027 size 1503095, 45.37MB/s 1 day 07:44:08 5.0%
2017-04-21 20:33:16.999 INFO UPLOAD_PROGRESS Uploaded chunk 59028 size 4056558, 45.37MB/s 1 day 07:44:07 5.0%
2017-04-21 20:33:17.380 INFO UPLOAD_PROGRESS Uploaded chunk 59029 size 5281851, 45.36MB/s 1 day 07:44:24 5.0%
2017-04-21 20:33:17.599 INFO UPLOAD_PROGRESS Uploaded chunk 59030 size 1178119, 45.36MB/s 1 day 07:44:23 5.0%
2017-04-21 20:33:18.228 INFO UPLOAD_PROGRESS Uploaded chunk 59031 size 10634466, 45.36MB/s 1 day 07:44:38 5.0%
2017-04-21 20:33:18.530 INFO UPLOAD_PROGRESS Uploaded chunk 59032 size 1804842, 45.36MB/s 1 day 07:44:37 5.0%
2017-04-21 20:33:18.841 INFO UPLOAD_PROGRESS Uploaded chunk 59033 size 5112876, 45.36MB/s 1 day 07:44:35 5.0%
2017-04-21 20:33:19.128 INFO UPLOAD_PROGRESS Uploaded chunk 59034 size 3685344, 45.35MB/s 1 day 07:44:52 5.0%
2017-04-21 20:33:19.520 INFO UPLOAD_PROGRESS Uploaded chunk 59035 size 7319901, 45.35MB/s 1 day 07:44:49 5.0%
2017-04-21 20:33:19.882 INFO UPLOAD_PROGRESS Uploaded chunk 59036 size 5167928, 45.35MB/s 1 day 07:44:47 5.0%
2017-04-21 20:33:20.340 INFO UPLOAD_PROGRESS Uploaded chunk 59037 size 7722673, 45.35MB/s 1 day 07:45:03 5.0%
2017-04-21 20:33:20.581 INFO UPLOAD_PROGRESS Uploaded chunk 59038 size 1748492, 45.35MB/s 1 day 07:45:02 5.0%
2017-04-21 20:33:20.582 INFO UPLOAD_PROGRESS Skipped chunk 59040 size 1804842, 45.35MB/s 1 day 07:45:01 5.0%
2017-04-21 20:33:21.057 INFO UPLOAD_PROGRESS Uploaded chunk 59039 size 5685196, 45.34MB/s 1 day 07:45:18 5.0%
2017-04-21 20:33:21.468 INFO UPLOAD_PROGRESS Uploaded chunk 59041 size 3899178, 45.34MB/s 1 day 07:45:16 5.0%
2017-04-21 20:33:21.938 INFO UPLOAD_PROGRESS Uploaded chunk 59042 size 1213698, 45.34MB/s 1 day 07:45:16 5.0%
2017-04-21 20:33:22.228 INFO UPLOAD_PROGRESS Uploaded chunk 59043 size 5340565, 45.34MB/s 1 day 07:45:32 5.0%
2017-04-21 20:33:22.481 INFO UPLOAD_PROGRESS Uploaded chunk 59044 size 1199504, 45.34MB/s 1 day 07:45:32 5.0%
2017-04-21 20:33:22.482 INFO UPLOAD_PROGRESS Skipped chunk 59046 size 1804842, 45.34MB/s 1 day 07:45:31 5.0%
2017-04-21 20:33:22.778 INFO UPLOAD_PROGRESS Uploaded chunk 59045 size 4821003, 45.34MB/s 1 day 07:45:29 5.0%
2017-04-21 20:33:23.034 INFO UPLOAD_PROGRESS Uploaded chunk 59047 size 3371478, 45.33MB/s 1 day 07:45:47 5.0%
2017-04-21 20:33:23.319 INFO UPLOAD_PROGRESS Uploaded chunk 59048 size 1741398, 45.33MB/s 1 day 07:45:46 5.0%
2017-04-21 20:33:23.634 INFO UPLOAD_PROGRESS Uploaded chunk 59049 size 6316801, 45.33MB/s 1 day 07:45:43 5.0%
2017-04-21 20:33:24.160 INFO UPLOAD_PROGRESS Uploaded chunk 59050 size 14360197, 45.33MB/s 1 day 07:45:56 5.0%
2017-04-21 20:33:24.509 INFO UPLOAD_PROGRESS Uploaded chunk 59051 size 6649865, 45.33MB/s 1 day 07:45:53 5.0%
2017-04-21 20:33:24.818 INFO UPLOAD_PROGRESS Uploaded chunk 59052 size 8696336, 45.33MB/s 1 day 07:45:50 5.0%
2017-04-21 20:33:25.067 INFO UPLOAD_PROGRESS Uploaded chunk 59053 size 1966136, 45.32MB/s 1 day 07:46:08 5.0%
2017-04-21 20:33:25.532 INFO UPLOAD_PROGRESS Uploaded chunk 59054 size 9397132, 45.32MB/s 1 day 07:46:04 5.0%
2017-04-21 20:33:25.968 INFO UPLOAD_PROGRESS Uploaded chunk 59055 size 14621740, 45.33MB/s 1 day 07:45:58 5.0%
2017-04-21 20:33:26.369 INFO UPLOAD_PROGRESS Uploaded chunk 59056 size 4806236, 45.32MB/s 1 day 07:46:15 5.0%
2017-04-21 20:33:26.725 INFO UPLOAD_PROGRESS Uploaded chunk 59057 size 8327735, 45.32MB/s 1 day 07:46:11 5.0%
2017-04-21 20:33:27.011 INFO UPLOAD_PROGRESS Uploaded chunk 59058 size 4013463, 45.31MB/s 1 day 07:46:28 5.0%
2017-04-21 20:33:27.708 INFO UPLOAD_PROGRESS Uploaded chunk 59059 size 9942676, 45.31MB/s 1 day 07:46:24 5.0%
2017-04-21 20:33:28.226 INFO UPLOAD_PROGRESS Uploaded chunk 59060 size 9333426, 45.31MB/s 1 day 07:46:39 5.0%
2017-04-21 20:33:28.617 INFO UPLOAD_PROGRESS Uploaded chunk 59061 size 10220200, 45.31MB/s 1 day 07:46:35 5.0%
2017-04-21 20:33:29.143 INFO UPLOAD_PROGRESS Uploaded chunk 59062 size 8630475, 45.30MB/s 1 day 07:46:50 5.0%
2017-04-21 20:33:29.454 INFO UPLOAD_PROGRESS Uploaded chunk 59063 size 6653975, 45.31MB/s 1 day 07:46:47 5.0%
2017-04-21 20:33:29.947 INFO UPLOAD_PROGRESS Uploaded chunk 59064 size 8232034, 45.31MB/s 1 day 07:46:44 5.0%
2017-04-21 20:33:30.333 INFO UPLOAD_PROGRESS Uploaded chunk 59065 size 3373375, 45.30MB/s 1 day 07:47:02 5.0%
2017-04-21 20:33:30.579 INFO UPLOAD_PROGRESS Uploaded chunk 59066 size 1282448, 45.30MB/s 1 day 07:47:01 5.0%
2017-04-21 20:33:31.014 INFO UPLOAD_PROGRESS Uploaded chunk 59067 size 9373441, 45.29MB/s 1 day 07:47:16 5.0%
2017-04-21 20:33:31.317 INFO UPLOAD_PROGRESS Uploaded chunk 59068 size 1739316, 45.29MB/s 1 day 07:47:15 5.0%
2017-04-21 20:33:31.639 INFO UPLOAD_PROGRESS Uploaded chunk 59069 size 2421590, 45.29MB/s 1 day 07:47:14 5.0%
2017-04-21 20:33:31.926 INFO UPLOAD_PROGRESS Uploaded chunk 59070 size 1395614, 45.29MB/s 1 day 07:47:14 5.0%
2017-04-21 20:33:32.318 INFO UPLOAD_PROGRESS Uploaded chunk 59071 size 5718454, 45.29MB/s 1 day 07:47:30 5.0%
2017-04-21 20:33:32.754 INFO UPLOAD_PROGRESS Uploaded chunk 59072 size 11920131, 45.29MB/s 1 day 07:47:25 5.0%
2017-04-21 20:33:33.086 INFO UPLOAD_PROGRESS Uploaded chunk 59073 size 4554269, 45.28MB/s 1 day 07:47:42 5.0%
2017-04-21 20:33:33.552 INFO UPLOAD_PROGRESS Uploaded chunk 59074 size 8835313, 45.28MB/s 1 day 07:47:39 5.0%
2017-04-21 20:33:34.012 INFO UPLOAD_PROGRESS Uploaded chunk 59075 size 3611584, 45.28MB/s 1 day 07:47:56 5.0%
2017-04-21 20:33:34.321 INFO UPLOAD_PROGRESS Uploaded chunk 59076 size 5583100, 45.28MB/s 1 day 07:47:54 5.0%
2017-04-21 20:33:34.557 INFO UPLOAD_PROGRESS Uploaded chunk 59077 size 1888287, 45.28MB/s 1 day 07:47:53 5.0%
2017-04-21 20:33:35.325 INFO UPLOAD_PROGRESS Uploaded chunk 59078 size 2042239, 45.27MB/s 1 day 07:48:11 5.0%
2017-04-21 20:33:35.593 INFO UPLOAD_PROGRESS Uploaded chunk 59079 size 3020220, 45.27MB/s 1 day 07:48:10 5.0%
2017-04-21 20:33:35.918 INFO UPLOAD_PROGRESS Uploaded chunk 59080 size 3699206, 45.27MB/s 1 day 07:48:08 5.0%
2017-04-21 20:33:36.185 INFO UPLOAD_PROGRESS Uploaded chunk 59081 size 1292687, 45.27MB/s 1 day 07:48:27 5.0%
2017-04-21 20:33:36.461 INFO UPLOAD_PROGRESS Uploaded chunk 59082 size 2806041, 45.27MB/s 1 day 07:48:25 5.0%
2017-04-21 20:33:36.918 INFO UPLOAD_PROGRESS Uploaded chunk 59083 size 8764221, 45.27MB/s 1 day 07:48:22 5.0%
2017-04-21 20:33:37.342 INFO UPLOAD_PROGRESS Uploaded chunk 59084 size 6095347, 45.26MB/s 1 day 07:48:38 5.0%
2017-04-21 20:33:37.604 INFO UPLOAD_PROGRESS Uploaded chunk 59085 size 2691408, 45.26MB/s 1 day 07:48:37 5.0%
2017-04-21 20:33:38.048 INFO UPLOAD_PROGRESS Uploaded chunk 59086 size 8398847, 45.25MB/s 1 day 07:48:52 5.0%
2017-04-21 20:33:38.436 INFO UPLOAD_PROGRESS Uploaded chunk 59087 size 5188830, 45.26MB/s 1 day 07:48:50 5.0%
2017-04-21 20:33:38.759 INFO UPLOAD_PROGRESS Uploaded chunk 59088 size 3116646, 45.26MB/s 1 day 07:48:49 5.0%
2017-04-21 20:33:39.351 INFO UPLOAD_PROGRESS Uploaded chunk 59089 size 2516145, 45.25MB/s 1 day 07:49:07 5.0%
2017-04-21 20:33:39.713 INFO UPLOAD_PROGRESS Uploaded chunk 59090 size 1554663, 45.25MB/s 1 day 07:49:06 5.0%
2017-04-21 20:33:40.038 INFO UPLOAD_PROGRESS Uploaded chunk 59091 size 1632559, 45.24MB/s 1 day 07:49:24 5.0%
2017-04-21 20:33:40.816 INFO UPLOAD_PROGRESS Uploaded chunk 59092 size 6186519, 45.24MB/s 1 day 07:49:22 5.0%
2017-04-21 20:33:41.325 INFO UPLOAD_PROGRESS Uploaded chunk 59093 size 7763392, 45.24MB/s 1 day 07:49:37 5.0%
2017-04-21 20:33:41.829 INFO UPLOAD_PROGRESS Uploaded chunk 59094 size 9184048, 45.24MB/s 1 day 07:49:34 5.0%
2017-04-21 20:33:42.097 INFO UPLOAD_PROGRESS Uploaded chunk 59095 size 1753242, 45.23MB/s 1 day 07:49:52 5.0%
2017-04-21 20:33:42.360 INFO UPLOAD_PROGRESS Uploaded chunk 59096 size 1914000, 45.23MB/s 1 day 07:49:51 5.0%
2017-04-21 20:33:42.668 INFO UPLOAD_PROGRESS Uploaded chunk 59097 size 3284890, 45.23MB/s 1 day 07:49:50 5.0%
2017-04-21 20:33:42.924 INFO UPLOAD_PROGRESS Uploaded chunk 59098 size 1450370, 45.23MB/s 1 day 07:49:49 5.0%
2017-04-21 20:33:43.348 INFO UPLOAD_PROGRESS Uploaded chunk 59099 size 4449164, 45.23MB/s 1 day 07:50:06 5.0%
2017-04-21 20:33:44.238 INFO UPLOAD_PROGRESS Uploaded chunk 59100 size 2151933, 45.22MB/s 1 day 07:50:24 5.0%
2017-04-21 20:33:44.965 INFO UPLOAD_PROGRESS Uploaded chunk 59101 size 1339583, 45.22MB/s 1 day 07:50:24 5.0%
2017-04-21 20:33:45.308 INFO UPLOAD_PROGRESS Uploaded chunk 59102 size 2391997, 45.21MB/s 1 day 07:50:42 5.0%
2017-04-21 20:33:45.612 INFO UPLOAD_PROGRESS Uploaded chunk 59103 size 6087603, 45.21MB/s 1 day 07:50:39 5.0%
2017-04-21 20:33:46.021 INFO UPLOAD_PROGRESS Uploaded chunk 59104 size 5432484, 45.21MB/s 1 day 07:50:56 5.0%
2017-04-21 20:33:46.303 INFO UPLOAD_PROGRESS Uploaded chunk 59105 size 1421986, 45.21MB/s 1 day 07:50:55 5.0%
2017-04-21 20:33:46.693 INFO UPLOAD_PROGRESS Uploaded chunk 59106 size 5540417, 45.21MB/s 1 day 07:50:53 5.0%
2017-04-21 20:33:46.942 INFO UPLOAD_PROGRESS Uploaded chunk 59107 size 1097974, 45.21MB/s 1 day 07:50:52 5.0%
2017-04-21 20:33:47.351 INFO UPLOAD_PROGRESS Uploaded chunk 59108 size 1991058, 45.20MB/s 1 day 07:51:10 5.0%
2017-04-21 20:33:48.033 INFO UPLOAD_PROGRESS Uploaded chunk 59109 size 3032007, 45.19MB/s 1 day 07:51:28 5.0%
2017-04-21 20:33:48.326 INFO UPLOAD_PROGRESS Uploaded chunk 59110 size 2499713, 45.19MB/s 1 day 07:51:27 5.0%
2017-04-21 20:33:48.607 INFO UPLOAD_PROGRESS Uploaded chunk 59111 size 1339822, 45.19MB/s 1 day 07:51:26 5.0%
2017-04-21 20:33:49.022 INFO UPLOAD_PROGRESS Uploaded chunk 59112 size 2165280, 45.19MB/s 1 day 07:51:44 5.0%
2017-04-21 20:33:49.342 INFO UPLOAD_PROGRESS Uploaded chunk 59113 size 4545606, 45.19MB/s 1 day 07:51:43 5.0%
2017-04-21 20:33:49.885 INFO UPLOAD_PROGRESS Uploaded chunk 59114 size 4109052, 45.19MB/s 1 day 07:51:41 5.0%
2017-04-21 20:33:50.387 INFO UPLOAD_PROGRESS Uploaded chunk 59115 size 6809613, 45.18MB/s 1 day 07:51:57 5.0%
2017-04-21 20:33:50.659 INFO UPLOAD_PROGRESS Uploaded chunk 59116 size 1350219, 45.18MB/s 1 day 07:51:56 5.0%
2017-04-21 20:33:50.969 INFO UPLOAD_PROGRESS Uploaded chunk 59117 size 2907562, 45.18MB/s 1 day 07:51:55 5.0%
2017-04-21 20:33:51.320 INFO UPLOAD_PROGRESS Uploaded chunk 59118 size 2736925, 45.17MB/s 1 day 07:52:13 5.0%
2017-04-21 20:33:51.618 INFO UPLOAD_PROGRESS Uploaded chunk 59119 size 3360472, 45.18MB/s 1 day 07:52:11 5.0%
2017-04-21 20:33:52.079 INFO UPLOAD_PROGRESS Uploaded chunk 59120 size 12678942, 45.17MB/s 1 day 07:52:25 5.0%
2017-04-21 20:33:52.634 INFO UPLOAD_PROGRESS Uploaded chunk 59121 size 5733345, 45.17MB/s 1 day 07:52:23 5.0%
2017-04-21 20:33:53.032 INFO UPLOAD_PROGRESS Uploaded chunk 59122 size 1315102, 45.16MB/s 1 day 07:52:41 5.0%
2017-04-21 20:33:53.517 INFO UPLOAD_PROGRESS Uploaded chunk 59123 size 9799751, 45.16MB/s 1 day 07:52:37 5.0%
2017-04-21 20:33:53.851 INFO UPLOAD_PROGRESS Uploaded chunk 59124 size 3445414, 45.17MB/s 1 day 07:52:35 5.0%
2017-04-21 20:33:54.113 INFO UPLOAD_PROGRESS Uploaded chunk 59125 size 2004711, 45.16MB/s 1 day 07:52:54 5.0%
2017-04-21 20:33:54.505 INFO UPLOAD_PROGRESS Uploaded chunk 59126 size 1798549, 45.16MB/s 1 day 07:52:53 5.0%
2017-04-21 20:33:54.859 INFO UPLOAD_PROGRESS Uploaded chunk 59127 size 4555948, 45.16MB/s 1 day 07:52:51 5.0%
2017-04-21 20:33:55.256 INFO UPLOAD_PROGRESS Uploaded chunk 59128 size 5752677, 45.15MB/s 1 day 07:53:07 5.0%
2017-04-21 20:33:55.823 INFO UPLOAD_PROGRESS Uploaded chunk 59129 size 11337568, 45.15MB/s 1 day 07:53:03 5.0%
2017-04-21 20:33:56.090 INFO UPLOAD_PROGRESS Uploaded chunk 59130 size 1455943, 45.15MB/s 1 day 07:53:21 5.0%
2017-04-21 20:33:56.326 INFO UPLOAD_PROGRESS Uploaded chunk 59131 size 1310487, 45.15MB/s 1 day 07:53:20 5.0%
2017-04-21 20:33:56.797 INFO UPLOAD_PROGRESS Uploaded chunk 59132 size 6070650, 45.15MB/s 1 day 07:53:18 5.0%
2017-04-21 20:33:57.187 INFO UPLOAD_PROGRESS Uploaded chunk 59133 size 13364843, 45.14MB/s 1 day 07:53:31 5.0%
2017-04-21 20:33:58.621 INFO UPLOAD_PROGRESS Uploaded chunk 59134 size 4358840, 45.14MB/s 1 day 07:53:48 5.0%
2017-04-21 20:33:59.126 INFO UPLOAD_PROGRESS Uploaded chunk 59135 size 6532545, 45.13MB/s 1 day 07:54:04 5.0%
2017-04-21 20:33:59.878 INFO UPLOAD_PROGRESS Uploaded chunk 59136 size 2472706, 45.13MB/s 1 day 07:54:03 5.0%
2017-04-21 20:34:00.139 INFO UPLOAD_PROGRESS Uploaded chunk 59137 size 1430654, 45.12MB/s 1 day 07:54:22 5.0%
2017-04-21 20:34:00.369 INFO UPLOAD_PROGRESS Uploaded chunk 59138 size 1291554, 45.12MB/s 1 day 07:54:21 5.0%
2017-04-21 20:34:00.650 INFO UPLOAD_PROGRESS Uploaded chunk 59139 size 1449501, 45.12MB/s 1 day 07:54:21 5.0%
2017-04-21 20:34:00.982 INFO UPLOAD_PROGRESS Uploaded chunk 59140 size 9179590, 45.13MB/s 1 day 07:54:17 5.0%
2017-04-21 20:34:01.262 INFO UPLOAD_PROGRESS Uploaded chunk 59141 size 1637793, 45.12MB/s 1 day 07:54:35 5.0%
2017-04-21 20:34:01.578 INFO UPLOAD_PROGRESS Uploaded chunk 59142 size 5396599, 45.12MB/s 1 day 07:54:33 5.0%
2017-04-21 20:34:02.108 INFO UPLOAD_PROGRESS Uploaded chunk 59143 size 4194154, 45.11MB/s 1 day 07:54:50 5.0%
2017-04-21 20:34:02.837 INFO UPLOAD_PROGRESS Uploaded chunk 59144 size 11234754, 45.11MB/s 1 day 07:54:45 5.0%
2017-04-21 20:34:03.117 INFO UPLOAD_PROGRESS Uploaded chunk 59145 size 1689423, 45.11MB/s 1 day 07:55:03 5.0%
2017-04-21 20:34:03.392 INFO UPLOAD_PROGRESS Uploaded chunk 59146 size 1964860, 45.11MB/s 1 day 07:55:03 5.0%
2017-04-21 20:34:04.205 INFO UPLOAD_PROGRESS Uploaded chunk 59147 size 5454846, 45.10MB/s 1 day 07:55:19 5.0%
2017-04-21 20:34:04.604 INFO UPLOAD_PROGRESS Uploaded chunk 59148 size 3236323, 45.10MB/s 1 day 07:55:18 5.0%
2017-04-21 20:34:04.892 INFO UPLOAD_PROGRESS Uploaded chunk 59149 size 1838063, 45.10MB/s 1 day 07:55:17 5.0%
2017-04-21 20:34:05.385 INFO UPLOAD_PROGRESS Uploaded chunk 59150 size 4471455, 45.09MB/s 1 day 07:55:34 5.0%
2017-04-21 20:34:05.729 INFO UPLOAD_PROGRESS Uploaded chunk 59151 size 2746065, 45.09MB/s 1 day 07:55:33 5.0%
2017-04-21 20:34:06.446 INFO UPLOAD_PROGRESS Uploaded chunk 59152 size 14537990, 45.09MB/s 1 day 07:55:46 5.0%
2017-04-21 20:34:06.702 INFO UPLOAD_PROGRESS Uploaded chunk 59153 size 1292713, 45.09MB/s 1 day 07:55:45 5.0%
2017-04-21 20:34:07.518 INFO UPLOAD_PROGRESS Uploaded chunk 59154 size 9211910, 45.08MB/s 1 day 07:56:00 5.0%
2017-04-21 20:34:07.842 INFO UPLOAD_PROGRESS Uploaded chunk 59155 size 2773644, 45.08MB/s 1 day 07:55:59 5.0%
2017-04-21 20:34:08.187 INFO UPLOAD_PROGRESS Uploaded chunk 59156 size 6943017, 45.08MB/s 1 day 07:56:15 5.0%
2017-04-21 20:34:08.482 INFO UPLOAD_PROGRESS Uploaded chunk 59157 size 2603510, 45.08MB/s 1 day 07:56:14 5.0%
2017-04-21 20:34:08.791 INFO UPLOAD_PROGRESS Uploaded chunk 59158 size 6884787, 45.08MB/s 1 day 07:56:11 5.0%
2017-04-21 20:34:09.073 INFO UPLOAD_PROGRESS Uploaded chunk 59159 size 2098281, 45.07MB/s 1 day 07:56:29 5.0%
2017-04-21 20:34:09.417 INFO UPLOAD_PROGRESS Uploaded chunk 59160 size 1786951, 45.07MB/s 1 day 07:56:28 5.0%
2017-04-21 20:34:09.853 INFO UPLOAD_PROGRESS Uploaded chunk 59161 size 3415003, 45.07MB/s 1 day 07:56:27 5.0%
2017-04-21 20:34:10.229 INFO UPLOAD_PROGRESS Uploaded chunk 59162 size 4886306, 45.07MB/s 1 day 07:56:44 5.0%
2017-04-21 20:34:10.571 INFO UPLOAD_PROGRESS Uploaded chunk 59163 size 1307286, 45.07MB/s 1 day 07:56:43 5.0%
2017-04-21 20:34:11.305 INFO UPLOAD_PROGRESS Uploaded chunk 59164 size 15635282, 45.06MB/s 1 day 07:56:56 5.0%
2017-04-21 20:34:11.575 INFO UPLOAD_PROGRESS Uploaded chunk 59165 size 2273763, 45.06MB/s 1 day 07:56:55 5.0%
2017-04-21 20:34:11.953 INFO UPLOAD_PROGRESS Uploaded chunk 59166 size 5081883, 45.06MB/s 1 day 07:56:52 5.0%
2017-04-21 20:34:12.280 INFO UPLOAD_PROGRESS Uploaded chunk 59167 size 4691735, 45.06MB/s 1 day 07:57:09 5.0%
2017-04-21 20:34:12.720 INFO UPLOAD_PROGRESS Uploaded chunk 59168 size 9218096, 45.06MB/s 1 day 07:57:06 5.0%
2017-04-21 20:34:13.046 INFO UPLOAD_PROGRESS Uploaded chunk 59169 size 5000462, 45.05MB/s 1 day 07:57:22 5.0%
2017-04-21 20:34:13.513 INFO UPLOAD_PROGRESS Uploaded chunk 59170 size 11056200, 45.05MB/s 1 day 07:57:18 5.0%
2017-04-21 20:34:14.059 INFO UPLOAD_PROGRESS Uploaded chunk 59171 size 10067450, 45.05MB/s 1 day 07:57:32 5.0%
2017-04-21 20:34:14.325 INFO UPLOAD_PROGRESS Uploaded chunk 59172 size 1927196, 45.05MB/s 1 day 07:57:32 5.0%
2017-04-21 20:34:14.685 INFO UPLOAD_PROGRESS Uploaded chunk 59173 size 9481864, 45.05MB/s 1 day 07:57:28 5.0%
2017-04-21 20:34:15.017 INFO UPLOAD_PROGRESS Uploaded chunk 59174 size 4376217, 45.04MB/s 1 day 07:57:45 5.0%
2017-04-21 20:34:15.777 INFO UPLOAD_PROGRESS Uploaded chunk 59175 size 3077451, 45.04MB/s 1 day 07:57:43 5.0%
2017-04-21 20:34:16.065 INFO UPLOAD_PROGRESS Uploaded chunk 59176 size 2239451, 45.04MB/s 1 day 07:58:01 5.0%
2017-04-21 20:34:16.426 INFO UPLOAD_PROGRESS Uploaded chunk 59177 size 4510627, 45.04MB/s 1 day 07:57:59 5.0%
2017-04-21 20:34:16.671 INFO UPLOAD_PROGRESS Uploaded chunk 59178 size 1052689, 45.04MB/s 1 day 07:57:59 5.0%
2017-04-21 20:34:17.052 INFO UPLOAD_PROGRESS Uploaded chunk 59179 size 16777216, 45.03MB/s 1 day 07:58:11 5.0%
2017-04-21 20:34:17.609 INFO UPLOAD_PROGRESS Uploaded chunk 59180 size 7957347, 45.03MB/s 1 day 07:58:07 5.0%
2017-04-21 20:34:17.820 INFO UPLOAD_PROGRESS Uploaded chunk 59181 size 1259814, 45.03MB/s 1 day 07:58:07 5.0%
2017-04-21 20:34:18.298 INFO UPLOAD_PROGRESS Uploaded chunk 59182 size 4796350, 45.03MB/s 1 day 07:58:24 5.0%
2017-04-21 20:34:18.623 INFO UPLOAD_PROGRESS Uploaded chunk 59183 size 1800536, 45.03MB/s 1 day 07:58:23 5.0%
2017-04-21 20:34:18.875 INFO UPLOAD_PROGRESS Uploaded chunk 59184 size 2598102, 45.03MB/s 1 day 07:58:22 5.0%
2017-04-21 20:34:19.190 INFO UPLOAD_PROGRESS Uploaded chunk 59185 size 2004606, 45.02MB/s 1 day 07:58:40 5.0%
2017-04-21 20:34:19.547 INFO UPLOAD_PROGRESS Uploaded chunk 59186 size 12143358, 45.02MB/s 1 day 07:58:35 5.0%
2017-04-21 20:34:19.907 INFO UPLOAD_PROGRESS Uploaded chunk 59187 size 3552693, 45.02MB/s 1 day 07:58:33 5.0%
2017-04-21 20:34:20.166 INFO UPLOAD_PROGRESS Uploaded chunk 59188 size 1904980, 45.02MB/s 1 day 07:58:52 5.0%
2017-04-21 20:34:20.452 INFO UPLOAD_PROGRESS Uploaded chunk 59189 size 3471989, 45.02MB/s 1 day 07:58:50 5.0%
2017-04-21 20:34:20.668 INFO UPLOAD_PROGRESS Uploaded chunk 59190 size 1367721, 45.02MB/s 1 day 07:58:49 5.0%
2017-04-21 20:34:21.033 INFO UPLOAD_PROGRESS Uploaded chunk 59191 size 3615536, 45.01MB/s 1 day 07:59:07 5.0%
2017-04-21 20:34:21.505 INFO UPLOAD_PROGRESS Uploaded chunk 59192 size 8563848, 45.01MB/s 1 day 07:59:03 5.0%
2017-04-21 20:34:21.805 INFO UPLOAD_PROGRESS Uploaded chunk 59193 size 2318424, 45.01MB/s 1 day 07:59:02 5.0%
2017-04-21 20:34:22.161 INFO UPLOAD_PROGRESS Uploaded chunk 59194 size 3433244, 45.00MB/s 1 day 07:59:20 5.0%
2017-04-21 20:34:22.628 INFO UPLOAD_PROGRESS Uploaded chunk 59195 size 12190394, 45.01MB/s 1 day 07:59:15 5.0%
2017-04-21 20:34:23.067 INFO UPLOAD_PROGRESS Uploaded chunk 59196 size 10606612, 45.00MB/s 1 day 07:59:29 5.0%
2017-04-21 20:34:23.382 INFO UPLOAD_PROGRESS Uploaded chunk 59197 size 4772312, 45.00MB/s 1 day 07:59:27 5.0%
2017-04-21 20:34:23.650 INFO UPLOAD_PROGRESS Uploaded chunk 59198 size 2978391, 45.00MB/s 1 day 07:59:26 5.0%
2017-04-21 20:34:24.133 INFO UPLOAD_PROGRESS Uploaded chunk 59199 size 2161113, 44.99MB/s 1 day 07:59:44 5.0%
2017-04-21 20:34:24.415 INFO UPLOAD_PROGRESS Uploaded chunk 59200 size 2331237, 44.99MB/s 1 day 07:59:43 5.0%
2017-04-21 20:34:25.464 INFO UPLOAD_PROGRESS Uploaded chunk 59201 size 7682132, 44.99MB/s 1 day 07:59:58 5.0%
2017-04-21 20:34:25.730 INFO UPLOAD_PROGRESS Uploaded chunk 59202 size 2251923, 44.99MB/s 1 day 07:59:58 5.0%
2017-04-21 20:34:26.066 INFO UPLOAD_PROGRESS Uploaded chunk 59203 size 5336368, 44.98MB/s 1 day 08:00:14 5.0%
2017-04-21 20:34:26.482 INFO UPLOAD_PROGRESS Uploaded chunk 59204 size 4355002, 44.98MB/s 1 day 08:00:12 5.0%
2017-04-21 20:34:26.760 INFO UPLOAD_PROGRESS Uploaded chunk 59205 size 1978540, 44.98MB/s 1 day 08:00:12 5.0%
2017-04-21 20:34:27.109 INFO UPLOAD_PROGRESS Uploaded chunk 59206 size 6126527, 44.98MB/s 1 day 08:00:28 5.0%
2017-04-21 20:34:27.922 INFO UPLOAD_PROGRESS Uploaded chunk 59207 size 7926936, 44.98MB/s 1 day 08:00:25 5.0%
2017-04-21 20:34:28.240 INFO UPLOAD_PROGRESS Uploaded chunk 59208 size 4351119, 44.97MB/s 1 day 08:00:42 5.0%
2017-04-21 20:34:28.526 INFO UPLOAD_PROGRESS Uploaded chunk 59209 size 2791963, 44.97MB/s 1 day 08:00:40 5.0%
2017-04-21 20:34:28.816 INFO UPLOAD_PROGRESS Uploaded chunk 59210 size 2556666, 44.97MB/s 1 day 08:00:39 5.0%
2017-04-21 20:34:29.005 INFO UPLOAD_PROGRESS Uploaded chunk 59211 size 1335773, 44.97MB/s 1 day 08:00:58 5.0%
2017-04-21 20:34:29.414 INFO UPLOAD_PROGRESS Uploaded chunk 59212 size 8144009, 44.97MB/s 1 day 08:00:54 5.0%
2017-04-21 20:34:29.692 INFO UPLOAD_PROGRESS Uploaded chunk 59213 size 1374462, 44.97MB/s 1 day 08:00:54 5.0%
2017-04-21 20:34:29.949 INFO UPLOAD_PROGRESS Uploaded chunk 59214 size 1570213, 44.97MB/s 1 day 08:00:53 5.0%
2017-04-21 20:34:30.258 INFO UPLOAD_PROGRESS Uploaded chunk 59215 size 6112459, 44.96MB/s 1 day 08:01:09 5.0%
2017-04-21 20:34:30.511 INFO UPLOAD_PROGRESS Uploaded chunk 59216 size 3022322, 44.96MB/s 1 day 08:01:08 5.0%
2017-04-21 20:34:30.796 INFO UPLOAD_PROGRESS Uploaded chunk 59217 size 5560553, 44.96MB/s 1 day 08:01:06 5.0%
2017-04-21 20:34:31.081 INFO UPLOAD_PROGRESS Uploaded chunk 59218 size 1203171, 44.95MB/s 1 day 08:01:24 5.0%
2017-04-21 20:34:31.916 INFO UPLOAD_PROGRESS Uploaded chunk 59219 size 6533205, 44.96MB/s 1 day 08:01:21 5.0%
2017-04-21 20:34:32.157 INFO UPLOAD_PROGRESS Uploaded chunk 59220 size 2529564, 44.95MB/s 1 day 08:01:39 5.0%
2017-04-21 20:34:32.669 INFO UPLOAD_PROGRESS Uploaded chunk 59221 size 3428021, 44.95MB/s 1 day 08:01:38 5.0%
2017-04-21 20:34:33.076 INFO UPLOAD_PROGRESS Uploaded chunk 59222 size 1524419, 44.94MB/s 1 day 08:01:56 5.0%
2017-04-21 20:34:33.376 INFO UPLOAD_PROGRESS Uploaded chunk 59223 size 2143596, 44.94MB/s 1 day 08:01:55 5.0%
2017-04-21 20:34:33.707 INFO UPLOAD_PROGRESS Uploaded chunk 59224 size 1980546, 44.94MB/s 1 day 08:01:54 5.0%
2017-04-21 20:34:33.928 INFO UPLOAD_PROGRESS Uploaded chunk 59225 size 1141332, 44.94MB/s 1 day 08:01:54 5.0%
2017-04-21 20:34:34.523 INFO UPLOAD_PROGRESS Uploaded chunk 59226 size 6431884, 44.94MB/s 1 day 08:02:10 5.0%
2017-04-21 20:34:34.816 INFO UPLOAD_PROGRESS Uploaded chunk 59227 size 1757256, 44.94MB/s 1 day 08:02:09 5.0%
2017-04-21 20:34:35.261 INFO UPLOAD_PROGRESS Uploaded chunk 59228 size 15506718, 44.93MB/s 1 day 08:02:22 5.0%
2017-04-21 20:34:35.571 INFO UPLOAD_PROGRESS Uploaded chunk 59229 size 1701030, 44.93MB/s 1 day 08:02:21 5.0%
2017-04-21 20:34:35.859 INFO UPLOAD_PROGRESS Uploaded chunk 59230 size 2667067, 44.93MB/s 1 day 08:02:20 5.0%
2017-04-21 20:34:36.195 INFO UPLOAD_PROGRESS Uploaded chunk 59231 size 7263689, 44.93MB/s 1 day 08:02:36 5.0%
2017-04-21 20:34:36.660 INFO UPLOAD_PROGRESS Uploaded chunk 59232 size 8006133, 44.93MB/s 1 day 08:02:32 5.0%
2017-04-21 20:34:37.299 INFO UPLOAD_PROGRESS Uploaded chunk 59233 size 1060309, 44.92MB/s 1 day 08:02:51 5.0%
2017-04-21 20:34:37.557 INFO UPLOAD_PROGRESS Uploaded chunk 59234 size 1175082, 44.92MB/s 1 day 08:02:50 5.0%
2017-04-21 20:34:37.920 INFO UPLOAD_PROGRESS Uploaded chunk 59235 size 6197524, 44.92MB/s 1 day 08:02:48 5.0%
2017-04-21 20:34:38.184 INFO UPLOAD_PROGRESS Uploaded chunk 59236 size 1480101, 44.91MB/s 1 day 08:03:06 5.0%
2017-04-21 20:34:38.477 INFO UPLOAD_PROGRESS Uploaded chunk 59237 size 1609310, 44.91MB/s 1 day 08:03:05 5.0%
2017-04-21 20:34:38.777 INFO UPLOAD_PROGRESS Uploaded chunk 59238 size 2276899, 44.92MB/s 1 day 08:03:04 5.0%
2017-04-21 20:34:39.138 INFO UPLOAD_PROGRESS Uploaded chunk 59239 size 2956458, 44.91MB/s 1 day 08:03:22 5.0%
2017-04-21 20:34:39.630 INFO UPLOAD_PROGRESS Uploaded chunk 59240 size 4701185, 44.91MB/s 1 day 08:03:20 5.0%
2017-04-21 20:34:40.229 INFO UPLOAD_PROGRESS Uploaded chunk 59241 size 3912436, 44.90MB/s 1 day 08:03:37 5.0%
2017-04-21 20:34:40.570 INFO UPLOAD_PROGRESS Uploaded chunk 59242 size 5179430, 44.90MB/s 1 day 08:03:35 5.0%
2017-04-21 20:34:41.172 INFO UPLOAD_PROGRESS Uploaded chunk 59243 size 1069155, 44.90MB/s 1 day 08:03:53 5.0%
2017-04-21 20:34:41.408 INFO UPLOAD_PROGRESS Uploaded chunk 59244 size 1302476, 44.90MB/s 1 day 08:03:53 5.0%
2017-04-21 20:34:41.798 INFO UPLOAD_PROGRESS Uploaded chunk 59245 size 2839146, 44.90MB/s 1 day 08:03:52 5.0%
2017-04-21 20:34:42.035 INFO UPLOAD_PROGRESS Uploaded chunk 59246 size 1373668, 44.89MB/s 1 day 08:04:10 5.0%
2017-04-21 20:34:42.304 INFO UPLOAD_PROGRESS Uploaded chunk 59247 size 1762186, 44.89MB/s 1 day 08:04:09 5.0%
2017-04-21 20:34:42.917 INFO UPLOAD_PROGRESS Uploaded chunk 59248 size 4488537, 44.89MB/s 1 day 08:04:07 5.0%
2017-04-21 20:34:43.273 INFO UPLOAD_PROGRESS Uploaded chunk 59249 size 1902100, 44.88MB/s 1 day 08:04:25 5.0%
2017-04-21 20:34:43.687 INFO UPLOAD_PROGRESS Uploaded chunk 59250 size 5029334, 44.88MB/s 1 day 08:04:23 5.0%
2017-04-21 20:34:44.404 INFO UPLOAD_PROGRESS Uploaded chunk 59251 size 1761414, 44.88MB/s 1 day 08:04:41 5.0%
2017-04-21 20:34:45.175 INFO UPLOAD_PROGRESS Uploaded chunk 59252 size 3374710, 44.87MB/s 1 day 08:04:59 5.0%
2017-04-21 20:34:45.570 INFO UPLOAD_PROGRESS Uploaded chunk 59253 size 6299601, 44.87MB/s 1 day 08:04:56 5.0%
2017-04-21 20:34:46.209 INFO UPLOAD_PROGRESS Uploaded chunk 59254 size 2353321, 44.86MB/s 1 day 08:05:14 5.0%
2017-04-21 20:34:46.499 INFO UPLOAD_PROGRESS Uploaded chunk 59255 size 5080241, 44.86MB/s 1 day 08:05:12 5.0%
2017-04-21 20:34:46.915 INFO UPLOAD_PROGRESS Uploaded chunk 59256 size 4929023, 44.87MB/s 1 day 08:05:10 5.0%
2017-04-21 20:34:47.248 INFO UPLOAD_PROGRESS Uploaded chunk 59257 size 5690342, 44.86MB/s 1 day 08:05:26 5.0%
2017-04-21 20:34:47.571 INFO UPLOAD_PROGRESS Uploaded chunk 59258 size 2214148, 44.86MB/s 1 day 08:05:25 5.0%
2017-04-21 20:34:48.113 INFO UPLOAD_PROGRESS Uploaded chunk 59259 size 16777216, 44.85MB/s 1 day 08:05:37 5.0%
2017-04-21 20:34:48.383 INFO UPLOAD_PROGRESS Uploaded chunk 59260 size 2702731, 44.86MB/s 1 day 08:05:36 5.0%
2017-04-21 20:34:48.735 INFO UPLOAD_PROGRESS Uploaded chunk 59261 size 2745000, 44.86MB/s 1 day 08:05:35 5.0%
2017-04-21 20:34:49.072 INFO UPLOAD_PROGRESS Uploaded chunk 59262 size 5632847, 44.85MB/s 1 day 08:05:51 5.0%
2017-04-21 20:34:49.540 INFO UPLOAD_PROGRESS Uploaded chunk 59263 size 4671684, 44.85MB/s 1 day 08:05:49 5.0%
2017-04-21 20:34:49.863 INFO UPLOAD_PROGRESS Uploaded chunk 59264 size 6202246, 44.85MB/s 1 day 08:05:47 5.0%
2017-04-21 20:34:50.165 INFO UPLOAD_PROGRESS Uploaded chunk 59265 size 3761859, 44.84MB/s 1 day 08:06:04 5.0%
2017-04-21 20:34:50.483 INFO UPLOAD_PROGRESS Uploaded chunk 59266 size 4744087, 44.84MB/s 1 day 08:06:02 5.0%
2017-04-21 20:34:50.725 INFO UPLOAD_PROGRESS Uploaded chunk 59267 size 1345077, 44.85MB/s 1 day 08:06:02 5.0%
2017-04-21 20:34:51.047 INFO UPLOAD_PROGRESS Uploaded chunk 59268 size 3705805, 44.84MB/s 1 day 08:06:19 5.0%
2017-04-21 20:34:51.334 INFO UPLOAD_PROGRESS Uploaded chunk 59269 size 2723989, 44.84MB/s 1 day 08:06:18 5.0%
2017-04-21 20:34:51.718 INFO UPLOAD_PROGRESS Uploaded chunk 59270 size 9241830, 44.84MB/s 1 day 08:06:14 5.0%
2017-04-21 20:34:52.081 INFO UPLOAD_PROGRESS Uploaded chunk 59271 size 10825079, 44.83MB/s 1 day 08:06:28 5.0%
2017-04-21 20:34:52.442 INFO UPLOAD_PROGRESS Uploaded chunk 59272 size 4347315, 44.84MB/s 1 day 08:06:26 5.0%
2017-04-21 20:34:52.750 INFO UPLOAD_PROGRESS Uploaded chunk 59273 size 3489918, 44.84MB/s 1 day 08:06:25 5.0%
2017-04-21 20:34:53.223 INFO UPLOAD_PROGRESS Uploaded chunk 59274 size 11061532, 44.83MB/s 1 day 08:06:39 5.0%
2017-04-21 20:34:53.646 INFO UPLOAD_PROGRESS Uploaded chunk 59275 size 11127358, 44.83MB/s 1 day 08:06:34 5.0%
2017-04-21 20:34:53.959 INFO UPLOAD_PROGRESS Uploaded chunk 59276 size 6344638, 44.83MB/s 1 day 08:06:32 5.0%
2017-04-21 20:34:54.743 INFO UPLOAD_PROGRESS Uploaded chunk 59277 size 3491750, 44.83MB/s 1 day 08:06:49 5.0%
2017-04-21 20:34:55.066 INFO UPLOAD_PROGRESS Uploaded chunk 59278 size 3252529, 44.82MB/s 1 day 08:07:07 5.0%
2017-04-21 20:34:55.390 INFO UPLOAD_PROGRESS Uploaded chunk 59279 size 1891031, 44.82MB/s 1 day 08:07:06 5.0%
2017-04-21 20:34:56.050 INFO UPLOAD_PROGRESS Uploaded chunk 59280 size 1320529, 44.81MB/s 1 day 08:07:24 5.0%
2017-04-21 20:34:56.606 INFO UPLOAD_PROGRESS Uploaded chunk 59281 size 6419629, 44.81MB/s 1 day 08:07:21 5.0%
2017-04-21 20:34:57.038 INFO UPLOAD_PROGRESS Uploaded chunk 59282 size 6143516, 44.81MB/s 1 day 08:07:38 5.0%
2017-04-21 20:34:57.246 INFO UPLOAD_PROGRESS Uploaded chunk 59283 size 1369271, 44.81MB/s 1 day 08:07:37 5.0%
2017-04-21 20:34:58.029 INFO UPLOAD_PROGRESS Uploaded chunk 59284 size 6358333, 44.80MB/s 1 day 08:07:53 5.0%
2017-04-21 20:34:58.337 INFO UPLOAD_PROGRESS Uploaded chunk 59285 size 5033308, 44.80MB/s 1 day 08:07:51 5.0%
2017-04-21 20:34:58.826 INFO UPLOAD_PROGRESS Uploaded chunk 59286 size 15782120, 44.80MB/s 1 day 08:07:45 5.0%
2017-04-21 20:34:59.159 INFO UPLOAD_PROGRESS Uploaded chunk 59287 size 5772771, 44.80MB/s 1 day 08:08:01 5.0%
2017-04-21 20:34:59.798 INFO UPLOAD_PROGRESS Uploaded chunk 59288 size 3563753, 44.80MB/s 1 day 08:07:59 5.0%
2017-04-21 20:35:00.259 INFO UPLOAD_PROGRESS Uploaded chunk 59289 size 2363738, 44.79MB/s 1 day 08:08:17 5.0%
2017-04-21 20:35:00.677 INFO UPLOAD_PROGRESS Uploaded chunk 59290 size 7231768, 44.79MB/s 1 day 08:08:14 5.0%
2017-04-21 20:35:01.060 INFO UPLOAD_PROGRESS Uploaded chunk 59291 size 6932970, 44.79MB/s 1 day 08:08:30 5.0%
2017-04-21 20:35:01.310 INFO UPLOAD_PROGRESS Uploaded chunk 59292 size 1049935, 44.79MB/s 1 day 08:08:30 5.0%
2017-04-21 20:35:01.614 INFO UPLOAD_PROGRESS Uploaded chunk 59293 size 1836511, 44.79MB/s 1 day 08:08:29 5.0%
2017-04-21 20:35:01.901 INFO UPLOAD_PROGRESS Uploaded chunk 59294 size 4396311, 44.79MB/s 1 day 08:08:27 5.0%
2017-04-21 20:35:02.723 INFO UPLOAD_PROGRESS Uploaded chunk 59295 size 12971577, 44.78MB/s 1 day 08:08:41 5.0%
2017-04-21 20:35:03.010 INFO UPLOAD_PROGRESS Uploaded chunk 59296 size 3066663, 44.78MB/s 1 day 08:08:58 5.0%
2017-04-21 20:35:03.318 INFO UPLOAD_PROGRESS Uploaded chunk 59297 size 9049475, 44.78MB/s 1 day 08:08:54 5.0%
2017-04-21 20:35:03.616 INFO UPLOAD_PROGRESS Uploaded chunk 59298 size 2916772, 44.78MB/s 1 day 08:08:53 5.0%
2017-04-21 20:35:03.974 INFO UPLOAD_PROGRESS Uploaded chunk 59299 size 3427307, 44.78MB/s 1 day 08:08:52 5.0%
2017-04-21 20:35:04.292 INFO UPLOAD_PROGRESS Uploaded chunk 59300 size 5225129, 44.77MB/s 1 day 08:09:08 5.0%
2017-04-21 20:35:04.604 INFO UPLOAD_PROGRESS Uploaded chunk 59301 size 3302682, 44.77MB/s 1 day 08:09:07 5.0%
2017-04-21 20:35:04.909 INFO UPLOAD_PROGRESS Uploaded chunk 59302 size 3675885, 44.77MB/s 1 day 08:09:05 5.0%
2017-04-21 20:35:05.318 INFO UPLOAD_PROGRESS Uploaded chunk 59303 size 11566724, 44.77MB/s 1 day 08:09:19 5.0%
2017-04-21 20:35:05.590 INFO UPLOAD_PROGRESS Uploaded chunk 59304 size 1912119, 44.77MB/s 1 day 08:09:19 5.0%
2017-04-21 20:35:05.963 INFO UPLOAD_PROGRESS Uploaded chunk 59305 size 5041189, 44.77MB/s 1 day 08:09:16 5.0%
2017-04-21 20:35:06.375 INFO UPLOAD_PROGRESS Uploaded chunk 59306 size 5331078, 44.76MB/s 1 day 08:09:33 5.0%
2017-04-21 20:35:07.089 INFO UPLOAD_PROGRESS Uploaded chunk 59307 size 9312536, 44.76MB/s 1 day 08:09:48 5.0%
2017-04-21 20:35:07.380 INFO UPLOAD_PROGRESS Uploaded chunk 59308 size 3206006, 44.76MB/s 1 day 08:09:47 5.0%
2017-04-21 20:35:07.792 INFO UPLOAD_PROGRESS Uploaded chunk 59309 size 4739855, 44.76MB/s 1 day 08:09:45 5.0%
2017-04-21 20:35:08.201 INFO UPLOAD_PROGRESS Uploaded chunk 59310 size 5043663, 44.75MB/s 1 day 08:10:01 5.0%
2017-04-21 20:35:08.496 INFO UPLOAD_PROGRESS Uploaded chunk 59311 size 2802235, 44.75MB/s 1 day 08:10:00 5.0%
2017-04-21 20:35:08.840 INFO UPLOAD_PROGRESS Uploaded chunk 59312 size 1583232, 44.75MB/s 1 day 08:09:59 5.0%
2017-04-21 20:35:09.120 INFO UPLOAD_PROGRESS Uploaded chunk 59313 size 2011926, 44.74MB/s 1 day 08:10:17 5.0%
2017-04-21 20:35:09.390 INFO UPLOAD_PROGRESS Uploaded chunk 59314 size 2354926, 44.74MB/s 1 day 08:10:16 5.0%
2017-04-21 20:35:09.716 INFO UPLOAD_PROGRESS Uploaded chunk 59315 size 3672839, 44.74MB/s 1 day 08:10:15 5.0%
2017-04-21 20:35:09.996 INFO UPLOAD_PROGRESS Uploaded chunk 59316 size 1641258, 44.75MB/s 1 day 08:10:14 5.0%
2017-04-21 20:35:10.324 INFO UPLOAD_PROGRESS Uploaded chunk 59317 size 3038780, 44.74MB/s 1 day 08:10:32 5.0%
2017-04-21 20:35:10.689 INFO UPLOAD_PROGRESS Uploaded chunk 59318 size 6903283, 44.74MB/s 1 day 08:10:29 5.0%
2017-04-21 20:35:10.973 INFO UPLOAD_PROGRESS Uploaded chunk 59319 size 6273325, 44.74MB/s 1 day 08:10:26 5.0%
2017-04-21 20:35:11.233 INFO UPLOAD_PROGRESS Uploaded chunk 59320 size 1207956, 44.73MB/s 1 day 08:10:45 5.0%
2017-04-21 20:35:11.775 INFO UPLOAD_PROGRESS Uploaded chunk 59321 size 16777216, 44.74MB/s 1 day 08:10:38 5.0%
2017-04-21 20:35:12.082 INFO UPLOAD_PROGRESS Uploaded chunk 59322 size 3208151, 44.73MB/s 1 day 08:10:55 5.0%
2017-04-21 20:35:12.382 INFO UPLOAD_PROGRESS Uploaded chunk 59323 size 3986250, 44.73MB/s 1 day 08:10:53 5.0%
2017-04-21 20:35:13.278 INFO UPLOAD_PROGRESS Uploaded chunk 59324 size 8843295, 44.72MB/s 1 day 08:11:08 5.0%
2017-04-21 20:35:13.628 INFO UPLOAD_PROGRESS Uploaded chunk 59325 size 7289096, 44.73MB/s 1 day 08:11:05 5.0%
2017-04-21 20:35:14.041 INFO UPLOAD_PROGRESS Uploaded chunk 59326 size 8519794, 44.72MB/s 1 day 08:11:21 5.0%
2017-04-21 20:35:14.315 INFO UPLOAD_PROGRESS Uploaded chunk 59327 size 2537740, 44.72MB/s 1 day 08:11:20 5.0%
2017-04-21 20:35:14.546 INFO UPLOAD_PROGRESS Uploaded chunk 59328 size 1235747, 44.72MB/s 1 day 08:11:19 5.0%
2017-04-21 20:35:15.150 INFO UPLOAD_PROGRESS Uploaded chunk 59329 size 2736366, 44.71MB/s 1 day 08:11:37 5.0%
2017-04-21 20:35:15.372 INFO UPLOAD_PROGRESS Uploaded chunk 59330 size 1073257, 44.71MB/s 1 day 08:11:36 5.0%
2017-04-21 20:35:15.738 INFO UPLOAD_PROGRESS Uploaded chunk 59331 size 1568240, 44.71MB/s 1 day 08:11:36 5.0%
2017-04-21 20:35:16.686 INFO UPLOAD_PROGRESS Uploaded chunk 59332 size 5401919, 44.71MB/s 1 day 08:11:52 5.0%
2017-04-21 20:35:17.281 INFO UPLOAD_PROGRESS Uploaded chunk 59333 size 6793071, 44.70MB/s 1 day 08:12:08 5.0%
2017-04-21 20:35:17.569 INFO UPLOAD_PROGRESS Uploaded chunk 59334 size 2260606, 44.70MB/s 1 day 08:12:07 5.0%
2017-04-21 20:35:18.073 INFO UPLOAD_PROGRESS Uploaded chunk 59335 size 3757268, 44.69MB/s 1 day 08:12:25 5.0%
2017-04-21 20:35:18.356 INFO UPLOAD_PROGRESS Uploaded chunk 59336 size 5653447, 44.70MB/s 1 day 08:12:22 5.0%
2017-04-21 20:35:19.136 INFO UPLOAD_PROGRESS Uploaded chunk 59337 size 1259944, 44.69MB/s 1 day 08:12:40 5.0%
2017-04-21 20:35:19.549 INFO UPLOAD_PROGRESS Uploaded chunk 59338 size 4018508, 44.69MB/s 1 day 08:12:39 5.0%
2017-04-21 20:35:20.825 INFO UPLOAD_PROGRESS Uploaded chunk 59339 size 16749196, 44.68MB/s 1 day 08:12:51 5.0%
2017-04-21 20:35:21.640 INFO UPLOAD_PROGRESS Uploaded chunk 59340 size 1086159, 44.68MB/s 1 day 08:13:09 5.0%
2017-04-21 20:35:21.961 INFO UPLOAD_PROGRESS Uploaded chunk 59341 size 3060887, 44.68MB/s 1 day 08:13:08 5.0%
2017-04-21 20:35:22.222 INFO UPLOAD_PROGRESS Uploaded chunk 59342 size 1368246, 44.67MB/s 1 day 08:13:26 5.0%
2017-04-21 20:35:22.463 INFO UPLOAD_PROGRESS Uploaded chunk 59343 size 1477415, 44.67MB/s 1 day 08:13:25 5.0%
2017-04-21 20:35:22.827 INFO UPLOAD_PROGRESS Uploaded chunk 59344 size 7469508, 44.67MB/s 1 day 08:13:22 5.0%
2017-04-21 20:35:23.234 INFO UPLOAD_PROGRESS Uploaded chunk 59345 size 7834814, 44.67MB/s 1 day 08:13:38 5.0%
2017-04-21 20:35:23.527 INFO UPLOAD_PROGRESS Uploaded chunk 59346 size 2685010, 44.67MB/s 1 day 08:13:37 5.0%
2017-04-21 20:35:23.813 INFO UPLOAD_PROGRESS Uploaded chunk 59347 size 6620309, 44.67MB/s 1 day 08:13:34 5.0%
2017-04-21 20:35:24.133 INFO UPLOAD_PROGRESS Uploaded chunk 59348 size 1753335, 44.66MB/s 1 day 08:13:52 5.0%
2017-04-21 20:35:24.450 INFO UPLOAD_PROGRESS Uploaded chunk 59349 size 2623671, 44.66MB/s 1 day 08:13:51 5.0%
2017-04-21 20:35:24.807 INFO UPLOAD_PROGRESS Uploaded chunk 59350 size 8892905, 44.66MB/s 1 day 08:13:47 5.0%
2017-04-21 20:35:25.073 INFO UPLOAD_PROGRESS Uploaded chunk 59351 size 4300881, 44.66MB/s 1 day 08:14:04 5.0%
2017-04-21 20:35:25.479 INFO UPLOAD_PROGRESS Uploaded chunk 59352 size 5901166, 44.66MB/s 1 day 08:14:02 5.0%
2017-04-21 20:35:25.798 INFO UPLOAD_PROGRESS Uploaded chunk 59353 size 1736387, 44.66MB/s 1 day 08:14:01 5.0%
2017-04-21 20:35:26.220 INFO UPLOAD_PROGRESS Uploaded chunk 59354 size 7721799, 44.65MB/s 1 day 08:14:16 5.0%
2017-04-21 20:35:26.509 INFO UPLOAD_PROGRESS Uploaded chunk 59355 size 1296861, 44.65MB/s 1 day 08:14:16 5.0%
2017-04-21 20:35:26.811 INFO UPLOAD_PROGRESS Uploaded chunk 59356 size 4415019, 44.65MB/s 1 day 08:14:14 5.0%
2017-04-21 20:35:27.161 INFO UPLOAD_PROGRESS Uploaded chunk 59357 size 10119040, 44.65MB/s 1 day 08:14:29 5.0%
2017-04-21 20:35:27.626 INFO UPLOAD_PROGRESS Uploaded chunk 59358 size 1760170, 44.65MB/s 1 day 08:14:28 5.0%
2017-04-21 20:35:27.957 INFO UPLOAD_PROGRESS Uploaded chunk 59359 size 4949714, 44.65MB/s 1 day 08:14:26 5.0%
2017-04-21 20:35:28.532 INFO UPLOAD_PROGRESS Uploaded chunk 59360 size 4738202, 44.64MB/s 1 day 08:14:43 5.0%
2017-04-21 20:35:28.846 INFO UPLOAD_PROGRESS Uploaded chunk 59361 size 3597468, 44.64MB/s 1 day 08:14:41 5.0%
2017-04-21 20:35:29.105 INFO UPLOAD_PROGRESS Uploaded chunk 59362 size 3117711, 44.63MB/s 1 day 08:14:59 5.0%
2017-04-21 20:35:29.480 INFO UPLOAD_PROGRESS Uploaded chunk 59363 size 8546368, 44.64MB/s 1 day 08:14:55 5.0%
2017-04-21 20:35:29.739 INFO UPLOAD_PROGRESS Uploaded chunk 59364 size 2436279, 44.64MB/s 1 day 08:14:54 5.0%
2017-04-21 20:35:30.024 INFO UPLOAD_PROGRESS Uploaded chunk 59365 size 1334111, 44.63MB/s 1 day 08:15:12 5.0%
2017-04-21 20:35:30.333 INFO UPLOAD_PROGRESS Uploaded chunk 59366 size 3703255, 44.63MB/s 1 day 08:15:11 5.0%
2017-04-21 20:35:30.659 INFO UPLOAD_PROGRESS Uploaded chunk 59367 size 5931823, 44.63MB/s 1 day 08:15:08 5.0%
2017-04-21 20:35:30.918 INFO UPLOAD_PROGRESS Uploaded chunk 59368 size 1281719, 44.63MB/s 1 day 08:15:08 5.0%
2017-04-21 20:35:31.578 INFO UPLOAD_PROGRESS Uploaded chunk 59369 size 1732396, 44.62MB/s 1 day 08:15:26 5.0%
2017-04-21 20:35:31.956 INFO UPLOAD_PROGRESS Uploaded chunk 59370 size 3245720, 44.62MB/s 1 day 08:15:24 5.0%
2017-04-21 20:35:32.286 INFO UPLOAD_PROGRESS Uploaded chunk 59371 size 2508298, 44.62MB/s 1 day 08:15:42 5.0%
2017-04-21 20:35:32.603 INFO UPLOAD_PROGRESS Uploaded chunk 59372 size 2072819, 44.62MB/s 1 day 08:15:41 5.0%
2017-04-21 20:35:32.934 INFO UPLOAD_PROGRESS Uploaded chunk 59373 size 3100242, 44.62MB/s 1 day 08:15:40 5.0%
2017-04-21 20:35:33.388 INFO UPLOAD_PROGRESS Uploaded chunk 59374 size 5583249, 44.61MB/s 1 day 08:15:56 5.0%
2017-04-21 20:35:33.879 INFO UPLOAD_PROGRESS Uploaded chunk 59375 size 9445292, 44.61MB/s 1 day 08:15:52 5.0%
2017-04-21 20:35:34.591 INFO UPLOAD_PROGRESS Uploaded chunk 59376 size 1188284, 44.61MB/s 1 day 08:16:11 5.0%
2017-04-21 20:35:34.966 INFO UPLOAD_PROGRESS Uploaded chunk 59377 size 4411869, 44.61MB/s 1 day 08:16:09 5.0%
2017-04-21 20:35:35.304 INFO UPLOAD_PROGRESS Uploaded chunk 59378 size 4729668, 44.60MB/s 1 day 08:16:26 5.0%
2017-04-21 20:35:35.774 INFO UPLOAD_PROGRESS Uploaded chunk 59379 size 7320787, 44.60MB/s 1 day 08:16:23 5.0%
2017-04-21 20:35:36.413 INFO UPLOAD_PROGRESS Uploaded chunk 59380 size 11910505, 44.60MB/s 1 day 08:16:36 5.0%
2017-04-21 20:35:36.795 INFO UPLOAD_PROGRESS Uploaded chunk 59381 size 6839865, 44.60MB/s 1 day 08:16:34 5.0%
2017-04-21 20:35:37.154 INFO UPLOAD_PROGRESS Uploaded chunk 59382 size 8110329, 44.59MB/s 1 day 08:16:49 5.0%
2017-04-21 20:35:37.922 INFO UPLOAD_PROGRESS Uploaded chunk 59383 size 6891667, 44.59MB/s 1 day 08:16:46 5.0%
2017-04-21 20:35:38.398 INFO UPLOAD_PROGRESS Uploaded chunk 59384 size 15926337, 44.59MB/s 1 day 08:16:58 5.0%
2017-04-21 20:35:38.676 INFO UPLOAD_PROGRESS Uploaded chunk 59385 size 1428290, 44.59MB/s 1 day 08:16:58 5.0%
2017-04-21 20:35:38.928 INFO UPLOAD_PROGRESS Uploaded chunk 59386 size 1999383, 44.59MB/s 1 day 08:16:57 5.0%
2017-04-21 20:35:39.259 INFO UPLOAD_PROGRESS Uploaded chunk 59387 size 2224763, 44.58MB/s 1 day 08:17:15 5.0%
2017-04-21 20:35:39.649 INFO UPLOAD_PROGRESS Uploaded chunk 59388 size 4061201, 44.58MB/s 1 day 08:17:13 5.0%
2017-04-21 20:35:40.067 INFO UPLOAD_PROGRESS Uploaded chunk 59389 size 8453644, 44.58MB/s 1 day 08:17:28 5.0%
2017-04-21 20:35:40.772 INFO UPLOAD_PROGRESS Uploaded chunk 59390 size 4829961, 44.58MB/s 1 day 08:17:26 5.0%
2017-04-21 20:35:41.110 INFO UPLOAD_PROGRESS Uploaded chunk 59391 size 1559360, 44.57MB/s 1 day 08:17:44 5.0%
2017-04-21 20:35:41.416 INFO UPLOAD_PROGRESS Uploaded chunk 59392 size 4149901, 44.57MB/s 1 day 08:17:42 5.0%
2017-04-21 20:35:41.664 INFO UPLOAD_PROGRESS Uploaded chunk 59393 size 1251265, 44.57MB/s 1 day 08:17:42 5.0%
2017-04-21 20:35:41.953 INFO UPLOAD_PROGRESS Uploaded chunk 59394 size 4668736, 44.57MB/s 1 day 08:17:40 5.0%
2017-04-21 20:35:42.376 INFO UPLOAD_PROGRESS Uploaded chunk 59395 size 5861196, 44.56MB/s 1 day 08:17:56 5.0%
2017-04-21 20:35:42.625 INFO UPLOAD_PROGRESS Uploaded chunk 59396 size 1576950, 44.56MB/s 1 day 08:17:56 5.0%
2017-04-21 20:35:42.989 INFO UPLOAD_PROGRESS Uploaded chunk 59397 size 3488592, 44.57MB/s 1 day 08:17:54 5.0%
2017-04-21 20:35:43.514 INFO UPLOAD_PROGRESS Uploaded chunk 59398 size 2059012, 44.56MB/s 1 day 08:18:12 5.0%
2017-04-21 20:35:44.289 INFO UPLOAD_PROGRESS Uploaded chunk 59399 size 1404570, 44.55MB/s 1 day 08:18:30 5.0%
2017-04-21 20:35:47.029 INFO UPLOAD_PROGRESS Uploaded chunk 59400 size 4333447, 44.53MB/s 1 day 08:19:25 5.0%
2017-04-21 20:35:47.316 INFO UPLOAD_PROGRESS Uploaded chunk 59401 size 3206442, 44.53MB/s 1 day 08:19:24 5.0%
2017-04-21 20:35:49.975 INFO UPLOAD_PROGRESS Uploaded chunk 59402 size 8557405, 44.52MB/s 1 day 08:19:58 5.0%
2017-04-21 20:35:50.340 INFO UPLOAD_PROGRESS Uploaded chunk 59403 size 11486958, 44.51MB/s 1 day 08:20:12 5.0%
2017-04-21 20:36:11.347 ERROR UPLOAD_CHUNK Failed to upload the chunk
Charles Apr 24 1:21PM 2017
That does not seem to be an option for the backup part of the CLI version. Also, it appears I would lose the ability to send e-mail notifications upon failure if I used the CLI version (and scheduling via a windows service, but I could run it via Task Scheduler).
For now I will limit my upload speed, but perhaps a api rate limit can be imposed for GCP or even better, an "exclusive" option for backups that will not perform the "file exists" api call when uploading which could also save some money on api calls (maybe not enough to matter).
edit: I will let you know if the rate limit helps.
gchen Apr 24 7:17PM 2017
This is definitely caused by rate limiting on the server side.
Duplicacy uses goamz to access Google Cloud Storage. I think its retry mechanism can be improved to handle this rate limiting issue.
Charles Apr 24 7:59PM 2017
Thanks for the confirmation.
I set a rate limit of 10000 KBps (should be 80 Mbps or roughly 10 MBps). I am still copying at speeds between 50 and 60 MBps. This isn't the first time I've had troubles getting settings from the GUI to apply to the backup process. I also had troubles with frequency.
I checked the schedule file in the .duplicacy folder and it seems to have applied okay.
{
"backup_scheduled" : true,
"rate_limit" : "10000",
"retires" : 3,
"enable_vss" : true,
"email_options" : {
"subject" : "",
"from" : "",
"to" : "",
"username" : "",
"server" : "",
"failure_only" : false
},
"backup_frequency" : 86400,
"prune_enabled" : true,
"backup_threads" : 1,
"prune_parameters" : [
0,
120,
7,
92,
1,
91
],
"email_enabled" : false,
"backup_time" : 60677,
"backup_days" : 127
}
gchen Apr 24 8:29PM 2017
Please try the latest build at https://github.com/gilbertchen/duplicacy-cli/releases
The option for rate limiting is -limit-rate. However, this only limits the rate of transferring bytes over the network. Since many chunks already exist on the storage, they are actually not uploaded but they are counted in the upload speed being displayed. Therefore, those upload speeds are alway significantly higher than the rate limit.
Charles Apr 25 11:08AM 2017
Still failing at 10000 KBps. Same error message.
I don't know if this log info helps, but here it is.
2017-04-25 00:26:00.858 INFO UPLOAD_PROGRESS Uploaded chunk 114061 size 9710648, 40.24MB/s 1 day 10:15:32 9.0% 2017-04-25 00:26:02.016 INFO UPLOAD_PROGRESS Uploaded chunk 114062 size 3121269, 40.23MB/s 1 day 10:15:51 9.0% 2017-04-25 00:26:03.054 INFO UPLOAD_PROGRESS Uploaded chunk 114063 size 5961670, 40.23MB/s 1 day 10:16:00 9.0% 2017-04-25 00:26:04.843 INFO UPLOAD_PROGRESS Uploaded chunk 114064 size 16777216, 40.22MB/s 1 day 10:16:06 9.0% 2017-04-25 00:26:05.832 INFO UPLOAD_PROGRESS Uploaded chunk 114065 size 7156805, 40.22MB/s 1 day 10:16:14 9.0% 2017-04-25 00:26:06.224 INFO UPLOAD_PROGRESS Uploaded chunk 114066 size 2208678, 40.22MB/s 1 day 10:16:23 9.0% 2017-04-25 00:26:07.887 INFO UPLOAD_PROGRESS Uploaded chunk 114067 size 15252565, 40.22MB/s 1 day 10:16:29 9.0% 2017-04-25 00:26:08.589 INFO UPLOAD_PROGRESS Uploaded chunk 114068 size 5652306, 40.21MB/s 1 day 10:16:38 9.0% 2017-04-25 00:26:09.062 INFO UPLOAD_PROGRESS Uploaded chunk 114069 size 3153143, 40.21MB/s 1 day 10:16:47 9.0% 2017-04-25 00:26:09.347 INFO UPLOAD_PROGRESS Uploaded chunk 114070 size 1307461, 40.21MB/s 1 day 10:16:47 9.0% 2017-04-25 00:26:09.841 INFO UPLOAD_PROGRESS Uploaded chunk 114071 size 3102476, 40.21MB/s 1 day 10:16:46 9.0% 2017-04-25 00:26:10.377 INFO UPLOAD_PROGRESS Uploaded chunk 114072 size 3376395, 40.21MB/s 1 day 10:16:55 9.0% 2017-04-25 00:26:10.812 INFO UPLOAD_PROGRESS Uploaded chunk 114073 size 2695242, 40.21MB/s 1 day 10:16:54 9.0% 2017-04-25 00:26:11.241 INFO UPLOAD_PROGRESS Uploaded chunk 114074 size 2599460, 40.21MB/s 1 day 10:17:04 9.0% 2017-04-25 00:26:12.472 INFO UPLOAD_PROGRESS Uploaded chunk 114075 size 11243435, 40.20MB/s 1 day 10:17:11 9.0% 2017-04-25 00:26:12.954 INFO UPLOAD_PROGRESS Uploaded chunk 114076 size 3004988, 40.20MB/s 1 day 10:17:10 9.0% 2017-04-25 00:26:13.540 INFO UPLOAD_PROGRESS Uploaded chunk 114077 size 4538367, 40.20MB/s 1 day 10:17:19 9.0% 2017-04-25 00:26:14.511 INFO UPLOAD_PROGRESS Uploaded chunk 114078 size 6338832, 40.20MB/s 1 day 10:17:27 9.0% 2017-04-25 00:26:15.315 INFO UPLOAD_PROGRESS Uploaded chunk 114079 size 6562701, 40.19MB/s 1 day 10:17:35 9.0% 2017-04-25 00:26:16.067 INFO UPLOAD_PROGRESS Uploaded chunk 114080 size 3029580, 40.19MB/s 1 day 10:17:45 9.0% 2017-04-25 00:26:17.009 INFO UPLOAD_PROGRESS Uploaded chunk 114081 size 1926082, 40.19MB/s 1 day 10:17:54 9.0% 2017-04-25 00:26:17.504 INFO UPLOAD_PROGRESS Uploaded chunk 114082 size 2824735, 40.19MB/s 1 day 10:17:53 9.0% 2017-04-25 00:26:18.521 INFO UPLOAD_PROGRESS Uploaded chunk 114083 size 8780378, 40.19MB/s 1 day 10:18:01 9.0% 2017-04-25 00:26:18.913 INFO UPLOAD_PROGRESS Uploaded chunk 114084 size 1983981, 40.19MB/s 1 day 10:18:01 9.0% 2017-04-25 00:26:20.186 INFO UPLOAD_PROGRESS Uploaded chunk 114085 size 11246045, 40.18MB/s 1 day 10:18:18 9.0% 2017-04-25 00:26:20.737 INFO UPLOAD_PROGRESS Uploaded chunk 114086 size 1305542, 40.18MB/s 1 day 10:18:17 9.0% 2017-04-25 00:26:22.050 INFO UPLOAD_PROGRESS Uploaded chunk 114087 size 11646296, 40.18MB/s 1 day 10:18:35 9.0% 2017-04-25 00:26:22.919 INFO UPLOAD_PROGRESS Uploaded chunk 114088 size 7269006, 40.18MB/s 1 day 10:18:33 9.0% 2017-04-25 00:26:23.521 INFO UPLOAD_PROGRESS Uploaded chunk 114089 size 4780615, 40.17MB/s 1 day 10:18:41 9.0% 2017-04-25 00:26:23.821 INFO UPLOAD_PROGRESS Uploaded chunk 114090 size 1163609, 40.17MB/s 1 day 10:18:41 9.0% 2017-04-25 00:26:24.642 INFO UPLOAD_PROGRESS Uploaded chunk 114091 size 6456576, 40.17MB/s 1 day 10:18:49 9.0% 2017-04-25 00:26:26.208 INFO UPLOAD_PROGRESS Uploaded chunk 114092 size 14300276, 40.16MB/s 1 day 10:19:06 9.0% 2017-04-25 00:26:27.304 INFO UPLOAD_PROGRESS Uploaded chunk 114093 size 9713777, 40.16MB/s 1 day 10:19:13 9.0% 2017-04-25 00:26:27.575 INFO UPLOAD_PROGRESS Uploaded chunk 114094 size 1096103, 40.16MB/s 1 day 10:19:13 9.0% 2017-04-25 00:26:28.109 INFO UPLOAD_PROGRESS Uploaded chunk 114095 size 3860574, 40.16MB/s 1 day 10:19:22 9.0% 2017-04-25 00:26:28.885 INFO UPLOAD_PROGRESS Uploaded chunk 114096 size 2888449, 40.16MB/s 1 day 10:19:21 9.0% 2017-04-25 00:26:30.172 INFO UPLOAD_PROGRESS Uploaded chunk 114097 size 11611887, 40.15MB/s 1 day 10:19:38 9.0% 2017-04-25 00:26:31.570 INFO UPLOAD_PROGRESS Uploaded chunk 114098 size 13059774, 40.15MB/s 1 day 10:19:45 9.0% 2017-04-25 00:26:32.015 INFO UPLOAD_PROGRESS Uploaded chunk 114099 size 3023350, 40.15MB/s 1 day 10:19:54 9.0% 2017-04-25 00:26:32.737 INFO UPLOAD_PROGRESS Uploaded chunk 114100 size 5565408, 40.15MB/s 1 day 10:19:53 9.0% 2017-04-25 00:26:33.769 INFO UPLOAD_PROGRESS Uploaded chunk 114101 size 9220030, 40.15MB/s 1 day 10:20:00 9.0% 2017-04-25 00:26:34.786 INFO UPLOAD_PROGRESS Uploaded chunk 114102 size 3301334, 40.14MB/s 1 day 10:20:09 9.0% 2017-04-25 00:26:35.218 INFO UPLOAD_PROGRESS Uploaded chunk 114103 size 2431739, 40.14MB/s 1 day 10:20:19 9.0% 2017-04-25 00:26:35.840 INFO UPLOAD_PROGRESS Uploaded chunk 114104 size 4356261, 40.14MB/s 1 day 10:20:18 9.0% 2017-04-25 00:26:36.145 INFO UPLOAD_PROGRESS Uploaded chunk 114105 size 1054372, 40.14MB/s 1 day 10:20:27 9.0% 2017-04-25 00:26:37.415 INFO UPLOAD_PROGRESS Uploaded chunk 114106 size 11418260, 40.14MB/s 1 day 10:20:34 9.0% 2017-04-25 00:26:38.287 INFO UPLOAD_PROGRESS Uploaded chunk 114107 size 7084017, 40.13MB/s 1 day 10:20:43 9.0% 2017-04-25 00:26:39.102 INFO UPLOAD_PROGRESS Uploaded chunk 114108 size 6445595, 40.13MB/s 1 day 10:20:51 9.0% 2017-04-25 00:26:39.751 INFO UPLOAD_PROGRESS Uploaded chunk 114109 size 3899959, 40.13MB/s 1 day 10:20:50 9.0% 2017-04-25 00:26:40.103 INFO UPLOAD_PROGRESS Uploaded chunk 114110 size 1691228, 40.13MB/s 1 day 10:21:00 9.0% 2017-04-25 00:26:40.702 INFO UPLOAD_PROGRESS Uploaded chunk 114111 size 4889830, 40.13MB/s 1 day 10:20:58 9.0% 2017-04-25 00:26:41.137 INFO UPLOAD_PROGRESS Uploaded chunk 114112 size 2494939, 40.12MB/s 1 day 10:21:08 9.0% 2017-04-25 00:26:41.522 INFO UPLOAD_PROGRESS Uploaded chunk 114113 size 1930320, 40.12MB/s 1 day 10:21:07 9.0% 2017-04-25 00:26:42.158 INFO UPLOAD_PROGRESS Uploaded chunk 114114 size 5146836, 40.12MB/s 1 day 10:21:16 9.0% 2017-04-25 00:26:42.976 INFO UPLOAD_PROGRESS Uploaded chunk 114115 size 6789722, 40.12MB/s 1 day 10:21:14 9.0% 2017-04-25 00:26:44.156 INFO UPLOAD_PROGRESS Uploaded chunk 114116 size 10255870, 40.12MB/s 1 day 10:21:31 9.0% 2017-04-25 00:26:44.460 INFO UPLOAD_PROGRESS Uploaded chunk 114117 size 1125725, 40.12MB/s 1 day 10:21:31 9.0% 2017-04-25 00:26:44.867 INFO UPLOAD_PROGRESS Uploaded chunk 114118 size 1793175, 40.12MB/s 1 day 10:21:31 9.0% 2017-04-25 00:26:45.183 INFO UPLOAD_PROGRESS Uploaded chunk 114119 size 1626726, 40.11MB/s 1 day 10:21:40 9.0% 2017-04-25 00:26:47.060 INFO UPLOAD_PROGRESS Uploaded chunk 114120 size 16777216, 40.11MB/s 1 day 10:21:56 9.0% 2017-04-25 00:26:47.411 INFO UPLOAD_PROGRESS Uploaded chunk 114121 size 1831570, 40.11MB/s 1 day 10:21:55 9.0% 2017-04-25 00:26:47.966 INFO UPLOAD_PROGRESS Uploaded chunk 114122 size 3822479, 40.11MB/s 1 day 10:21:54 9.0% 2017-04-25 00:26:48.522 INFO UPLOAD_PROGRESS Uploaded chunk 114123 size 3972755, 40.11MB/s 1 day 10:22:03 9.0% 2017-04-25 00:26:50.112 INFO UPLOAD_PROGRESS Uploaded chunk 114124 size 14756914, 40.10MB/s 1 day 10:22:20 9.0% 2017-04-25 00:26:50.678 INFO UPLOAD_PROGRESS Uploaded chunk 114125 size 1055295, 40.10MB/s 1 day 10:22:19 9.0% 2017-04-25 00:26:51.946 INFO UPLOAD_PROGRESS Uploaded chunk 114126 size 11608608, 40.10MB/s 1 day 10:22:26 9.0% 2017-04-25 00:26:52.457 INFO UPLOAD_PROGRESS Uploaded chunk 114127 size 3623483, 40.10MB/s 1 day 10:22:35 9.0% 2017-04-25 00:26:53.869 INFO UPLOAD_PROGRESS Uploaded chunk 114128 size 12629679, 40.09MB/s 1 day 10:22:42 9.0% 2017-04-25 00:26:54.285 INFO UPLOAD_PROGRESS Uploaded chunk 114129 size 2081237, 40.09MB/s 1 day 10:22:52 9.0% 2017-04-25 00:26:54.873 INFO UPLOAD_PROGRESS Uploaded chunk 114130 size 3056207, 40.09MB/s 1 day 10:22:51 9.0% 2017-04-25 00:26:55.333 INFO UPLOAD_PROGRESS Uploaded chunk 114131 size 3219691, 40.09MB/s 1 day 10:23:00 9.0% 2017-04-25 00:26:55.677 INFO UPLOAD_PROGRESS Uploaded chunk 114132 size 1481414, 40.09MB/s 1 day 10:23:00 9.0% 2017-04-25 00:26:56.202 INFO UPLOAD_PROGRESS Uploaded chunk 114133 size 3662514, 40.08MB/s 1 day 10:23:09 9.0% 2017-04-25 00:26:56.606 INFO UPLOAD_PROGRESS Uploaded chunk 114134 size 2271229, 40.08MB/s 1 day 10:23:08 9.0% 2017-04-25 00:26:58.216 INFO UPLOAD_PROGRESS Uploaded chunk 114135 size 15007075, 40.08MB/s 1 day 10:23:24 9.0% 2017-04-25 00:26:58.825 INFO UPLOAD_PROGRESS Uploaded chunk 114136 size 4775203, 40.08MB/s 1 day 10:23:23 9.0% 2017-04-25 00:26:59.467 INFO UPLOAD_PROGRESS Uploaded chunk 114137 size 3594666, 40.08MB/s 1 day 10:23:32 9.0% 2017-04-25 00:26:59.808 INFO UPLOAD_PROGRESS Uploaded chunk 114138 size 1758991, 40.08MB/s 1 day 10:23:32 9.0% 2017-04-25 00:27:00.343 INFO UPLOAD_PROGRESS Uploaded chunk 114139 size 3764498, 40.07MB/s 1 day 10:23:41 9.0% 2017-04-25 00:27:00.952 INFO UPLOAD_PROGRESS Uploaded chunk 114140 size 1848469, 40.07MB/s 1 day 10:23:40 9.0% 2017-04-25 00:27:02.123 INFO UPLOAD_PROGRESS Uploaded chunk 114141 size 10159956, 40.07MB/s 1 day 10:23:57 9.0% 2017-04-25 00:27:02.520 INFO UPLOAD_PROGRESS Uploaded chunk 114142 size 2312260, 40.07MB/s 1 day 10:23:57 9.0% 2017-04-25 00:27:03.449 INFO UPLOAD_PROGRESS Uploaded chunk 114143 size 7909862, 40.07MB/s 1 day 10:24:05 9.0% 2017-04-25 00:27:03.888 INFO UPLOAD_PROGRESS Uploaded chunk 114144 size 2067322, 40.07MB/s 1 day 10:24:04 9.0% 2017-04-25 00:27:04.889 INFO UPLOAD_PROGRESS Uploaded chunk 114145 size 8675153, 40.06MB/s 1 day 10:24:12 9.0% 2017-04-25 00:27:05.320 INFO UPLOAD_PROGRESS Uploaded chunk 114146 size 2313180, 40.06MB/s 1 day 10:24:21 9.0% 2017-04-25 00:27:05.946 INFO UPLOAD_PROGRESS Uploaded chunk 114147 size 2802832, 40.06MB/s 1 day 10:24:21 9.0% 2017-04-25 00:27:06.781 INFO UPLOAD_PROGRESS Uploaded chunk 114148 size 6352424, 40.06MB/s 1 day 10:24:29 9.0% 2017-04-25 00:27:07.403 INFO UPLOAD_PROGRESS Uploaded chunk 114149 size 5126147, 40.06MB/s 1 day 10:24:38 9.0% 2017-04-25 00:27:07.722 INFO UPLOAD_PROGRESS Uploaded chunk 114150 size 1716056, 40.06MB/s 1 day 10:24:37 9.0% 2017-04-25 00:27:08.166 INFO UPLOAD_PROGRESS Uploaded chunk 114151 size 3062477, 40.05MB/s 1 day 10:24:47 9.0% 2017-04-25 00:27:08.598 INFO UPLOAD_PROGRESS Uploaded chunk 114152 size 1911408, 40.05MB/s 1 day 10:24:46 9.0% 2017-04-25 00:27:09.414 INFO UPLOAD_PROGRESS Uploaded chunk 114153 size 1204325, 40.05MB/s 1 day 10:24:56 9.0% 2017-04-25 00:27:10.097 INFO UPLOAD_PROGRESS Uploaded chunk 114154 size 5384198, 40.05MB/s 1 day 10:25:04 9.0% 2017-04-25 00:27:10.716 INFO UPLOAD_PROGRESS Uploaded chunk 114155 size 4332229, 40.05MB/s 1 day 10:25:03 9.0% 2017-04-25 00:27:12.046 INFO UPLOAD_PROGRESS Uploaded chunk 114156 size 12038008, 40.04MB/s 1 day 10:25:20 9.0% 2017-04-25 00:27:12.716 INFO UPLOAD_PROGRESS Uploaded chunk 114157 size 2685237, 40.04MB/s 1 day 10:25:19 9.0% 2017-04-25 00:27:13.043 INFO UPLOAD_PROGRESS Uploaded chunk 114158 size 1614499, 40.04MB/s 1 day 10:25:29 9.0% 2017-04-25 00:27:13.334 INFO UPLOAD_PROGRESS Uploaded chunk 114159 size 1060100, 40.04MB/s 1 day 10:25:29 9.0% 2017-04-25 00:27:13.829 INFO UPLOAD_PROGRESS Uploaded chunk 114160 size 3631357, 40.04MB/s 1 day 10:25:28 9.0% 2017-04-25 00:27:14.443 INFO UPLOAD_PROGRESS Uploaded chunk 114161 size 4318202, 40.04MB/s 1 day 10:25:37 9.0% 2017-04-25 00:27:15.273 INFO UPLOAD_PROGRESS Uploaded chunk 114162 size 7139175, 40.03MB/s 1 day 10:25:45 9.0% 2017-04-25 00:27:15.869 INFO UPLOAD_PROGRESS Uploaded chunk 114163 size 4229144, 40.03MB/s 1 day 10:25:44 9.0% 2017-04-25 00:27:16.536 INFO UPLOAD_PROGRESS Uploaded chunk 114164 size 5224605, 40.03MB/s 1 day 10:25:52 9.0% 2017-04-25 00:27:17.774 INFO UPLOAD_PROGRESS Uploaded chunk 114165 size 11492457, 40.03MB/s 1 day 10:25:59 9.0% 2017-04-25 00:27:18.834 INFO UPLOAD_PROGRESS Uploaded chunk 114166 size 9342994, 40.03MB/s 1 day 10:26:07 9.0% 2017-04-25 00:27:19.600 INFO UPLOAD_PROGRESS Uploaded chunk 114167 size 5620592, 40.02MB/s 1 day 10:26:15 9.0% 2017-04-25 00:27:19.990 INFO UPLOAD_PROGRESS Uploaded chunk 114168 size 2189957, 40.02MB/s 1 day 10:26:15 9.0% 2017-04-25 00:27:20.684 INFO UPLOAD_PROGRESS Uploaded chunk 114169 size 5035737, 40.02MB/s 1 day 10:26:24 9.0% 2017-04-25 00:27:21.156 INFO UPLOAD_PROGRESS Uploaded chunk 114170 size 2858383, 40.02MB/s 1 day 10:26:33 9.0% 2017-04-25 00:27:21.485 INFO UPLOAD_PROGRESS Uploaded chunk 114171 size 1398414, 40.02MB/s 1 day 10:26:33 9.0% 2017-04-25 00:27:21.763 INFO UPLOAD_PROGRESS Uploaded chunk 114172 size 1105100, 40.02MB/s 1 day 10:26:32 9.0% 2017-04-25 00:27:22.130 INFO UPLOAD_PROGRESS Uploaded chunk 114173 size 1792441, 40.01MB/s 1 day 10:26:42 9.0% 2017-04-25 00:27:23.051 INFO UPLOAD_PROGRESS Uploaded chunk 114174 size 7923683, 40.01MB/s 1 day 10:26:50 9.0% 2017-04-25 00:27:23.997 INFO UPLOAD_PROGRESS Uploaded chunk 114175 size 7082990, 40.01MB/s 1 day 10:26:48 9.0% 2017-04-25 00:27:24.580 INFO UPLOAD_PROGRESS Uploaded chunk 114176 size 4503195, 40.01MB/s 1 day 10:26:57 9.0% 2017-04-25 00:27:25.167 INFO UPLOAD_PROGRESS Uploaded chunk 114177 size 4598736, 40.01MB/s 1 day 10:27:06 9.0% 2017-04-25 00:27:25.760 INFO UPLOAD_PROGRESS Uploaded chunk 114178 size 4446811, 40.01MB/s 1 day 10:27:04 9.0% 2017-04-25 00:27:26.202 INFO UPLOAD_PROGRESS Uploaded chunk 114179 size 2834262, 40.00MB/s 1 day 10:27:14 9.0% 2017-04-25 00:27:26.805 INFO UPLOAD_PROGRESS Uploaded chunk 114180 size 4119379, 40.00MB/s 1 day 10:27:13 9.0% 2017-04-25 00:27:27.146 INFO UPLOAD_PROGRESS Uploaded chunk 114181 size 1128394, 40.00MB/s 1 day 10:27:22 9.0% 2017-04-25 00:27:27.721 INFO UPLOAD_PROGRESS Uploaded chunk 114182 size 4108835, 40.00MB/s 1 day 10:27:21 9.0% 2017-04-25 00:27:28.161 INFO UPLOAD_PROGRESS Uploaded chunk 114183 size 2383326, 40.00MB/s 1 day 10:27:31 9.0% 2017-04-25 00:27:28.828 INFO UPLOAD_PROGRESS Uploaded chunk 114184 size 2241554, 40.00MB/s 1 day 10:27:30 9.0% 2017-04-25 00:27:29.310 INFO UPLOAD_PROGRESS Uploaded chunk 114185 size 3381778, 40.00MB/s 1 day 10:27:39 9.0% 2017-04-25 00:27:30.932 INFO UPLOAD_PROGRESS Uploaded chunk 114186 size 15326831, 39.99MB/s 1 day 10:27:45 9.0% 2017-04-25 00:27:31.576 INFO UPLOAD_PROGRESS Uploaded chunk 114187 size 5102852, 39.99MB/s 1 day 10:27:54 9.0% 2017-04-25 00:27:32.496 INFO UPLOAD_PROGRESS Uploaded chunk 114188 size 7535392, 39.99MB/s 1 day 10:28:02 9.0% 2017-04-25 00:27:32.923 INFO UPLOAD_PROGRESS Uploaded chunk 114189 size 2096902, 39.99MB/s 1 day 10:28:01 9.0% 2017-04-25 00:27:33.540 INFO UPLOAD_PROGRESS Uploaded chunk 114190 size 2043588, 39.98MB/s 1 day 10:28:11 9.0% 2017-04-25 00:27:34.110 INFO UPLOAD_PROGRESS Uploaded chunk 114191 size 4607881, 39.98MB/s 1 day 10:28:20 9.0% 2017-04-25 00:27:34.789 INFO UPLOAD_PROGRESS Uploaded chunk 114192 size 5486218, 39.98MB/s 1 day 10:28:18 9.0% 2017-04-25 00:27:35.402 INFO UPLOAD_PROGRESS Uploaded chunk 114193 size 4411294, 39.98MB/s 1 day 10:28:27 9.0% 2017-04-25 00:27:35.752 INFO UPLOAD_PROGRESS Uploaded chunk 114194 size 1585039, 39.98MB/s 1 day 10:28:27 9.0% 2017-04-25 00:27:56.749 ERROR UPLOAD_CHUNK Failed to upload the chunk 7708989e2e517dd31b0baed6733c6797193c0a1865431ec5b300cdf0ea977001: Put https://storage.googleapis.com/[redacted]/[redacted]/chunks/7708989e2e517dd31b0baed6733c6797193c0a1865431ec5b300cdf0ea977001: dial tcp 173.194.219.128:443: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.
gchen Apr 25 12:46PM 2017
Are you running the latest 1.2.5 version? The time between the failure and the last successful upload was about 21 seconds (27:35 -> 27:56), but in the latest 1.2.5 version the retry mechanism has been changed to allow for 8 attempts and a maximum wait of 5 minutes.
Charles Apr 25 3:07PM 2017
I was using the GUI version. I downloaded the CLI version you mentioned after the test with the GUI version where I throttled the upload speed. The CLI version doesn't seem to produce the text log, but it also failed. I did at least use the -stats flag.
I didn't see a way to turn on the log similar to the gui version in the documentation.
Terminal out put of \Scripts\Duplicacy\duplicacy_win_x64_1.2.5.exe backup -vss -stats -limit-rate 12500
Skipped chunk 3329 size 11383365, 42.08MB/s 1 day 10:19:54 0.3% Skipped chunk 3330 size 1210868, 42.08MB/s 1 day 10:19:45 0.3% Skipped chunk 3331 size 6784692, 42.10MB/s 1 day 10:18:56 0.3% Skipped chunk 3332 size 2596051, 42.10MB/s 1 day 10:18:37 0.3% Skipped chunk 3333 size 1692088, 42.11MB/s 1 day 10:18:25 0.3% Skipped chunk 3334 size 7616082, 42.13MB/s 1 day 10:17:30 0.3% Skipped chunk 3335 size 1433062, 42.13MB/s 1 day 10:17:20 0.3% Skipped chunk 3336 size 8380075, 42.15MB/s 1 day 10:16:20 0.3% Skipped chunk 3337 size 3969406, 42.05MB/s 1 day 10:21:08 0.3% Skipped chunk 3338 size 1237734, 42.05MB/s 1 day 10:20:59 0.3% Skipped chunk 3339 size 13561563, 42.09MB/s 1 day 10:19:22 0.3% Skipped chunk 3340 size 1069825, 42.09MB/s 1 day 10:19:14 0.3% Skipped chunk 3341 size 1177936, 42.09MB/s 1 day 10:19:05 0.3% Skipped chunk 3342 size 1902182, 42.10MB/s 1 day 10:18:52 0.3% Failed to find the path for the chunk e5f69f43124ce626ca70dc8b5c6579bd39718a9cfb c591bc80c9c0a076f411b8: Head https://storage.googleapis.com/[redacted]/[redacted]/chunks/e5f69f43124ce626ca70dc8b5c6579bd39718a9cfbc591bc80c9c0a076f411b8: dial tcp 74.125.21.128:443: connectex: A connection attempt failed because the conne cted party did not properly respond after a period of time, or established conne ction failed because connected host has failed to respond.
Charles Apr 25 3:52PM 2017
This may be a wild shot in the dark, but another idea occurs to me.
Okay, so it looks like the app is trying to dial tcp 74.125.21.128:443 directly. Is the ip address resolved and stored somewhere for repeat usage throughout the backup process? If so, maybe it needs to be re-resolved. I just performed some rudimentary test where I ran nslookup and pinged storage.googleapis.com a few times in a few different ways. I noticed that each time I pinged it, I sometimes resolved a different address (see output below).
Perhaps google tries to load balance by dropping connections on one IP, forcing the clients to choose the next ip available in the list. This is just speculation though.
https://serverfault.com/questions/102879/how-do-dns-clients-choose-an-ip-address-when-they-get-multiple-answers "A client program picking an IP address from a list (of A/AAAA-records) will generally try the addresses in the order they where returned by the DNS server (round robin). If the client cannot connect to the first IP address returned, it should try the second and so on. For example all major browsers do this, however many other Internet client programs "forget" this step and fail if they cannot connect to the first IP address."
>ping storage.googleapis.com
Pinging storage.l.googleusercontent.com [173.194.219.128] with 32 bytes of data:
Reply from 173.194.219.128: bytes=32 time=5ms TTL=45
Reply from 173.194.219.128: bytes=32 time=4ms TTL=45
Reply from 173.194.219.128: bytes=32 time=4ms TTL=45
Reply from 173.194.219.128: bytes=32 time=4ms TTL=45
Ping statistics for 173.194.219.128:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 4ms, Maximum = 5ms, Average = 4ms
-------------------------
>ping storage.googleapis.com
Pinging storage.l.googleusercontent.com [74.125.21.128] with 32 bytes of data:
Reply from 74.125.21.128: bytes=32 time=5ms TTL=45
Reply from 74.125.21.128: bytes=32 time=4ms TTL=45
Reply from 74.125.21.128: bytes=32 time=5ms TTL=45
Reply from 74.125.21.128: bytes=32 time=5ms TTL=45
Ping statistics for 74.125.21.128:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 4ms, Maximum = 5ms, Average = 4ms
gchen Apr 25 4:05PM 2017
Can you run the CLI version with the -log option to see if the retry mechanism is working:
\Scripts\Duplicacy\duplicacy_win_x64_1.2.5.exe -log backup -vss -stats
The -limit-rate option is not needed because it doesn't actually limit the API call rates.
It is possible that they do rate limiting by forcing you to switch to a different IP, but this should have been handled gracefully by the retry mechanism. I think the right solution is to use the official google client to access Google Cloud Storage, instead of relying on a third-party S3 client like goamz.
Charles Apr 25 4:53PM 2017
Ah. Running it again with -log, though it is obviously going to fail again, we can at least look for the change you are expecting.
I agree that using the Google Cloud official API's instead of S3 interoperability would be a great thing to have in there. I used Cloudberry with Google Cloud Storage which doesn't use S3 interoperability with no problems and it completed the 5TB initial backup in less than 24 hours. I went to sleep and woke up and it was done. (that said, I would much rather use this). If we can't get this working as is, I would be willing to stick around to see the completion of an official Google Storage API integration (depending on how long that would take).
Charles Apr 26 8:32AM 2017
It doesn't look like it is working at all now.
2017-04-25 19:59:07.346 INFO UPLOAD_PROGRESS Skipped chunk 10537 size 3900210, 3 9.17MB/s 1 day 12:37:34 0.9% 2017-04-25 19:59:07.402 INFO UPLOAD_PROGRESS Skipped chunk 10538 size 2112008, 3 9.17MB/s 1 day 12:37:29 0.9% 2017-04-25 19:59:07.491 INFO UPLOAD_PROGRESS Skipped chunk 10539 size 3530716, 3 9.18MB/s 1 day 12:37:21 0.9% 2017-04-25 19:59:07.657 INFO UPLOAD_PROGRESS Skipped chunk 10540 size 8385868, 3 9.18MB/s 1 day 12:37:00 0.9% 2017-04-25 19:59:07.728 INFO UPLOAD_PROGRESS Skipped chunk 10541 size 3612533, 3 9.18MB/s 1 day 12:36:51 0.9% 2017-04-25 19:59:07.950 INFO UPLOAD_PROGRESS Skipped chunk 10542 size 7704403, 3 9.19MB/s 1 day 12:36:32 0.9% 2017-04-25 19:59:08.023 INFO UPLOAD_PROGRESS Skipped chunk 10543 size 5135929, 3 9.16MB/s 1 day 12:37:59 0.9% 2017-04-25 19:59:08.201 INFO UPLOAD_PROGRESS Skipped chunk 10544 size 9810784, 3 9.17MB/s 1 day 12:37:35 0.9% 2017-04-25 19:59:08.277 INFO UPLOAD_PROGRESS Skipped chunk 10545 size 3463709, 3 9.17MB/s 1 day 12:37:27 0.9% 2017-04-25 19:59:08.566 INFO UPLOAD_PROGRESS Skipped chunk 10546 size 16777216, 39.19MB/s 1 day 12:36:46 0.9% 2017-04-25 19:59:08.643 INFO UPLOAD_PROGRESS Skipped chunk 10547 size 2249968, 3 9.19MB/s 1 day 12:36:40 0.9% 2017-04-25 19:59:08.751 INFO UPLOAD_PROGRESS Skipped chunk 10548 size 5949424, 3 9.19MB/s 1 day 12:36:26 0.9% 2017-04-25 19:59:08.859 INFO UPLOAD_PROGRESS Skipped chunk 10549 size 5380412, 3 9.20MB/s 1 day 12:36:13 0.9% 2017-04-25 19:59:29.899 ERROR UPLOAD_CHUNK Failed to find the path for the chunk 14cc12c7da7ecd9745bc46d64e58d6fe7211b5a541a260ec65cf5582649a7eb7: Head https:// storage.googleapis.com/[redacted]/[redacted]/chunks/14cc12c7da7ecd9745bc46d64e5 8d6fe7211b5a541a260ec65cf5582649a7eb7: dial tcp 74.125.21.128:443: connectex: A connection attempt failed because the connected party did not properly respond a fter a period of time, or established connection failed because connected host h as failed to respond.
gchen Apr 26 11:08AM 2017
So the retry mechanism didn't kick in. I found out that the shouldRetry function didn't catch the error (type *net.OpError with Op being "dial").
Please download the rebuilt 1.2.5 version (fixed by this commit) and try it again.
I've also started working on a new GCS storage based on the official Google client library. Normally this is at most one-day work but I'm busy with a big job (multithreaded downloading/uploading for all storages), so it will probably be done by next week.
Charles Apr 26 12:08PM 2017
Thanks a ton! I'll give it a try.
A week sounds pretty fast to me. The multithreading sounds great too!
Charles Apr 26 7:44PM 2017
Well it's been running for quite some time now. Looks like it may be working.
Charles Apr 27 9:28AM 2017
I'd say the fix is a success since it ran all night. It does seem like it is going to take quite some time though. (only a few days, so not that bad)
I stopped the process just now. I am going to wait to test the multithreading changes. Speed is an important factor. I need to know that I can throw a fist full of new data at it and not have to worry about my data not being backed up for a week or so while the process runs. I know I could create multiple backup sets, and I used to, but this option doesn't solve the problem as well as it used to due to personal workflow changes.
Charles May 5 6:43PM 2017
Not to sound impatient because I thought your original estimate of the changes was kind of short anyway, but any word on the next release with multithreading and new GCP support? I'm waiting on this before finishing my testing.
gchen May 6 3:47PM 2017
I finished the code changes for the multithreading support. Working on tests now and I hope to get the new version ready by the end of next week at the latest. The new GCP backend should also be out about the same time if everything goes as planned.
gchen May 14 9:27PM 2017
Version 2.0.0 (CLI only) is now available on the github page. The -threads option is now supported by all storage backends for both the backup and the restore commands.
To use the new GCS backend, set the storage url to gcs://bucket. You will need a token file, which can be downloaded by either authorizing Dupliacy to access your Google Cloud Storage account or by downloading it from your Google service account page.
Please let me know if you find any issues.
Charles May 15 11:34AM 2017
I have the token registered. Does this only support the root directory of the bucket? Currently I am in a directory 1 level beyond the root. I tried setting the storage to gcs://duplicacy/myFolderName, but receive the following error.
2017-05-15 12:29:56.333 ERROR STORAGE_CONFIG Failed to download the configuration file from the storage: googleapi: Error 400: Invalid bucket name: 'duplicacy/aydindirl-dc', invalid
Charles May 15 11:39AM 2017
PS without the /myFolderName, it of course just says that it has not been initialized.
gchen May 15 9:59PM 2017
I redid the 2.0.0 release. It should now accept a storage path after the bucket in the GCS storage url.
Charles May 16 9:19AM 2017
Seems to be working. I gave it 8 threads and over 5 TB to run until completion this time.
This warning message is a bit odd, but it didn't prevent it from running. Almost looks like it tried to scan the .duplicacy directory.
2017-05-16 08:14:53.593 INFO VSS_CREATE Creating a shadow copy for E:\ 2017-05-16 08:15:03.493 INFO VSS_DONE Shadow copy {15673AD1-7260-48E6-A415-8B782261620C} created 2017-05-16 08:15:03.494 INFO BACKUP_INDEXING Indexing E:\ 2017-05-16 08:15:03.542 INFO SNAPSHOT_FILTER Loaded 8 include/exclude pattern(s) 2017-05-16 08:19:39.194 INFO LIST_ATTRIBUTES Discarding file attributes 2017-05-16 08:20:53.327 WARN LIST_FAILURE Failed to list subdirectory: open E:\/.duplicacy\shadow\/System Volume Information: Access is denied.
gchen May 16 10:30AM 2017
Earlier versions of Go have trouble accessing the UNC paths to shadow volumes (such as \\?\GLOBALROOT\Device\HarddiskVolumeShadowCopy16) so a symbolic link .duplicacy\shadow has to be created which points to the newly created shadow volume. When it reads E:\/.duplicacy\shadow\/System Volume Information, it is just a shadow copy of E:\System Volume Information.
Charles May 17 8:24AM 2017
2017-05-16 18:57:31.352 INFO UPLOAD_PROGRESS Uploaded chunk 301210 size 4778347, 35.97MB/s 1 day 07:37:14 25.1% 2017-05-16 18:57:31.687 INFO UPLOAD_PROGRESS Uploaded chunk 301211 size 2300343, 35.97MB/s 1 day 07:37:14 25.1% 2017-05-16 18:57:32.906 INFO UPLOAD_PROGRESS Uploaded chunk 301212 size 6354316, 35.97MB/s 1 day 07:37:16 25.1% 2017-05-16 18:57:33.345 INFO UPLOAD_PROGRESS Uploaded chunk 301213 size 4656704, 35.97MB/s 1 day 07:37:19 25.1% 2017-05-16 18:57:33.707 INFO UPLOAD_PROGRESS Uploaded chunk 301214 size 2680479, 35.97MB/s 1 day 07:37:19 25.1% 2017-05-16 18:57:34.101 INFO UPLOAD_PROGRESS Uploaded chunk 301215 size 6142834, 35.97MB/s 1 day 07:37:21 25.1% 2017-05-16 18:57:34.979 INFO UPLOAD_PROGRESS Uploaded chunk 301216 size 11652732, 35.97MB/s 1 day 07:37:20 25.1% 2017-05-16 18:57:35.811 INFO UPLOAD_PROGRESS Uploaded chunk 301217 size 10109932, 35.97MB/s 1 day 07:37:22 25.1% 2017-05-16 18:57:36.263 INFO UPLOAD_PROGRESS Uploaded chunk 301218 size 4170093, 35.97MB/s 1 day 07:37:24 25.1% 2017-05-16 18:57:37.031 INFO UPLOAD_PROGRESS Uploaded chunk 301219 size 11407268, 35.97MB/s 1 day 07:37:26 25.1% 2017-05-16 18:57:37.451 INFO UPLOAD_PROGRESS Uploaded chunk 301220 size 2712485, 35.97MB/s 1 day 07:37:26 25.1% 2017-05-16 18:57:37.934 INFO UPLOAD_PROGRESS Uploaded chunk 301221 size 4371216, 35.97MB/s 1 day 07:37:25 25.1% 2017-05-16 18:57:38.252 INFO UPLOAD_PROGRESS Uploaded chunk 301222 size 1677083, 35.97MB/s 1 day 07:37:28 25.1% 2017-05-16 18:57:39.348 INFO UPLOAD_PROGRESS Uploaded chunk 301223 size 16777216, 35.97MB/s 1 day 07:37:29 25.1% 2017-05-16 18:57:39.751 INFO UPLOAD_PROGRESS Uploaded chunk 301224 size 4283336, 35.97MB/s 1 day 07:37:29 25.1% 2017-05-16 18:57:40.228 INFO UPLOAD_PROGRESS Uploaded chunk 301225 size 7206354, 35.97MB/s 1 day 07:37:31 25.1% 2017-05-16 18:57:40.579 INFO UPLOAD_PROGRESS Uploaded chunk 301226 size 2676535, 35.97MB/s 1 day 07:37:31 25.1% 2017-05-16 18:57:40.933 INFO UPLOAD_PROGRESS Uploaded chunk 301227 size 2012126, 35.97MB/s 1 day 07:37:31 25.1% 2017-05-16 18:57:41.398 INFO UPLOAD_PROGRESS Uploaded chunk 301228 size 2848430, 35.96MB/s 1 day 07:37:33 25.1% 2017-05-16 18:57:41.729 INFO UPLOAD_PROGRESS Uploaded chunk 301229 size 1334207, 35.96MB/s 1 day 07:37:33 25.1% 2017-05-16 18:57:42.003 INFO UPLOAD_PROGRESS Uploaded chunk 301230 size 1925156, 35.96MB/s 1 day 07:37:36 25.1% 2017-05-16 18:57:43.156 INFO UPLOAD_PROGRESS Uploaded chunk 301231 size 11814879, 35.96MB/s 1 day 07:37:38 25.1% 2017-05-16 18:57:43.719 INFO UPLOAD_PROGRESS Uploaded chunk 301232 size 3451290, 35.96MB/s 1 day 07:37:37 25.1% 2017-05-16 18:57:44.477 INFO UPLOAD_PROGRESS Uploaded chunk 301233 size 9205176, 35.96MB/s 1 day 07:37:39 25.1% 2017-05-16 18:57:44.832 INFO UPLOAD_PROGRESS Uploaded chunk 301234 size 1699071, 35.96MB/s 1 day 07:37:39 25.1% 2017-05-16 18:57:45.168 INFO UPLOAD_PROGRESS Uploaded chunk 301235 size 1564499, 35.96MB/s 1 day 07:37:42 25.1% 2017-05-16 18:57:45.964 INFO UPLOAD_PROGRESS Uploaded chunk 301236 size 9931223, 35.96MB/s 1 day 07:37:41 25.1% 2017-05-16 18:57:46.275 INFO UPLOAD_PROGRESS Uploaded chunk 301237 size 1266517, 35.96MB/s 1 day 07:37:44 25.1% 2017-05-16 18:57:47.102 INFO UPLOAD_PROGRESS Uploaded chunk 301238 size 14956466, 35.96MB/s 1 day 07:37:45 25.1% 2017-05-16 18:57:47.524 INFO UPLOAD_PROGRESS Uploaded chunk 301239 size 5600455, 35.96MB/s 1 day 07:37:44 25.1% 2017-05-16 18:57:48.651 INFO UPLOAD_PROGRESS Uploaded chunk 301240 size 16777216, 35.96MB/s 1 day 07:37:46 25.1% 2017-05-16 18:57:49.250 INFO UPLOAD_PROGRESS Uploaded chunk 301241 size 8099832, 35.96MB/s 1 day 07:37:48 25.1% 2017-05-16 18:57:49.556 INFO UPLOAD_PROGRESS Uploaded chunk 301242 size 1375306, 35.96MB/s 1 day 07:37:48 25.1% 2017-05-16 18:57:49.895 INFO UPLOAD_PROGRESS Uploaded chunk 301243 size 2291442, 35.96MB/s 1 day 07:37:47 25.1% 2017-05-16 18:57:50.226 INFO UPLOAD_PROGRESS Uploaded chunk 301244 size 2074003, 35.96MB/s 1 day 07:37:50 25.1% 2017-05-16 18:57:50.699 INFO UPLOAD_PROGRESS Uploaded chunk 301245 size 6310147, 35.96MB/s 1 day 07:37:49 25.1% 2017-05-16 18:57:51.128 INFO UPLOAD_PROGRESS Uploaded chunk 301246 size 6102071, 35.96MB/s 1 day 07:37:52 25.1% 2017-05-16 18:57:51.545 INFO UPLOAD_PROGRESS Uploaded chunk 301247 size 3112765, 35.96MB/s 1 day 07:37:52 25.1% 2017-05-16 18:57:51.860 INFO UPLOAD_PROGRESS Uploaded chunk 301248 size 2296881, 35.96MB/s 1 day 07:37:51 25.1% 2017-05-16 18:57:52.269 INFO UPLOAD_PROGRESS Uploaded chunk 301249 size 2276301, 35.96MB/s 1 day 07:37:54 25.1% 2017-05-16 18:57:52.693 INFO UPLOAD_PROGRESS Uploaded chunk 301250 size 5147593, 35.96MB/s 1 day 07:37:53 25.1% 2017-05-16 18:57:53.049 INFO UPLOAD_PROGRESS Uploaded chunk 301251 size 3391892, 35.96MB/s 1 day 07:37:56 25.1% 2017-05-16 18:57:53.919 INFO UPLOAD_PROGRESS Uploaded chunk 301252 size 14985564, 35.96MB/s 1 day 07:37:54 25.1% 2017-05-16 18:57:54.371 INFO UPLOAD_PROGRESS Uploaded chunk 301253 size 3663569, 35.96MB/s 1 day 07:37:57 25.1% 2017-05-16 18:57:54.660 INFO UPLOAD_PROGRESS Uploaded chunk 301254 size 1231399, 35.96MB/s 1 day 07:37:57 25.1% 2017-05-16 18:57:55.354 INFO UPLOAD_PROGRESS Uploaded chunk 301255 size 8671454, 35.96MB/s 1 day 07:37:59 25.1% 2017-05-16 18:57:55.700 INFO UPLOAD_PROGRESS Uploaded chunk 301256 size 2874809, 35.96MB/s 1 day 07:37:59 25.1% 2017-05-16 18:57:56.028 INFO UPLOAD_PROGRESS Uploaded chunk 301257 size 1678656, 35.95MB/s 1 day 07:38:02 25.1% 2017-05-16 18:57:56.620 INFO UPLOAD_PROGRESS Uploaded chunk 301258 size 6741827, 35.95MB/s 1 day 07:38:01 25.1% 2017-05-16 18:57:56.917 INFO UPLOAD_PROGRESS Uploaded chunk 301259 size 2216930, 35.95MB/s 1 day 07:38:01 25.1% 2017-05-16 18:57:57.072 ERROR UPLOAD_CHUNK Failed to upload the chunk 3c8bc56bb9bbd558583dc50ff822647beb27107e1e261310ee08d1faec545ea3: googleapi: Error 503: Backend Error, backendError
Charles May 17 8:29AM 2017
Ran for about 10 hrs before getting this error (see log above). First uploaded chunk was about 5 hrs before the error, the rest was all skipped.
2017-05-16 13:58:40.095 INFO UPLOAD_PROGRESS Skipped chunk 268774 size 6890693, 59.92MB/s 19:43:18 22.2% 2017-05-16 13:58:40.180 INFO UPLOAD_PROGRESS Skipped chunk 268775 size 4136196, 59.92MB/s 19:43:18 22.2% 2017-05-16 13:58:40.231 INFO UPLOAD_PROGRESS Skipped chunk 268776 size 4459409, 59.92MB/s 19:43:18 22.2% 2017-05-16 13:58:40.432 INFO UPLOAD_PROGRESS Skipped chunk 268777 size 11993642, 59.92MB/s 19:43:17 22.2% 2017-05-16 13:58:40.582 INFO UPLOAD_PROGRESS Skipped chunk 268778 size 9719979, 59.92MB/s 19:43:16 22.2% 2017-05-16 13:58:40.679 INFO UPLOAD_PROGRESS Skipped chunk 268779 size 6204391, 59.92MB/s 19:43:16 22.2% 2017-05-16 13:58:40.731 INFO UPLOAD_PROGRESS Skipped chunk 268780 size 3143608, 59.92MB/s 19:43:16 22.2% 2017-05-16 13:58:40.805 INFO UPLOAD_PROGRESS Skipped chunk 268781 size 5303558, 59.92MB/s 19:43:15 22.2% 2017-05-16 13:58:40.849 INFO UPLOAD_PROGRESS Skipped chunk 268782 size 2129622, 59.92MB/s 19:43:15 22.2% 2017-05-16 13:58:40.891 INFO UPLOAD_PROGRESS Skipped chunk 268783 size 3417456, 59.92MB/s 19:43:15 22.2% 2017-05-16 13:58:40.950 INFO UPLOAD_PROGRESS Skipped chunk 268784 size 7167851, 59.92MB/s 19:43:14 22.2% 2017-05-16 13:58:41.026 INFO UPLOAD_PROGRESS Skipped chunk 268785 size 5628600, 59.92MB/s 19:43:17 22.2% 2017-05-16 13:58:41.152 INFO UPLOAD_PROGRESS Skipped chunk 268786 size 9803939, 59.92MB/s 19:43:17 22.2% 2017-05-16 13:58:41.197 INFO UPLOAD_PROGRESS Skipped chunk 268787 size 1859756, 59.92MB/s 19:43:17 22.2% 2017-05-16 13:58:41.247 INFO UPLOAD_PROGRESS Skipped chunk 268788 size 4054580, 59.92MB/s 19:43:16 22.2% 2017-05-16 13:58:41.356 INFO UPLOAD_PROGRESS Skipped chunk 268789 size 10461919, 59.92MB/s 19:43:16 22.2% 2017-05-16 13:58:41.610 INFO UPLOAD_PROGRESS Skipped chunk 268790 size 9315676, 59.92MB/s 19:43:15 22.2% 2017-05-16 13:58:41.703 INFO UPLOAD_PROGRESS Skipped chunk 268791 size 7178719, 59.92MB/s 19:43:14 22.2% 2017-05-16 13:58:41.750 INFO UPLOAD_PROGRESS Skipped chunk 268792 size 2199389, 59.92MB/s 19:43:14 22.2% 2017-05-16 13:58:41.803 INFO UPLOAD_PROGRESS Skipped chunk 268793 size 2164017, 59.92MB/s 19:43:14 22.2% 2017-05-16 13:58:41.847 INFO UPLOAD_PROGRESS Skipped chunk 268794 size 5326768, 59.92MB/s 19:43:14 22.2% 2017-05-16 13:58:42.405 INFO UPLOAD_PROGRESS Uploaded chunk 268795 size 3133691, 59.92MB/s 19:43:17 22.2% 2017-05-16 13:58:42.980 INFO UPLOAD_PROGRESS Uploaded chunk 268796 size 6482551, 59.92MB/s 19:43:16 22.2% 2017-05-16 13:58:43.388 INFO UPLOAD_PROGRESS Uploaded chunk 268797 size 2773082, 59.92MB/s 19:43:20 22.2% 2017-05-16 13:58:43.751 INFO UPLOAD_PROGRESS Uploaded chunk 268798 size 1462498, 59.92MB/s 19:43:20 22.2% 2017-05-16 13:58:44.112 INFO UPLOAD_PROGRESS Uploaded chunk 268799 size 1810903, 59.92MB/s 19:43:23 22.2% 2017-05-16 13:58:44.627 INFO UPLOAD_PROGRESS Uploaded chunk 268800 size 3987189, 59.92MB/s 19:43:23 22.2% 2017-05-16 13:58:45.239 INFO UPLOAD_PROGRESS Uploaded chunk 268801 size 5770960, 59.91MB/s 19:43:26 22.2% 2017-05-16 13:58:45.729 INFO UPLOAD_PROGRESS Uploaded chunk 268802 size 3485746, 59.91MB/s 19:43:26 22.2%
gchen May 17 9:16PM 2017
Although the official GCS client library claims that a retry mechanism is in place for all methods, after some code digging I came to the conclusion that at least the upload method doesn't support it because of the way the code is written.
I'll add a retry logic to the outer layer in the GCS storage backend. Should be done by the end of tomorrow.
Charles May 18 9:19AM 2017
I thought it was weird when I received the message because I had also read in like one of the first paragraphs of their documentation that all of their requests implement an infinite retry. Thanks for looking into it.
Charles May 18 10:27AM 2017
On another note.. Unfortunately it's hard to get a grasp on how fast this application is and how long it has left since I have not gotten it to complete on my larger data set yet.
I did a basic verify and it thinks that I have 200 GB uploaded while GCP reports 1.21 TiB which I'm guessing is because each time it fails and restarts it removes the incomplete snapshot and starts over again, skipping existing chunks with the pre-upload check. This throws off the time estimate quite a bit. It also slows down a retry for such a large dataset quite a bit. I'm having to wait for a lot of skipped chunks before it starts uploading again.
From what I can tell from my smaller backup, it will not take as much time to check every single chunk. In fact, it doesn't seem to do that at all. Can you confirm this behavior? I'd hate to get all of this uploaded only to find out that each backup takes a couple of days.
Assuming that backups after the initial are faster, is it possible to support partial backups whether due to failure or OS restart or other reasons? I suppose the chunks that were still in progress would have to be re-checked/ uploaded in case they didn't finish, but it would make this type of situation much faster.
Lastly, with this much data, should I consider changing the chunk size? This may be subject to opinion and situation. I have a mix of many small files as well as some extra large files and every size in between (probably up to ~100 GB for 1 file). I don't mind starting the upload over if it would be better. I was thinking about doing that anyway to get an idea of how fast the backup is.
gchen May 18 8:55PM 2017
If the previous backup successfully finished, Duplicacy will just check the timestamp and size of each file to see if any of them changed from the last backup. It will then run the chunking algorithm on changed files and upload chunks created from these files. This is the default behavior unless the -hash option is specified.
However, if previous backups all failed for some reason, Duplicacy will have to run the chunk algorithm on all files. This is unfortunate as you can see from your tests that a considerable amount of time is spent just to recreate all chunks and then find out they all had already been uploaded. The solution I have in mind is to save the incomplete snapshot locally, which stores the files that have been scanned and uploaded. And then when you specify a -resume option, it will load this incomplete snapshot and only work on files not included or different from those in that snapshot.
gchen May 18 10:20PM 2017
I uploaded a new 2.0.0 build that implements the retry logic for uploading chunks. Please give it a try.
Charles May 19 7:05AM 2017
It's running now. It will be about 5 hours again before it starts uploading I'm guessing.
One thing I noticed that is not really a big deal, but may be worth noting in the documentation/ guide what the limitations are (if any): I upped the thread count from 8 to 12 to try to get passed the skipped chunks part faster and it actually slowed down significantly. I went from 10-20 chunks per second to 1-10. Looking in the task manager, It seemed that only 1 core was being used at a time with 12 threads specified, but when I changed it back to 8, several are being used again. 12 is completely unnecessary and I don't think I want to use that much processing power anyway, but I thought I'd let you know what I observed.
Did you have any recommendation on chunk size? Should I leave it default? TBH, variable sized chunking is a bit hard to conceptualize how that even works in the deduplication processes.
Charles May 19 7:11AM 2017
Maybe I should have just let it keep running at 12. 8 eventually did the same thing (slowed down and started using only about 20% of 1 core. It's probably just processing a larger file or something when it does that and only 1 thread can access it at a time so it sort of waits. So maybe there is no problem there.
gchen May 19 9:51PM 2017
The -threads option only sets the number of uploading threads. There is only one chunking thread that splits files into chunks and then dispatches them to the uploading threads. Therefore, I doubt 8 or 12 uploading threads would make a big difference when all chunks have been uploaded before.
The number one reason to use a large chunk size is to reduce the overhead of uploading chunks. This is because Duplicacy performs 2 API calls for each new chunk, one for checking the existence and the other for actually uploading the chunk. So the larger the chunk, the more time will be spent on the uploading.
In your case you can use multiple threads to take advantage of your fast uplink so I don't think you need to increase the chunk size. On the other hand, if you have many files smaller than the default chunk size that may change frequently, you may consider decreasing the chunk size instead. Duplicacy may pack multiple small files into a single chunk (think of this as if it were packing all files, large or small, into a huge tar file and then running the variable-size chunking algorithm on the tar file), so these small files will not see the benefit of deduplication between changes.
Charles May 21 3:48PM 2017
That makes sense. Thanks for the explanation. Without consistent data sizes and uses I guess I'm kind of in limbo. I'll just stick with the default for now. If I'm able to achieve a good amount of deduplication, then it'll be worth it. If not, I may switch it to a higher chunk size and re-upload after some testing while I still have some GCP trial credit. I may create 64 GB of data in one sitting and the faster that gets backed up the better. We'll see.
It seems to be working now. It says it has been running since Friday and still has a day and a half left (Sunday now) with 8 threads on a Gigabit connection. Not the fastest backup I've tried, but very possibly the fastest I've tried at this deduplication level. We will see when it finishes.
2017-05-21 16:44:00.530 INFO UPLOAD_PROGRESS Skipped chunk 714171 size 16777216, 17.23MB/s 1 day 12:48:47 58.2%
Charles May 22 10:18AM 2017
Another thought occurs to me and perhaps you can provide some insight. Since we know the program is working now, how much of a speed increase do you think we would see if I turned off the detailed logging?
Even though I am using multiple threads, there has to be some point in the application where a context switch takes place to write to the log file (edit: to the screen which I am piping to a log file). There are a number of techniques that could be used to speed this process along but from my own experience, something like that would be implemented further along in the incremental development process.
Do you have any thoughts or experience on how this would affect backup speeds for the current iteration?
PS Thank you for your excellent level of support and putting up with my constant curiosities about the inner workings and my abrasive opinions and questions. Obviously my interest in the program you have built goes beyond getting my data backed up. I can't wait to get my hands on the source code.
2017-05-22 10:43:54.064 INFO UPLOAD_PROGRESS Uploaded chunk 846122 size 2431489, 15.30MB/s 1 day 05:58:09 69.8% 2017-05-22 10:43:54.555 INFO UPLOAD_PROGRESS Uploaded chunk 846123 size 4517584, 15.30MB/s 1 day 05:58:09 69.8% 2017-05-22 10:43:55.305 INFO UPLOAD_PROGRESS Uploaded chunk 846124 size 5912045, 15.30MB/s 1 day 05:58:08 69.8% 2017-05-22 10:43:55.892 INFO UPLOAD_PROGRESS Uploaded chunk 846125 size 2124090, 15.30MB/s 1 day 05:58:08 69.8% 2017-05-22 10:43:56.322 INFO UPLOAD_PROGRESS Uploaded chunk 846126 size 1430229, 15.30MB/s 1 day 05:58:09 69.8% 2017-05-22 10:43:56.764 INFO UPLOAD_PROGRESS Uploaded chunk 846127 size 5855485, 15.30MB/s 1 day 05:58:08 69.8% 2017-05-22 10:43:57.201 INFO UPLOAD_PROGRESS Uploaded chunk 846128 size 1945520, 15.30MB/s 1 day 05:58:08 69.8% 2017-05-22 10:43:57.678 INFO UPLOAD_PROGRESS Uploaded chunk 846129 size 4581043, 15.30MB/s 1 day 05:58:08 69.8% 2017-05-22 10:43:58.094 INFO UPLOAD_PROGRESS Uploaded chunk 846130 size 5970921, 15.30MB/s 1 day 05:58:08 69.8% 2017-05-22 10:43:58.406 INFO UPLOAD_PROGRESS Uploaded chunk 846131 size 2172488, 15.30MB/s 1 day 05:58:08 69.8% 2017-05-22 10:43:58.806 INFO UPLOAD_PROGRESS Uploaded chunk 846132 size 2962683, 15.30MB/s 1 day 05:58:07 69.8% 2017-05-22 10:43:59.287 INFO UPLOAD_PROGRESS Uploaded chunk 846133 size 4371412, 15.30MB/s 1 day 05:58:07 69.8% 2017-05-22 10:43:59.678 INFO UPLOAD_PROGRESS Uploaded chunk 846134 size 4819426, 15.30MB/s 1 day 05:58:07 69.8% 2017-05-22 10:44:00.157 INFO UPLOAD_PROGRESS Uploaded chunk 846135 size 5437442, 15.30MB/s 1 day 05:58:07 69.8% 2017-05-22 10:44:01.048 INFO UPLOAD_PROGRESS Uploaded chunk 846136 size 13229446, 15.30MB/s 1 day 05:58:06 69.8% 2017-05-22 10:44:01.449 INFO UPLOAD_PROGRESS Uploaded chunk 846137 size 4054628, 15.30MB/s 1 day 05:58:06 69.8% 2017-05-22 10:44:01.900 INFO UPLOAD_PROGRESS Uploaded chunk 846138 size 1072854, 15.30MB/s 1 day 05:58:06 69.8% 2017-05-22 10:44:02.522 INFO UPLOAD_PROGRESS Uploaded chunk 846139 size 7562051, 15.30MB/s 1 day 05:58:05 69.8% 2017-05-22 10:44:02.994 INFO UPLOAD_PROGRESS Uploaded chunk 846140 size 2883132, 15.30MB/s 1 day 05:58:05 69.8% 2017-05-22 10:44:03.398 INFO UPLOAD_PROGRESS Uploaded chunk 846141 size 3689545, 15.30MB/s 1 day 05:58:05 69.8% 2017-05-22 10:44:03.753 INFO UPLOAD_PROGRESS Uploaded chunk 846142 size 1576206, 15.30MB/s 1 day 05:58:05 69.8% 2017-05-22 10:44:04.233 INFO UPLOAD_PROGRESS Uploaded chunk 846143 size 3350373, 15.30MB/s 1 day 05:58:05 69.8% 2017-05-22 10:44:04.584 INFO UPLOAD_PROGRESS Uploaded chunk 846144 size 2011109, 15.30MB/s 1 day 05:58:05 69.8% 2017-05-22 10:44:04.928 INFO UPLOAD_PROGRESS Uploaded chunk 846145 size 2492654, 15.30MB/s 1 day 05:58:05 69.8% 2017-05-22 10:44:05.260 INFO UPLOAD_PROGRESS Uploaded chunk 846146 size 1675592, 15.30MB/s 1 day 05:58:05 69.8%
gchen May 22 12:32PM 2017
I don't think detailed logging would slow down the backup to the degree that it becomes noticeable, although there is no data to back this claim.
The upload speed is a bit slow for a 1 Gbps uplink. I'll add performance profiling to my to do list.
gchen May 22 9:50PM 2017
The source code is now available from the github repository
Charles May 23 3:29PM 2017
Oh how exciting. I'll have to wait to look at that until this weekend most likely.
FACEPALM But it's so close.....
2017-05-23 11:59:28.582 INFO UPLOAD_PROGRESS Uploaded chunk 1112591 size 10407000, 14.82MB/s 07:56:32 92.2% 2017-05-23 11:59:29.279 INFO UPLOAD_PROGRESS Uploaded chunk 1112592 size 5680803, 14.82MB/s 07:56:32 92.2% 2017-05-23 11:59:29.579 INFO UPLOAD_PROGRESS Uploaded chunk 1112593 size 3775219, 14.82MB/s 07:56:32 92.2% 2017-05-23 11:59:29.907 INFO UPLOAD_PROGRESS Uploaded chunk 1112594 size 3586406, 14.82MB/s 07:56:32 92.2% 2017-05-23 11:59:30.399 INFO UPLOAD_PROGRESS Uploaded chunk 1112595 size 10377805, 14.82MB/s 07:56:31 92.2% 2017-05-23 11:59:30.529 ERROR CHUNK_MAKER Failed to read 0 bytes: read E:\/.duplicacy\shadow\\/Shares/Access/VirtualMachines/MINTMINE/MINTMINE-Snapshot1.vmem: The device is not ready. 2017-05-23 11:59:30.669 INFO UPLOAD_PROGRESS Uploaded chunk 1112596 size 1966714, 14.82MB/s 07:56:31 92.2% 2017-05-23 12:01:01.273 INFO VSS_DELETE The shadow copy has been successfully deleted
gchen May 24 7:16AM 2017
There is probably no need to backup the .vmem file. I'm not sure if that was a VSS issue (i.e., the VM hypervisor didn't implement the VSS writer correctly) or if that kind of file was not supposed to be readable even with VSS enabled. In any case, you can add an exclude pattern -*.vmem to .duplicacy/filters and try again.
Charles May 24 7:48AM 2017
I suppose this will work as a temporary solution. None of my other backup solutions have had a problem with this though. I don't know how they handle it.
gchen May 24 10:22AM 2017
Is that a VMware workstation file? I'll try it myself.
Charles May 24 2:53PM 2017
It is. Based on the file name it's from where I took a snapshot of it. vmem is probably the snapshot of the ram since the machine was running.
Charles May 25 3:47PM 2017
How deflating
2017-05-25 15:37:28.912 INFO UPLOAD_PROGRESS Uploaded chunk 1056557 size 11039277, 51.90MB/s 01:18:36 95.2% 2017-05-25 15:37:29.321 INFO UPLOAD_PROGRESS Uploaded chunk 1056558 size 2101654, 51.90MB/s 01:18:36 95.2% 2017-05-25 15:37:30.374 INFO UPLOAD_PROGRESS Uploaded chunk 1056559 size 10895912, 51.90MB/s 01:18:36 95.2% 2017-05-25 15:37:30.793 INFO UPLOAD_PROGRESS Uploaded chunk 1056560 size 4828920, 51.90MB/s 01:18:36 95.2% 2017-05-25 15:37:31.161 INFO UPLOAD_PROGRESS Uploaded chunk 1056561 size 2174038, 51.90MB/s 01:18:36 95.2% 2017-05-25 15:37:31.553 INFO UPLOAD_PROGRESS Uploaded chunk 1056562 size 3060655, 51.90MB/s 01:18:36 95.2% 2017-05-25 15:37:34.836 ERROR UPLOAD_CHUNK Failed to find the path for the chunk d470effc0ed2eacb3f904e8dfa3ee3baddf93681dc2c5e25f5f2cf4fbe89dc4b: Get https://www.googleapis.com/storage/v1/b/[redacted]/o/[redacted]%2Fchunks%2Fd470effc0ed2eacb3f904e8dfa3ee3baddf93681dc2c5e25f5f2cf4fbe89dc4b?alt=json&projection=full: unexpected EOF
gchen May 25 10:30PM 2017
So this unexpected EOF needs to be caught and retried. This is likely a temporary error and may not happen if you run the backup again. However, I think it is really annoying that the backup has to start from the beginning even though it is faster than the actual uploading with all those skipped chunks. So the next major task for me is to implement fast resume. My plan is that it should be done by the end of next week.
I would also suggest that you create a separate repository for your virtual machines that backs up to a different storage location. For the storage you can set all three chunk size parameters to the same value. This will enable the fixed-size chunking algorithm which should be much faster than the variable-size chunking algorithm. We have a backup tool called Vertical Backup that does exactly that for VMWare ESXi.
Charles May 26 7:10AM 2017
Agreed. It went for 18 hours only to have a temporary internet outage so I had to start it again. Then it took over 24 hours just to run into this error at 95%. My server has been threatening me that it is going to restart for updates too... which I keep cancelling via command line.
I can create a separate backup set for my vm's, but they don't really change often. These are mostly backups of vm's that I no longer use from an old version of vmware workstation that I got when I was in college. I might pull one down and fire it up to rebuild a discontinued piece of software that I still use or something, but for the most part they won't change. Most are growing disks with linux on them, so they aren't too big. It may not be a bad idea to pull them out into a separate backup set, but I have a feeling that once the initial backup is complete, it won't really slow my backup too much even if I change one of them.. unless you know something I don't.
I still use vm's for dev envs if I start a home project, so I will surely use a separate backup set for any of those (none right now), so it's good to know to set them to a fixed size for speed. I don't know if I want to use a separate piece of software though. Sounds like too much overhead for home use. Plus I'm not using esxi.
Charles May 28 11:30AM 2017
It finally finished. I started running it again. It looks like it is rechecking everything. Is this always going to be the case?
2017-05-27 22:34:49.190 INFO UPLOAD_PROGRESS Skipped chunk 298338 size 5996574, 70.00MB/s 15:12:37 26.1% 2017-05-27 22:34:49.245 INFO UPLOAD_PROGRESS Skipped chunk 298339 size 4719426, 70.00MB/s 15:12:37 26.1% 2017-05-27 22:34:49.267 INFO UPLOAD_PROGRESS Skipped chunk 298340 size 2166734, 70.00MB/s 15:12:37 26.1% 2017-05-27 22:34:49.293 INFO UPLOAD_PROGRESS Skipped chunk 298341 size 2612248, 70.00MB/s 15:12:37 26.1% 2017-05-27 22:34:49.360 INFO UPLOAD_PROGRESS Skipped chunk 298342 size 5740648, 70.00MB/s 15:12:37 26.1% 2017-05-27 22:34:49.425 INFO UPLOAD_PROGRESS Skipped chunk 298343 size 5551157, 70.00MB/s 15:12:36 26.1% 2017-05-27 22:34:49.467 INFO UPLOAD_PROGRESS Skipped chunk 298344 size 3867387, 70.00MB/s 15:12:36 26.1% 2017-05-27 22:34:49.479 INFO UPLOAD_PROGRESS Skipped chunk 298345 size 1379893, 70.00MB/s 15:12:36 26.1% 2017-05-27 22:34:49.551 INFO UPLOAD_PROGRESS Skipped chunk 298346 size 5980379, 70.00MB/s 15:12:36 26.1% 2017-05-27 22:34:49.577 INFO UPLOAD_PROGRESS Skipped chunk 298347 size 2464162, 70.00MB/s 15:12:36 26.1% 2017-05-27 22:34:49.656 INFO UPLOAD_PROGRESS Skipped chunk 298348 size 6188076, 70.00MB/s 15:12:35 26.1% 2017-05-27 22:34:49.673 INFO UPLOAD_PROGRESS Skipped chunk 298349 size 1274697, 70.00MB/s 15:12:35 26.1% 2017-05-27 22:34:49.698 INFO UPLOAD_PROGRESS Skipped chunk 298350 size 1805073, 70.00MB/s 15:12:35 26.1% 2017-05-27 22:34:49.766 INFO UPLOAD_PROGRESS Skipped chunk 298351 size 3877210, 70.00MB/s 15:12:35 26.1%
gchen May 29 7:30PM 2017
By default the backup command should skip files having the same timestamp AND size as the previous version. Once the second backup finishes, you should be able to see which files have been uploaded again. If the timestamp or size didn't change but the file got uploaded again that it is a bug.
Charles May 30 8:59AM 2017
It took 22 hours for the second run..
2017-05-28 15:45:24.351 INFO BACKUP_STATS All chunks: 1106654 total, 5071G bytes; 166 new, 873,922K bytes, 233,652K bytes uploaded 2017-05-28 15:45:24.351 INFO BACKUP_STATS Total running time: 22:42:22
gchen May 30 9:34AM 2017
Are you using the -hash option? If so, Duplicacy will check every file regardless of their timestamps and sizes.
Charles May 30 10:39AM 2017
Ah. I did have that option in there. I'm running it from a batch file and completely overlooked that. User error. Ha.
Much better. I guess all I have left to do is schedule the backups and figure out how I can get e-mail notification of failures. That and check out the source code at some point. Thanks for all of your help.
2017-05-30 11:08:19.635 INFO BACKUP_STATS All chunks: 1106718 total, 5071G bytes; 69 new, 369,819K bytes, 77,936K bytes uploaded 2017-05-30 11:08:19.635 INFO BACKUP_STATS Total running time: 00:09:28