Currently there is an overhead when backing up high volumes of files (2 million+). The effect on the system is as follows:
Dependant on the directory structures and file count, there is a high usage of physical memory due to the fact that SQL Lite is used to manage the comparison of changed data via storing the entries of the indexes for the backup. This process loads the index entries in memory for the previous index, as well as the new index before committing to disk.
One can expect to see roughly 1GB of local disk space used per 1 million files, and as such 2GB of RAM used for the backup process per 1 Million files. Extrapolating these figures out to 5-10 million files, the RAM usage becomes quite high.
Should the relevant amount of memory not be available, the backup process slows due to the need to swap to disk. The end result typically is slower backups that are resource intensive. The more RAM the faster the backup, with a recommendation of at least 2.5x the size of the index database file being available in free RAM. More would speed up the backup.
Attix5 is currently working on removing these limitations, and this article serves as the current operational procedure to alleviate the need for high RAM volume, and also dramatically speeds up the backup. The table below illustrates the results.
|File count in selection||Procedure implemented||Backup Time||DB size on disk||File change||Min free RAM required|
|2,8 Million||No||3hrs 20min||4.4GB||1 added||11GB|
|2.8 Million||Yes (first backup after change)||40min 24 sec||55Mb||1 added||< 512MB|
|2.8 Million||Yes (second backup)||3 min||55MB||1 added||< 512MB|
|2.8 Million||Yes (second backup)||2 min 40 sec||55MB||1 added||< 512MB|
Procedure Steps and overview
The procedure involves applying a filter that will not backup any files older than a certain date.
Applying this will still scan all data in the selection, but because there are no changes and due to the fact that the data is not in the required date range, the full scan range index does not get written to the index database. Doing so alleviates the strain on the system, which speeds up the backup.
- Select a manageable section of data 1-2 million files for first backup and start the backup.
Ensure that this is a single backup task. i.e. E:\folder1 and E:\folder2
- Once this initial backup is complete the index file on the Storage Server�?Ts �?o<AccountGUID>\Backups�?� directory (e.g. C:\ProgramData\Attix5 Pro\SS_Service\Data\fa84613b-b72d-4beb-914d-54011f860063\Backups\) needs to be modified and all �?o0�?� (zeroes) after the date structure changed to �?of�?�. e.g. 20140219181638.00000000-0000-0000-0000-000000000000.index to 20140219181638. ffffffff-ffff-ffff-ffff-ffffffffffff.index. Doing so will flag the backup as a rollup.
- It is important to ensure that these backups are all new initial file additions and that there are no patched files in the backup.
- Now deselect the initial folders i.e. E:\folder1 and E:\folder2.
- Select the next folders for backup e.g. E:\Folder3 and E:\Folder4 and follow the same process in renaming the index after successful backup along with then deselecting Folder3 and Folder4.
- Continue with this process until all the required data has been backed up finishing off by renaming the last index as described in point nr 2.
- Now reselect all the relevant folders previously backed up. In this case E:\Folder1,2,3 and 4.
- Before starting a new backup ensure that the backup group is��set to retain all roll-ups��and that an exclusion date is set. An example would be to exclude all files older that the last two years from being backed up in the global exclusions section either on the client��or under profiling in the console.
Exclusion setting on Client:
Exclusiong via Profiling:
- Once this is done backups can be scheduled on a daily or more frequent basis and should now use considerably less memory resources and will be noticeably faster as described in the example table above.
Note: Index files should only be renamed to show as Roll-ups and not copied and then renamed.
Old Article ID: 251
Previous Views: 1575
Posted: 07 Mar, 2014 by Van Rensburg J.
Updated: 31 Mar, 2014 by Van Rensburg J.