Backup transfer deduplication can be enabled in ESE by going to Tools > Options > Performance.
- When enabled, this feature aims to transfer identical files only once to the Storage Platform by leveraging previous backups by the same account or other accounts.
- In addition, if VSS is enabled on the ESE Agent, files included in the backup selection that are identical on the same volume are sent only once.
The "transfer deduplication" phase (as indicated in the progress window during a backup) works as follows:
- Identify all files that must be processed for backup (by hashing all selected files and generating a preliminary index).
- Communicate with the account's StorageServer to determine which files it hasn't received yet (by using their hashed values from the preliminary index).
- Start processing all files to ensure no changes have occurred in the meantime.
- Send only the files identified in point 2 to the Storage Platform.
This method reduces the file transfer size, and prevents files from being processed twice, increasing disk activity during a backup. However, the comparison between the preliminary index (generated in point 1) and the index from the previous backup (retrieved from the StorageServer in point 2), which is required for deduplication to take place, may extend the backup duration, resulting in a slower rather than a faster backup.
- Note: When backup transfer deduplication is enabled, partial backups (as with incomplete or interrupted backups) are no longer automatically purged on the Storage Platform unless the backup was successful. This is to be expected, since transfer deduplication leverages files from previous backups, including partial ones, to deduplicate transfers of subsequent backups. Purge partial backups manually from the Storage Platform Console if disk usage is being adversely affected.
Interaction with local backup copies
If local backups are also enabled, no deduplication occurs to the local backup destination.
Interaction with VSS
If VSS is disabled, identical files within the same volume will be sent separately and will not be deduplicated.
To enable VSS, Open the ESE Agent and go to Tools > Options > Advanced.
- Note: VSS does not apply to network shares.