1073 - Archiving

Page contents

How it works
Benefits
The Archiving process
Archiving rules
How to activate Archiving
How to configure Archiving

FAQs
General
Rehydration
Deletion
Use cases
Reporting
Deactivation

 

 

Overview

Redstor's Archiving service saves disk space by removing redundant, obsolete and trivial (ROT) data. Archiving is managed from the ESE Agent interface.

How it works

Rules are set (e.g. older than x years, not accessed in x months) by which files are selected to be archived. Each eligible file is stubbed, i.e. replaced with a much smaller (“sparse”) file. The originals files are stored on Redstor’s Storage Platform but are immediately accessible from the user’s computer via the sparse files. When a sparse file is accessed, its relevant data is immediately rehydrated onto the user’s computer. Files can also be selected for bulk rehydration.

Please note: Archiving only works on 64-bit Windows operating systems. See our System Requirements and Compatibility Matrix for more details.

 

Benefits

• Archiving saves disk usage costs by moving data from high-cost primary storage (local) to the lower-cost Storage Platform - helps customers deal with growing volumes of data without purchasing more primary storage.
• Archived data is always available on demand on the originating computer.
• Data redundancy is ensured before archiving occurs as multiple copies of files are held in separate locations. 

See Archiving in action here:

 

The Archiving process

1. Compatibility and version checks are performed.
2. The Account is checked for an active Archiving licence.
3. ESE confirms that the data has been backed up.
4. ESE confirms that the mirroring of data has completed so that two copies of the data exist in the lower tier storage.
5. The rehydration service is installed and checked. If it already exists, it is updated to the appropriate version and its status verified before archiving can proceed.
6. If the above checks are successful, all eligible files are stubbed, based on the archiving selection.
7. Files that no longer fulfil archiving selection criteria are rehydrated.

Note:

• When stubs are accessed, only the relevant data is rehydrated. This results in a stub being partially rehydrated. Also, files removed from the backup selection are automatically rehydrated at the next cycle.
• Archiving is triggered after backups to ensure that archived data is consistent.
• Files required by critical applications and frequently modified will not be stubbed. For example, SQL Server and Microsoft Exchange database files.
• Logs are generated during the process and all actions taken can be reviewed in the Logs tab.

 

Archiving rules

A file will be stubbed if it:

• is in the archive selection
• is in the most recent backup on the StorageServer
• is in the most recent backup on the MirrorServer 
• is not in the system state selection (or selected by any VSS writer) 
• has not been recently modified (based on not accessed in x days)
• has not been recently accessed (based on not accessed in x days)
• is greater or equal to 1KB 
• is less than or equal to 64GB 
• is not open 

A file will be rehydrated if it:

• has recently been modified (based on not accessed in x days)
• is not in the archive selection 
• is not in the most recent backup on the StorageServer
• is not in the most recent backup on the MirrorServer
• is not a stub from this account

 

How to activate Archiving

You will need to enable Archiving for each collection/group you want to archive. For help with this, see Article 1497.

 

How to configure Archiving

To configure Archiving for an account, follow the steps in Article 1498.

 

Archiving FAQs

General

What is the relevance of last access time to Archiving?

Archiving uses the last access time of a file to determine if it should be archived or not. However, in most installations the last access times are not kept up to date by Windows. The Archiving service therefore monitors access to files, and when a file is accessed it will update the last access time itself. This means that last access times for files will be updated immediately when a file is accessed for whatever reason (for instance, right-click > Properties on an image file to get the dimensions).

Last access times are only updated once Archiving is enabled. This means that for the first Archiving run, it is possible that files that have been recently accessed are archived because they were not accessed AFTER Archiving was enabled. In such a case we work on whatever is the most recent - Date accessed or Date modified. Naturally, these files will get rehydrated the moment they are accessed. The last access time will then be updated to determine whether they need to be archived next time.

This time difference just means that more files may initially get archived than were intended or expected to be. There is a setting which introduces a 'lag time', effectively freezing the stubbing process for a specified number of days in order to minimise this behaviour. The default is 0 and this can only be set in the a5backup.properties file.

To get the driver installed and update the last access times without running an archive, use the Calculate Savings feature. This can be cancelled after the driver has been installed. 

 

What happens when anti-virus software scans or changes the files?

The filter driver sits below the anti-virus drivers, so we intercept all the calls, including those from anti-virus. Anti-virus software opens the files in such a way that we can see it has been read only for scanning purposes, in which case only the already rehydrated portions are returned. This means our Archiving service can co-exist with anti-virus software, and the anti-virus will not trigger the rehydration of files.

In some anti-virus software, you may need to adjust the security settings for Archiving to work properly. In ESET, for example, you need to select the option Preserve last access timestamp to keep the original access time of scanned files, instead of updating them. If this option is not enabled, files will not be stubbed.

 

What is the minimum file size criteria (if any) for stubbing and archiving a file, assuming that very small files (< 1 MB) don't need to be archived?

Files smaller than 1kB will not be stubbed.

 

What is the exact size of the stub file on disk?

Depending on how the volume is formatted, usually 4KB.

 

Are you limited to the file length path for Archiving?

The path can be anything, since the filter driver only uses file IDs. The file name can therefore have special characters and be of any length. It all looks the same to the filter driver.

 

Rehydration

 

If files are encrypted due to ransomware, can these be restored as normal?

We need to distinguish between whether ransomware encrypted the files before or after the archiving process.

• If ransomware kicks in AFTER archiving, the service will rehydrate the files as it reads them and then write encrypted data back. At that point, archiving is no longer relevant as those files are now local. To recover, you will do a normal restore as you would for a backup.
• If the files were encrypted BEFORE archiving but after backup, there would be no rehydration activity, so files could still be restored as normal.

 

In a disaster recovery scenario where disks/folders containing stubs have been lost, will the Agent restore the stubs or will the data be restored in a rehydrated state?

The full (rehydrated) file will be restored. We do create stub files first when you restore with InstantData Permanent, but those stubs are rehydrated as part of the restore. 

 

Once files are rehydrated, will they re-stub again after the archive is triggered?

Yes. If a stubbed file gets rehydrated and is not accessed again for the specified number of days, it will be stubbed again. It is best practice to have a retention period of at least a few weeks to prevent files that are regularly accessed from getting stubbed and rehydrated continually.

 

Deletion

How long can data be archived for?

For as long as the backups are kept (based on the set retention).

 

If a stub is deleted, will the file be deleted as well?

Yes. A stub and a full file are treated in the same way. As soon as you delete a stub/file, it will be removed from the backups and eventually get flushed out with a roll-up.

 

If I restore a deleted archived file, what will be restored - the archived file or the original file?

We restore only full files, not the archived stub files.

 

What happens if there is insufficient disk space to rehydrate the files when they get recalled from the backup selection?

This is a common scenario (called oversubscription), in which you have more archived (stubbed) data than your local disk could hold when rehydrated. In these cases files are hydrated until the space runs out.

 

What will happen on roll-ups?

Normally, stubbed files remain in the backup selection – this should ensure they are not flushed out with a roll-up. If the stubbed files do not remain in the backup selection, they will automatically get rehydrated and eventually rolled up. As a fail-safe, we have a list we maintain on the Platform which includes all archived files. Those files are then excluded from deletion when roll-ups take place.  

 

Use cases

Does Archiving work with file servers? Will machines that are accessing remotely trigger the rehydration process?

Yes, file servers are a brilliant candidate for Archiving since many of their files are not being used. If Archiving is run on the file server itself, even UNC access will trigger the rehydration process.

 

Can I archive files on a UNC share on a separate server?

No, the filter driver needs to be installed on the host that is serving the files, not the Agent accessing them.

 

What happens if stub files are moved outside of the backup selection to somewhere else on disk?

They will be detected as moved and will get rehydrated on the next archive run – that is why the full system is scanned.

 

What happens if stub files are moved outside of the backup selection to a UNC path?

The filter driver will detect that the file is being moved from local storage and will rehydrate the file to the target location. The source stub will then be removed.

 

How does Archiving work with local copy backups?

Local copies are not supported with Archiving. Although it is possible to enable both features, you may end up with archived files missing in the local copy due to stubbing.

If the two must be used concurrently, the lower-risk configuration is to only enable Archiving after the local copy functionality has been running for a few backups. However, even in this configuration there is still a risk.

It is also not possible to rehydrate from the local copy, as archived data is not stored in local copy backups.

 

Are there any limitations around connectivity?

Rehydration is not possible when both the StorageServer and the MirrorServer are inaccessible. 

 

Can we override the date and stub files immediately when their backup has been mirrored?

Only if the Date accessed and Date modified dates meet the archiving criteria. Third-party applications such as Total Commander can set these.

 

If an application is not working properly for a specific file after stubbing, can we immediately rehydrate it and then exclude the file from Archiving to prevent it from being stubbed again?

Currently, only folders can be excluded. However, if a file is regularly accessed it will not get stubbed in the first place.

 

What are the troubleshooting steps if files are not being stubbed?

If files that meet the archiving criteria are not being stubbed:

1. Check in your file explorer that both the Date accessed and Date modified dates for these files are older than the threshold.
   
   
   
   
2. Ensure that each file is backed up and mirrored after making any date changes. The timestamps on disk must match the latest backup.
3. Check the Archive logs in C:\ProgramData\Redstor Pro\Backup Client ESE\logs. The reason for any stubbing or rehydration failures should be shown in brackets.

 

Are read-only files stubbable?

Yes.

 

Does rehydration resume automatically after a power failure?

Yes, it will continue where it left off. 

 

Will rehydration run from the MirrorServer if the StorageServer is down?

Yes.

 

If a StorageServer is lost, will my data be automatically hydrated?

No, but you will not be able to archive any additional data until storage redundancy is restored.

 

Why don’t partially downloaded files get re-archived at the next backup?

Because their last access time is current after a read operation. When the last access time meets the archiving criteria, they will be re-archived.

 

Can I see how much data is partially rehydrated?

Yes, you can do this by looking at the file properties.

Is the filter driver only active during the archive task?

The filter driver is always running, so it always gets notified when reads and writes come through. However, we have ensured that there is almost zero overhead time when stub files are not being read or written, which means the driver will not slow down your machine. This process is tested automatically when you submit the driver for verification to Microsoft.

 

Can system files be set for exclusion by default to prevent system crashes?

You can select any folder for exclusion. The Windows folder is excluded by default, and we automatically exclude all files seen by VSS as system state files. However, this is no guarantee that the right files are being excluded, and we recommend that you exclude any folder accessed during boot.

 

How does the Redstor service handle existing archiving solutions and their stub files?

We have a setting whereby you can exclude third-party stubs from backups (and therefore from archiving).

 

Is the stubbing and rehydrating of EFS encrypted files possible?

No, stubbing is not supported for such files, and they will be skipped.

 

Reporting

How do I generate a report on the status of my archives and related information?

You can find this information in the Archiving logs (default location C:\ProgramData\Redstor Pro\Backup Client ESE\logs). These show which files have been stubbed, which have been rehydrated, etc.

Archiving information is also available on the Storage Platform Console. Here you can see Archived Data, Archived Files, and Last Archive date.

Note: If these columns are not currently visible, you can enable them by going to View > Customise Columns.

You can also schedule (or generate on demand) an Archiving report for your whole database, or for a specific group or account, by opening the Console in the Reports view.

 

Is it possible to see in the Storage Platform Console which customers have enabled Archiving?

Yes. 

 

If I archive a file, will there be a reduction in the volume of total data selected for backup,  and does this free up space for additional files?

Your total data selected for backup will remain the same, regardless of how much of it is archived.

For example, if you have a 3.5 GB file, your data selected will be 3.5 GB. If you archive that file so that it only takes up 10KB on disk, the data protected is still 3.5 GB because that is what is stored on the Storage Platform and associated with the ESE account.

We have added extra columns in the console so you can see how much of your data is archived. 

 

How will Archiving affect my storage usage per account on the Platform? For example, is data deduplicated between my backups and the archives?

Your data selection remains exactly the same, and the data protected stays unchanged too. You are basically specifying that a portion of your backed-up data be archived, but you are still storing the same amount of data on the Platform and that data is associated with your ESE account.

 

Do you have a data insight tool that can work out how much will be archived and how much space will be saved?

There are two ways to do this:

1. In ESE, under Archive > Show advanced options, there is a button that allows you to calculate your savings.

2. Use our Storage Analyser to scan your entire system and see how much you can save. A slider is used to set the Archiving threshold to however many days you want. This adjusts the graphs accordingly, revealing what effect Archiving will have on a particular server. 

 

Deactivation

I’ve tried Archiving, but I don’t like it and want to remove it. How do I do this?

If you want to keep all your data, the simplest way is to rehydrate it all and then disable the feature.

1. Click the Archive button, then click on Show advanced options.
2. Choose Rehydrate all stubs and click on Start Rehydration.
3. When the process has completed, run the rehydration again.
4. Finally, to disable the feature, go to Options > Archiving and uncheck the Stub backed up files checkbox labeled not accessed in the last.

 

What happens if my ESE Agent is uninstalled?

Any further stubbing will stop. All existing stubs will remain stubbed until they are accessed. Files will only get rehydrated when they are completely read from disk. Even when being written to disk, they will remain partially rehydrated unless fully overwritten.

 

Do you get a warning if you uninstall the Redstor software before rehydrating?

You can uninstall ESE, but the filter driver and rehydration service will remain on the server – there is no uninstall option for that, other than from the command line. At present, this requires a manual step where you must rehydrate all files first.

 

Can you reverse the process?

Yes, install the ESE Agent again and reconnect to the account. The rehydration service will be set up again and Archiving can resume.

 

What happens if the account has been deleted?

It will no longer be possible to rehydrate stubs or recover files when an account that has been using Archiving is deleted. To guard against this, it is impossible to delete an account from the Console if it has any stub files still archived. However, there are other ways to delete an account.

There will be times when an account needs to be automatically deleted from an evaluation group after a certain period of time. For that reason, you cannot enable Archiving on an evaluation account. But it is still possible to delete an account that has been archiving. This can be done by downgrading the account to an evaluation account in a group which has auto-deletion settings enabled. Please note that all stubs must be rehydrated before an account can be deleted.

 

What if I just disable the account instead?

Rehydration is not possible if an account or group is disabled. If this happens, users will not be able to access their archived data. Be mindful when Archiving.