1239 - Archiving Q&A

General

What is the relevance of last accessed time to Archiving?

Archiving uses the last accessed time of a file to determine if it should be archived or not. However, in most installations the last accessed times are not kept up to date by Windows. Archiving therefore monitors access to files and when a file is accessed it will update the last access time itself. This means that last accessed times for files will be updated immediately when a file is accessed for whatever reason (for instance, right-click > Properties on an image file to get the dimensions).

Last accessed times are only updated once Archiving is enabled. This means that for the first Archiving run, it is possible that files that have been recently accessed are actually still archived because they have not been accessed AFTER Archiving was enabled. In such a case we work on whatever is the most recent - Last Accessed or Last Modified. Of course, these files will get rehydrated the moment they are accessed. The last accessed time will then be updated to determine whether they are archived next time.

This time difference just means that more files may initially get archived than were intended or expected to be. There is a setting which introduces a 'lag time', effectively freezing the stubbing process for a specified number of days in order to minimise this behaviour. The default is 0 and this can only be set in the properties file.

To get the driver installed and update Last Accessed times without running an archive, use the Calculate Savings feature. This can be cancelled after the driver has been installed. 

What happens when anti-virus software scans or changes the files?

The filter driver sits below the anti-virus drivers, so we intercept all the calls, including those from anti-virus. Anti-virus software opens the files in such a way that we can see it’s only been read for scanning purposes, in which case only the already rehydrated portions are returned. This means our Archiving service can co-exist with anti-virus software, and the anti-virus won’t trigger the rehydration of the files.

In some anti-virus software, you may need to adjust the security settings for Archiving to work properly. In ESET, for example, you need to select the option Preserve last access timestamp to keep the original access time of scanned files, instead of updating them. If this option is not enabled, files will not be stubbed.

What is the minimum file size criteria (if any) for stubbing and archiving a file, assuming that very small files (< 1 MB) don't need to be archived?

Files smaller than 1kB will not be stubbed.

What is the exact size of the stub file on disk?

Depending on how the volume is formatted, usually 4KB.

Are you limited to the file length path for Archiving?

The path can be anything since the filter driver only uses file IDs. So the filename can have obscure characters or be any length. It all looks the same to the filter driver.

Rehydration 

If files are encrypted due to ransomware, can these be restored as normal?

We need to distinguish between whether ransomware encrypted the files before or after the archiving process.

• If ransomware kicks in AFTER archiving, it will rehydrate a file as it reads it and will then write encrypted data back. At that point, archiving is no longer relevant as those files are now local. To recover, you will do a normal restore as you would for a backup.
• If the files were encrypted BEFORE archiving but after backup, there would be no rehydration activity, so files could still be restored as normal.
  

In a disaster recovery scenario, would the Agent restore stubs where disks/folders containing stubs were lost or would data be restored in a rehydrated state?

At present it will restore the full file. We do create stub files first when you restore with InstantData Permanent, but those stubs are rehydrated as part of the restore. The restoring of stubs is on our development roadmap.

Once files are rehydrated, will they re-stub again after the archive is triggered?

Yes. If a stubbed file gets rehydrated and is not accessed again for the specified number of days, it will then be stubbed again.

This is why it is best practice to have a retention period of at least a few weeks, because files that are regularly being accessed could otherwise get stubbed, rehydrated, etc.

Deletion

How long can data be archived for?

As long as the backups are kept (based on the retention settings for the collection/group).

If a stub is deleted will the file be deleted as well?

Yes. A stub and a normal file are treated in the same way. As soon as you delete a stub/file it will be removed from the backups and eventually that file will get flushed out with the roll-ups.

If I restore a deleted archived file, what will be restored - the archived file or the original file?

For now we only restore full files, we do not just restore stubs.

What happens if there is insufficient disk space to rehydrate the files when they get recalled from the backup selection?

This is a likely scenario - we call it over-subscription - where you have more data archived than your local disk can hold. In such a case we hydrate as far as we can.

What will happen on roll-ups?

Normally stubbed files will remain in the selection – this should ensure they are not flushed out with a roll-up. If not, they will automatically get rehydrated in which case they can be rolled up. As a fail-safe we have a special list that we maintain on the Platform, which includes all the archived files, and whenever we do roll-ups we will not delete those files.  

Use cases

Does this work with file servers? Will machines that are accessing remotely trigger the rehydration process?

Yes, it works with file servers. In fact, that will probably be the most common use case. File servers are a brilliant candidate for Archiving, especially since many of those files are not being used. So if you run the Redstor Pro software on the file server itself, even UNC access will trigger the rehydration process.

Can I archive files on a UNC share on a separate server?

No, the filter driver needs to be installed on the host that is serving the files, not the Agent accessing them.

What happens if stub files are moved outside of the backup selection to somewhere else on disk?

They will be detected as moved and will get rehydrated on the next archive run – that is why the full system is scanned.

What happens if stub files are moved outside of the backup selection to a UNC path?

The filter driver will detect that the file is moving from local storage and will rehydrate the file to the target location. The source stub will then be removed.

How does Archiving work with local copy backups?

Local copy backups are not supported with Archiving. Although it is possible to enable both features, you may currently end up with archived files missing in the local copy due to stubbing.

If an absolute must, then the lower risk configuration is to only enable Archiving after the local copy functionality has been running for a few backups. But even in this configuration there is still a risk. It is also not possible to rehydrate from the local copy, as Archive data is not stored in local copy backups.

Are there any limitations around connectivity?

Rehydration is not possible when both the SS and the MS are inaccessible. This is important to keep in mind for laptop users who may not always be connected to the internet.

Can we override the date and stub files immediately when their backup has been mirrored?

Only if the last accessed and modified dates meet the archival criteria. Third-party applications such as Total Commander can set these.

If an application is not working properly for a specific file after stubbing, can we exclude the file from Archiving to prevent it from being stubbed again and immediately rehydrate it?

Currently you can only exclude folders, but if a file is regularly accessed it will not get stubbed in the first place since it will have been accessed recently.

What is the troubleshooting methodology for why files have not been stubbed?

If files that meet the necessary conditions are not being stubbed:

1. Check in your file explorer that both the Last Accessed and Last Modified dates for these files are older than the threshold.
   
   
   
2. Ensure the file is backed up and mirrored (after making any date changes – the timestamps on disk must match the latest backup).
3. Check the Archive logs in C:\ProgramData\Redstor Pro\Backup Client ESE\logs. The reason for any stubbing or rehydration failures should be shown in brackets.

Are Read-Only files stubbable?

Yes.

Does rehydration have a resume support in a power-failure scenario?

Yes. It will resume where it has left off. 

Will the rehydration run from the mirror if the storage server is down?

Yes, and this happens completely transparently.

If a storage server is lost, will my data be automatically hydrated?

No, but you will not be able to archive any additional data until storage redundancy is restored.

Why don’t partially downloaded files get re-archived at the next backup?

Because their last access time is current after a read operation. When the last accessed time meets archival criteria, they will be re-archived.

Can I see how much data is partially rehydrated?

Yes, you can do this by looking at the file properties (see below).

Is the filter driver only active during the archive task, and is it doing anything when archiving is not happening?

The filter driver is always running so it always gets notified when reads and writes come through. We went to a lot of effort, though, to ensure that when you’re not reading or writing stub files there is almost zero overhead time. This process is tested automatically when you submit the driver for verification to Microsoft to ensure that the driver will not slow down the machine.

Can system files be set for exclusion by default to prevent system crashes?

You can select any folder for exclusion that you want. For now your Windows folder is excluded by default and we automatically exclude all files seen by VSS as system state files. However, that is no guarantee so we recommend that you exclude any folder accessed during boot.

How can the software handle existing archive solutions and their stub files?

We have a setting whereby you can exclude third-party stubs from the backup (and therefore archiving).

Is the stubbing and rehydrating of EFS encrypted files possible?

No, these are not supported for stubbing and will be skipped.

Reporting

Can I extract a report from the Redstor Agent that will provide detail on the status of my archives and related information?

Yes, you will find this information on the Archiving logs (default location C:\ProgramData\Redstor Pro\Backup Client ESE\logs). This displays which files have been stubbed, which have been rehydrated and also shows results in a summary.

Archiving information is also available in the Storage Platform Console. Here you can see Archived Data, Archived Files, and Last Archive date.

Note: If these columns are not currently visible, you can enable them by going to View > Customise Columns.

You can also schedule (or generate on demand) an Archiving report for your whole database, or for a specific group or account, by opening the Console in the Reports view.

Is it possible to see in the Storage Platform Console which customers have enabled Archiving?

Yes, the Console can be used to see which collections and groups are enabled for Archiving. You will see the Archiving fields being populated but it is only possible for ESE to see which customers are using Archiving once the first archive is performed.

If I archive a file, will there be a reduction in the volume of total data selected for backup,  and does this free up space for additional files to be backed up?

Your total data selected for backup will remain the same, regardless of how much of it is archived.

For example, if you have a 3.5 GB file, your data selected will be 3.5 GB. If you archive that file so that it only takes up 10KB on disk, the data protected is still 3.5 GB because that is what is stored on the Storage Platform and associated with the ESE account.

We have added extra columns in the console so you can see how much has been archived. 

How will Archiving affect my storage usage per account on the platform? For example, is data deduplicated between my backups and the archives?

Your data selection remains exactly the same, and the data protected stays unchanged too. You are basically specifying that a portion of your backed-up data is also archived, but you are still storing the same amount of data on the platform which is associated with the ESE account.

Do you have a data insight tool that can work out how much will be archived and how much space will be saved?

There are two ways to do this:

1. In the Pro software there is a button that allows you to calculate your savings.

2. We have made it possible to scan your entire system so you can see how much you can save.  A slider is used to set the Archiving threshold to however many days you want. This adjusts the graphs accordingly revealing what effect Archiving will have on a particular server. 

Deactivation

I’ve tried Archiving, but I don’t like it and want to remove it. How do I do this?

If you want to keep all your data, the simplest way is to rehydrate all your data and disable the feature.

1. Click the “Archive” button to open the Start Archiving menu, then click on “Show Advanced Options”.
2. Click on “Rehydrate all stubs” and finally click on “Start Rehydration”.
3. When the process has completed, run the rehydration again (see below for note about archive stats).
4. Finally, in the Archiving options, untick the checkbox marked “Stub backed up files not accessed in the last” to disable the feature.
   

What happens if my ESE Agent is uninstalled?

Any further stubbing will stop. All existing stubs will remain stubbed until they are accessed.  Files will only get rehydrated when they are completely read from disk. Even when being written to disk, they will remain partially rehydrated unless fully overwritten.

Do you get a warning if you uninstall the Redstor software before rehydrating?

You can uninstall Redstor Pro, but it will leave the filter driver and rehydration service on the server – there is no uninstall option for that, other than command line. At present, it requires a manual step where you must rehydrate all files first.

Can you reverse the process?

Yes, install the ESE Agent again and reconnect to the account. The rehydration service will be set up again and Archiving can resume.

What happens if the account has been deleted?

It will no longer be possible to rehydrate stubs or recover files when an account that has been using Archiving has been deleted.

To guard against this, it is impossible to use the Console to delete an account if it has any stub files still archived. 

However, there are other ways to delete an account.

There will be times when an account needs to be automatically deleted from an evaluation group after a certain time. For that reason you cannot enable Archiving on an evaluation account. But it is still possible to delete an account that has been archiving. This can be done by downgrading the account to evaluation in a group which has auto-deletion settings enabled. Please note that all stubs must be rehydrated before an account can be deleted.

What if I just disable the account instead?

Rehydration is not possible if an account or group is disabled. If this happens, users will not be able to access their archived data. Be mindful when Archiving.