19. Using Simple Storage Service (S3) with N2WS

Learn how to use N2WS Backup & Recovery to store backup snapshots in an S3 repository, lowering storage costs by up to 40%.

Contents:

19 Using Simple Storage Service (S3) with N2WS

19.1 Limitations

19.2 Cost Considerations

19.3 Overview of S3 and N2WS

19.4 Configuring S3 Workers

19.5 Configuring an S3 Repository

19.6 Configuring a Policy to Copy to S3

19.7 Managing Copy to S3 Backups

19.8 Recovering an S3 Backup

Using Simple Storage Service (S3) with N2WS

CPM can back up your EBS snapshot data to Amazon Web Services (AWS) S3 buckets. Using the CPM Copy to S3 feature, you can:

  • Define multiple folders, known as repositories, within a single S3 bucket
  • Define the frequency with which CPM backups are made to a Repository in S3, similar to DR backup. For example, copy every third generation of a CPM backup to S3.
  • Define backup retention based on time and/or number of generations per Policy.
  • Enable client-side data encryption per Repository independent of server-side encryption implemented by AWS at the repository level.
  • Lower your backup costs. For example, customers who keep weekly or monthly backups for a year may benefit from reduced costs by moving these backups from EBS snapshots to a CPM S3 Repository.

CPM keeps backups in S3 in the Veeam VBR repository format – the data is stored as block-level incremental backups. CPM can restore these backups to AWS, but customers can also use VBR capabilities to restore backups from the S3 repository to on-premises or public clouds.

Important: AWS Encryption at the bucket-level must be enabled.

Strongly Recommended:

  • S3 buckets used by Copy to S3 should not be used by other applications.
  • Versioning at the bucket level should be disabled.

Notes: Before continuing, consider the following:

  • Copy to S3 currently supports only backups of Windows and Linux instances. RDS, DynamoDB, etc. are not supported.
  • Independent volumes will be supported in a future release.

Note: Most CPM operations related to the S3 repository (e.g. converting EBS snapshots to the Veeam format, writing objects to S3, clean up, restoring, etc.) are performed by launching CPM worker instances in AWS. The worker instances are terminated when their tasks are completed.

Limitations

  • Only copy of instance backups is supported.
  • Instances with volumes totaling more than 8 TB cannot be copied.
  • Copy of standalone volumes is not supported.
  • Copy is not supported for other AWS resources that CPM supports, such as RDS and Aurora.
  • Snapshots consisting of ‘AMI-only’ cannot be copied to a S3 repository.
  • The root volume of instances purchased from Amazon Marketplace, such as instances with product code, cannot be copied to S3. The data volumes of such instances, if they exist, will be copied.
  • Backup records that were copied to S3 cannot be moved to Freezer.
  • User cannot delete specific snapshots from S3 repository. S3 snapshots are deleted according to retention policy. In addition, users can delete all S3 snapshots of a specific policy, account or an entire repository. See below.
  • A separate CPM server, for example, one with a different “CPM Cloud Protection Manager Data” volume, cannot reconnect to an existing S3 repository.
  • In order to use the Copy to S3 functionality the “cpmdata” policy must be enabled. See CPM User Guide for details on enabling the “cpmdata” policy.
  • For every policy that enables ‘Copy to S3’, all instances that are backed up by the policy need to be in the same region.
  • Only a single S3 operation is possible on a policy at any given time. Additional executions of Copy to S3 backups will not occur if the previous execution is still running. Restore from S3 is always possible.
  • AWS accounts have a default limit to the number of instances that can be launched. Copy to S3 launches extra instances as part of its operation and may fail is the AWS quota is reached. See CPM User Guide for details.
  • Copy and Restore of volumes to/from regions different from where the S3 bucket resides may incur long delays and additional bandwidth charges.
  • Instance names may not contain slashes (/) or backslashes (\) or the copy will fail.

Cost Considerations

N2W Software has the following recommendations to CPM customers for help lowering transfer and storage costs:

Lowering transfer fees:

  • When a ‘CPMWorker’ instance is using a public IP (or NAT/IGW within a VPC) to access an S3 bucket within the same region/account, it results in network transfer fees.
  • Using a VPC endpoint instead will enable instances to use their private IP to communicate with resources of other services within the AWS network, such as S3, without the cost of network transfer fees.
  • For further information on how to configure CPM with a VPC endpoint, see Appendix A – Recommended Configuration for Copy to S3.

Lowering storage fees:

  • Configuring your policies to copy to S3 less frequently, and for long durations, can lower your storage fees up to 40% compared to EC2 backup fees.

Overview of S3 and N2WS

The Copy to S3 feature is similar in many ways to the CPM Disaster Recovery (DR) feature. When Copy to S3 is enabled for a policy, copying EBS snapshot data to S3 begins at the completion of the EBS backup, similar to the way DR works. Copy to S3 can be used simultaneously with DR feature.

Workflow for Using S3 with N2WS CPM

  1. Define an S3 Repository – Click the S3 Repositories button and then Create New S3 Repository.
  2. Define a Policy with a Schedule, as usual. Then configure the policy to include Copy to S3 by selecting Copy to S3 in the Configure column and completing the form.
  3. If you are going to backup and restore S3 instances and volumes across accounts and regions, you can prepare a Worker Configuration using the Configure workers for S3 operations link.
  4. Use the Backup Monitor and Recovery Monitor, with some additional controls, to manage S3 snapshots as usual.

Configuring S3 Workers

When CPM copies data to or restores data from an S3 repository, it launches a temporary ‘worker’ instance to perform the actual work, such as writing objects into S3.

  • When performing backup operations, the ‘worker’ instance is launched in the region and account of the target instance. The backup ‘worker’ instance is configured using the Configure workers for S3 operations link in the bottom toolbar.
  • When performing restore operations, the ‘worker’ instance is launched in the region and account that the backed-up instances are to be restored to. The restore ‘worker’ instance is selected or configured according to the following criteria:

If a ‘worker’ for the target account/region combination was configured in the Configure workers for S3 operations page, that ‘worker’ instance will be used during the restore.

If such a ‘worker’ does not exist for the target account/region combination, CPM will attempt to assemble one based on CPM’s own configuration.

If CPM’s configuration cannot be used because the restore will be to a different account or region than CPM’s, the user will be prompted during the restore to configure the ‘worker’.

Note: If you plan to Copy to S3 only instances belonging to the same account and residing in the same region as that of the CPM server, worker configuration Is not required since the worker will derive its configuration from the CPM server instance.

Attempts to Copy to S3 instances and volumes from an account/region without a valid worker configuration will fail.

Worker Parameters

It is necessary to define a separate worker configuration for each planned account/region combination of Copy to S3 instance snapshots:

To configure S3 worker parameters:

  1. Click the Configure workers for S3 operations link in the bottom toolbar of the CPM GUI.
  2. Click New Worker configuration.

In the Account list, select the Account that the new worker is associated with.

In the Region list, select a Region. This configuration will be applied to all workers launched in this region for this account.

In the Key pair list, select a key pair. Using the default, Don’t use a key pair, disables SSH connections to this worker.

In the VPC list, select a VPC. The selected VPC must be able to access the subnet where CPM is running as well as the S3 endpoint.

In the Security Group list, select a security group. The selected security group must allow outgoing connections to the CPM server and to the S3 endpoint.

In the Subnet list, select a subnet, or choose Any to have CPM choose a random subnet from the selected VPC.

Note: If you choose ‘Any’ in the Subnet drop-down list, CPM will automatically choose a subnet that is in the same Availability Zone as the one you are restoring to. If you choose a specific subnet that is not in the same Availability Zone as the one you are restoring to, you will have to choose a different subnet from the Subnet drop-down list.

In the Network access list, select a network access method.

Note: Direct network access or indirect access via an HTTP proxy is required:

Direct – Select a Direct connection if no HTTP proxy is required.

via HTTP proxy – If an HTTP proxy is required, select and fill in the proxy values.

To edit or delete a worker configuration:

  1. In the bottom toolbar, click the Configure workers for S3 operations link.
  2. In the Action column for the worker, click Delete or Edit.

Configuring an S3 Repository

There can be multiple repositories in a single AWS S3 bucket.

  1. In CPM, click the S3 Repositories button.

Click Create New S3 Repository.

C:\Users\Janet\AppData\Local\Temp\ATT27202 3.jpg

In the Create S3 Repository screen, complete the following information:

Repository Name – Type the name of the new repository folder in the AWS S3 bucket.

Only alphanumeric characters and the underscore are allowed.

Repository Name must be unique to the bucket.

Description – Optional brief description of contents of repository.

Account – Select the account that has access to the S3 bucket.

Aws region – Select the region in which the S3 bucket is located.

Aws bucket name – Type the name of the S3 bucket that exists in this region.

Note: AWS encryption must have been enabled for the bucket.

Enable Encryption – Select Enabled to use additional client-side encryption support that is independent of the AWS-provided encryption at the bucket level. If enabled, enter a Password and Password Hint to be used for encryption key generation.

When complete, click Create.

Configuring a Policy to Copy to S3

Configuring a Policy for Copy to S3 backups includes definitions for the following:

  • Name of the S3 Repository defined in CPM.
  • Interval of AWS snapshots to copy.
  • Snapshot retention policy.

It is possible to retain a backup based on both time and number of generations copied. If both Time Retention and Generation Retention are enabled, both constraints must be met before old snapshots are deleted.

For example, when the automatic cleanup runs:

  • If Time Retention is enabled for 7 days and Generation Retention is disabled, S3 snapshots older than 7 days are deleted.

If run ASAP is executed 10 times in one day, none of the snapshots would be deleted until they are more than 7 days old.

  • If Generation Retention is enabled for 4 and Time Retention is disabled, the 4 most recent S3 snapshots are saved.
  • If Time Retention is enabled for 7 days and Generation Retention is enabled for 4 generations, a single S3 snapshot would be deleted after 7 days if the number of generations had reached 5.
  1. From the main screen, in the Policies tab, select a Policy and click Copy to S3 in the Configure column.

Complete the following fields:

  1. Enabled copy to S3 – Whether Copy to S3 is enabled. Default is Disabled.
  2. S3 Repository – Select the Repository in the S3 bucket to copy your backup to.
  3. Copy every – Select the interval between snapshots to copy. For example, if Copy every is 3, copy every 3rd CPM backup to S3.
  4. Generation Retention – Whether retention by generation is enabled for this policy. Default is Enabled.
  5. Num Generations – If Generation Retention is enabled, how many S3 generations to save.
  6. Time retention – Whether retention by time is enabled for this policy. Default is Enabled.
  7. Retention duration – If Time Retention is enabled, how long to save the backup: Days/Weeks/Months/Years.
  8. Click Apply.

Changing the S3 Retention Rules for a CPM Policy

You can set a different retention rules in each Policy.

To update the S3 retention rules for a policy:

  • From the S3 Repositories screen, select the target policy in the Related Policies column.
  • Or, from the Policies tab in the main screen, click Copy to S3 in the Configure column for the target policy.

Change the retention-related fields in the Backup copy settings window as described in section 4 and click Apply.

Managing Copy to S3 Backups

After a Policy with a Copy to S3 backup starts, you can follow its progress in the Backup Monitor.

The Copy to S3 portion of a Policy backup occurs after the non-S3 backups have completed.

Aborting an S3 Copy does not stop the non-S3 backup portion of the policy from completing. Only the Copy to S3 portion is stopped.

  1. Select the Backup Monitor.

C:\Users\Janet\AppData\Local\Microsoft\Windows\Temporary Internet Files\Content.MSO\8E0D0DFE.tmp C:\Users\Janet\AppData\Local\Microsoft\Windows\Temporary Internet Files\Content.MSO\594EEABC.tmp

In the S3 Copy Status column, the real-time status of an S3 Copy is shown. For copies in progress, the percentage completed is shown.

To stop an S3 Copy in Progress, click Abort S3 Copy in the Actions column.

To delete only the snapshots copied to a specific S3 repository:

  1. Click the S3 Repositories button.

  1. In the row of the target repository, click Delete in the Actions column.

Note: When deleting Policies and Snapshots in the Policies tab or Account and data in the Accounts tab, S3 copies are also deleted.

Recovering an S3 Backup

You can recover an S3 backup to the same or different regions and accounts.

  1. Select the Backup Monitor tab.
  2. On the row of the backup to recover, click Recover in the Actions column.
  3. In the Restore from drop-down list of the Recovery Panel screen, select the name of the S3 Repository to recover from.

C:\Users\Janet\AppData\Local\Temp\image003.png

The Restore to Region drop-down list opens.

In the Restore to Region drop-down list, select the Region to restore the S3 copy to. The source Region of the S3 copy is displayed in the Region column.

C:\Users\Janet\AppData\Local\Temp\image004.png

If you have multiple CPM accounts defined, you can choose a different target account to recover to.

In the Recover column, choose the recovery resource type: Instance or Volumes Only.

If you selected Instance:

  1. Change the Basic and Advanced Options default values as necessary.

If a worker has not been configured or assembled by CPM, the Worker Configuration section will open below the Advanced Options. Complete the form as necessary for the current recovery.

Note: If you choose ‘Any’ in the Subnet drop-down list, CPM will automatically choose a subnet that is in the same Availability Zone as the one you are restoring to. If you choose a specific subnet that is not in the same Availability Zone as the one you are restoring to, you will have to choose a different subnet from the Subnet drop-down list.

Click the Recover Instance button.

If you selected Volumes Only:

Change the default values as necessary. In the Attach Behavior drop-down list, select the appropriate behavior for the recovery:

  • Attach only if Device is Free
  • Switch Attached Volumes
  • Switch Attached Volumes and Delete Old Ones
  • If a worker has not been configured or assembled by CPM, the Worker Configuration section will open below the Advanced Options. Complete the form as necessary for the current recovery.

Note: If you choose ‘Any’ in the Subnet drop-down list, CPM will automatically choose a subnet that is in the same Availability Zone as the one you are restoring to. If you choose a specific subnet that is not in the same Availability Zone as the one you are restoring to, you will have to choose a different subnet from the Subnet drop-down list.

Click the Recover Volumes button.

The Recovery Monitor opens and shows the Status of the recovery.

To abort a recovery in progress, click Abort in the Actions column.

Share this post →