fbpx

21. Using Simple Storage Service (S3) with N2WS

Learn how to use N2WS Backup & Recovery to store backup snapshots in an S3 repository, lowering storage costs by up to 60%.

N2WS can back up your EBS snapshot data to Amazon Web Services (AWS) S3 buckets. Using the N2WS Copy to S3 feature, you can:

  • Define multiple folders, known as repositories, within a single S3 bucket
  • Define the frequency with which N2WS backups are made to a Repository in S3, similar to DR backup. For example, copy every third generation of an N2WS backup to S3.
  • Define backup retention based on time and/or number of generations per Policy.
  • Enable client-side data encryption per Repository independent of server-side encryption implemented by AWS at the repository level.
  • Lower your backup costs. For example, customers who keep weekly or monthly backups for a year may benefit from reduced costs by moving these backups from EBS snapshots to an N2WS S3 Repository. However, Copy to S3 is not designed for daily copies and is not supported for backup frequencies of less than 1 week.
  • N2WS keeps backups in S3 in the Veeam VBR repository format – the data is stored as block-level incremental backups. N2WS can restore these backups to AWS, but customers can also use VBR capabilities to restore backups from the S3 repository to on-premises or public clouds.

Note: Only one S3 operation is allowed at a time – Copy, Recovery, or retention Cleanup. For instance, an S3 Copy or S3 Recovery is not allowed when the S3 backup retention Cleanup is executing. If the S3 Cleanup process is running at the time of an S3 Copy or Recovery, you can abort the Cleanup process in order to allow the Copy or Recovery process to continue. See section 21.5.2.

Important: AWS Encryption at the bucket-level must be enabled.

Strongly Recommended:

  • S3 buckets used by Copy to S3 should not be used by other applications.
  • Versioning at the bucket level should be disabled.

Before continuing, consider the following:

  • Copy to S3 currently supports only backups of Windows and Linux instances. RDS, DynamoDB, etc. are not supported.
  • Independent volumes will be supported in a future release.

Note:

Most N2WS operations related to the S3 repository (e.g. converting EBS snapshots to the Veeam format, writing objects to S3, clean up, restoring, etc.) are performed by launching N2WS worker instances in AWS. The worker instances are terminated when their tasks are completed.

Limitations

Only copy of instance backups is supported.

  • Copy to S3 is supported for weekly and monthly backup frequencies only. Daily backup copies to S3 are not supported.
  • Instances with volumes selected for backup totaling more than 8 TB cannot be copied.
  • Copy of standalone volumes is not supported.
  • Copy is not supported for other AWS resources that N2WS supports, such as RDS and Aurora.
  • Snapshots consisting of ‘AMI-only’ cannot be copied to a S3 repository.
  • The root volume of instances purchased from Amazon Marketplace, such as instances with product code, cannot be copied to S3. The data volumes of such instances, if they exist, will be copied.
  • Backup records that were copied to S3 cannot be moved to Freezer.
  • User cannot delete specific snapshots from S3 repository. S3 snapshots are deleted according to retention policy. In addition, users can delete all S3 snapshots of a specific policy, account or an entire repository. See below.
  • A separate N2WS server, for example, one with a different “CPM Cloud Protection Manager Data” volume, cannot reconnect to an existing S3 repository.
  • In order to use the Copy to S3 functionality, the “cpmdata” policy must be enabled. See N2WS User Guide for details on enabling the “cpmdata” policy.
  • For every policy that enables ‘Copy to S3’, all instances that are backed up by the policy need to be in the same region.
  • Only a single S3 operation is possible on a policy at any given time. Additional executions of Copy to S3 backups will not occur if the previous execution is still running. Restore from S3 is always possible, except when Cleanup is running
  • AWS accounts have a default limit to the number of instances that can be launched. Copy to S3 launches extra instances as part of its operation and may fail is the AWS quota is reached. See N2WS User Guide for details.
  • Copy and Restore of volumes to/from regions different from where the S3 bucket resides may incur long delays and additional bandwidth charges.
  • Instance names may not contain slashes (/) or backslashes (\) or the copy will fail.

Cost Considerations

N2W Software has the following recommendations to N2WS customers for help lowering transfer and storage costs:

  • Lowering transfer fees:
    • When an ‘N2WSWorker’ instance is using a public IP (or NAT/IGW within a VPC) to access an S3 bucket within the same region/account, it results in network transfer fees.
    • Using a VPC endpoint instead will enable instances to use their private IP to communicate with resources of other services within the AWS network, such as S3, without the cost of network transfer fees.
    • For further information on how to configure N2WS with a VPC endpoint, see Appendix A – Recommended Configuration for Copy to S3.
  • Lowering storage fees:
    • Configuring your policies to copy to S3 less frequently, and for long durations, can lower your storage fees up to 40% compared to EC2 backup fees.

The following are conditions where it is recommended NOT to copy backup snapshots to S3:

  • S3 backup increments are more frequent than 1 week. The recommended minimum is weekly.
  • S3 retention periods shorter than 3 months. The recommended minimum is 3 months.
  • Data that needs immediate availability. S3 has longer RTO than EBS. Copy to S3 should be considered for archival purposes. For data that may require immediate availability, use regular EBS operations.

Overview of S3 and N2WS

The Copy to S3 feature is similar in many ways to the N2WS Disaster Recovery (DR) feature. When Copy to S3 is enabled for a policy, copying EBS snapshot data to S3 begins at the completion of the EBS backup, similar to the way DR works. Copy to S3 can be used simultaneously with DR feature.

Workflow for Using S3 with N2WS

  1. Define an S3 Repository – Click the S3 Repositories button and then Create New S3 Repository.
  2. Define a Policy with a Schedule, as usual. Then configure the policy to include Copy to S3 by selecting Copy to S3 in the Configure column and completing the form.
  3. If you are going to back up and restore S3 instances and volumes across accounts and regions, you can prepare a Worker Configuration using the Configure workers link.
  4. Use the Backup Monitor and Recovery Monitor, with some additional controls, to manage S3 snapshots as usual.

Configuring an S3 Repository

There can be multiple repositories in a single AWS S3 bucket.

  1. In N2WS, click the S3 Repositories button.
  2. Click Create New S3 Repository.C:\Users\Janet\AppData\Local\Temp\ATT27202 3.jpg
  3. In the Create S3 Repository screen, complete the following information:
    1. Repository Name – Type the name of the new repository folder in the AWS S3 bucket.
      1. Only alphanumeric characters and the underscore are allowed.
      2. Repository Name must be unique to the bucket.
    2. Description – Optional brief description of contents of repository.
    3. Account – Select the account that has access to the S3 bucket.
    4. Aws region – Select the region in which the S3 bucket is located.
    5. Aws bucket name – Type the name of the S3 bucket that exists in this region.
      1. NOTE: AWS encryption must have been enabled for the bucket.
    6. Enable Encryption – Select Enabled to use additional client-side encryption support that is independent of the AWS-provided encryption at the bucket level. If enabled, enter a Password and Password Hint to be used for encryption key generation.
  4. When complete, click Create.

Configuring a Policy to Copy to S3

Configuring a Policy for Copy to S3 backups includes definitions for the following:

  • Name of the S3 Repository defined in N2WS.
  • Interval of AWS snapshots to copy.
  • Snapshot retention policy.

It is possible to retain a backup based on both time and number of generations copied. If both Time Retention and Generation Retention are enabled, both constraints must be met before old snapshots are deleted.

For example, when the automatic cleanup runs:

  • If Time Retention is enabled for 7 days and Generation Retention is disabled, S3 snapshots older than 7 days are deleted.

If run ASAP is executed 10 times in one day, none of the snapshots would be deleted until they are more than 7 days old.

  • If Generation Retention is enabled for 4 and Time Retention is disabled, the 4 most recent S3 snapshots are saved.
  • If Time Retention is enabled for 7 days and Generation Retention is enabled for 4 generations, a single S3 snapshot would be deleted after 7 days if the number of generations had reached 5.
  1. From the main screen, in the Policies tab, select a Policy and click Copy to S3 in the Configure column.
  2. Complete the following fields:
    1. Enabled copy to S3 – Whether Copy to S3 is enabled. Default is Disabled.
    2. S3 Repository – Select the Repository in the S3 bucket to copy your backup to.
    3. Copy every – Select the interval between snapshots to copy. For example, if Copy every is 3, copy every 3rd N2WS backup to S3.
    4. Generation Retention – Whether retention by generation is enabled for this policy. Default is Enabled.
    5. Num Generations – If Generation Retention is enabled, how many S3 generations to save.
    6. Time retention – Whether retention by time is enabled for this policy. Default is Enabled.
    7. Retention duration – If Time Retention is enabled, how long to save the backup: Days/Weeks/Months/Years.
  3. Click Apply.

Changing the S3 Retention Rules for a N2WS Policy

You can set a different retention rule in each Policy.

To update the S3 retention rules for a policy:

  1. From the S3 Repositories screen, select the target policy in the Related Policies column.
  2. Or, from the Policies tab in the main screen, click Copy to S3 in the Configure column for the target policy.
  3. Change the retention-related fields in the Backup copy settings window as described in section 21.5 and click Apply.
Stopping an S3 Cleanup in Progress

If an S3 retention Cleanup is in progress, the Stop S3 Cleanup button will appear for the policy in the Operations column of the Policies tab. If you want to stop the Cleanup, click Stop S3 Cleanup. See the Note in section 21 for the reasons you might want to stop the S3 Cleanup.

  • Stopping S3 Cleanup does not stop the non-S3 cleanup portion of the policy from completing. Only the S3 cleanup portion is stopped.
  • Stopping S3 Cleanup of a policy containing several instances will stop the cleanup process for policy as follows:
    • N2WS will perform the cleanup of the current instance according to its retention policy.
    • N2WS will terminate all S3 Cleanups for the remainder of the instances in the policy.
    • N2WS will set the session status to Aborted.
    • N2WS user will get a ‘S3 Cleanup of your policy aborted by user’ notification by e-mail.

Managing Copy to S3 Backups

After a Policy with a Copy to S3 backup starts, you can follow its progress in the Backup Monitor.

  • The Copy to S3 portion of a Policy backup occurs after the non-S3 backups have completed.
  • Aborting an S3 Copy does not stop the non-S3 backup portion of the policy from completing. Only the Copy to S3 portion is stopped.
  1. Select the Backup Monitor.C:\Users\Janet\AppData\Local\Microsoft\Windows\Temporary Internet Files\Content.MSO\8E0D0DFE.tmp C:\Users\Janet\AppData\Local\Microsoft\Windows\Temporary Internet Files\Content.MSO\594EEABC.tmp
  2. In the S3 Copy Status column, the real-time status of an S3 Copy is shown. For copies in progress, the percentage completed is shown.
  3. To stop an S3 Copy in Progress, click Abort S3 Copy in the Actions column.
  4. To delete only the snapshots copied to a specific S3 repository:
    1. Click the S3 Repositories button.
    2. In the row of the target repository, click Delete in the Actions column.

Note: When deleting Policies and Snapshots in the Policies tab or Account and data in the Accounts tab, S3 copies are also deleted.

Recovering an S3 Backup

You can recover an S3 backup to the same or different regions and accounts.

  1. Select the Backup Monitor tab.
  2. On the row of the backup to recover, click Recover in the Actions column.
  3. In the Restore from drop down list of the Recovery Panel screen, select the name of the S3 Repository to recover from.
  4. C:\Users\Janet\AppData\Local\Temp\image003.pngWhen the Restore to Region drop-down list opens, select the Region to restore the S3 copy to. The source Region of the S3 copy is displayed in the Region column.C:\Users\Janet\AppData\Local\Temp\image004.png
  5. If you have multiple N2WS accounts defined, you can choose a different target account to recover to.
  6. In the Recover column, choose the recovery resource type: Instance or Volumes Only.
  7. If you selected Instance:
    1. Change the Basic and Advanced Options default values as necessary.
    2. If a worker has not been configured or assembled by N2WS, the Worker Configuration section will open below the Advanced Options. Complete the form as necessary for the current recovery.
      1. NOTE: If you choose ‘Any’ in the Subnet drop-down list, N2WS will automatically choose a subnet that is in the same Availability Zone as the one you are restoring to. If you choose a specific subnet that is not in the same Availability Zone as the one you are restoring to, you will have to choose a different subnet from the Subnet drop-down list.

    3. Click the Recover Instance button.
  8. If you selected Volumes Only:
    1. Change the default values as necessary. In the Attach Behavior drop-down list, select the appropriate behavior for the recovery:
      1. Attach only if Device is Free
      2. Switch Attached Volumes
      3. Switch Attached Volumes and Delete Old Ones
    2. If a worker has not been configured or assembled by N2WS, the Worker Configuration section will open below the Advanced Options. Complete the form as necessary for the current recovery.
      1. NOTE: If you choose ‘Any’ in the Subnet drop-down list, N2WS will automatically choose a subnet that is in the same Availability Zone as the one you are restoring to. If you choose a specific subnet that is not in the same Availability Zone as the one you are restoring to, you will have to choose a different subnet from the Subnet drop-down list.
    3. Click the Recover Volumes button.
  9. The Recovery Monitor opens and shows the Status of the recovery.
  10. To abort a recovery in progress, click Abort in the Actions column.

Share this post →

Share on twitter
Share on linkedin
Share on facebook
Share on email