When running infrastructures on the cloud, you need to ensure that they are properly backed up, in case they get accidentally deleted or there is a hardware degradation on the side of the cloud provider. (This can cause an instance to go into a stopped state, resulting in incorrect data and unavailability for further proper operations.) If you are running your infrastructure on Amazon, the best way to properly back up your EC2 data is by snapshotting its EBS volume.
In July 2018, Amazon released a service called Amazon Data Lifecycle Manager (Amazon DLM) for easier automation, retention, and deletion of EBS volumes. In this article, we will cover the features and uses of Amazon DLM, as well as its pros and cons and some third-party solutions for backing up your Amazon infrastructure.
What is Amazon Data Lifecycle Manager?
Amazon DLM is a tool that provides an automated way to back up, retain, and delete your EBS volumes. Prior to Amazon DLM, we had to rely on custom scripts running from AWS Lambda or an EC2 instance in order to back up EBS volumes. Also, these scripts had to take care of retention and deletion of snapshots.
Amazon DLM relies on EBS snapshots, which are incremental backups, so it contains only data that has been added or modified since the previous snapshot. This way, when a snapshot is deleted, only the data unique to that snapshot is deleted. Also, incremental backups based on snapshots are good for cost reduction.
In order for Amazon DLM to perform a backup on an EBS volume, you have to tag it; and then, through the Amazon DLM policy, instruct it when to perform the backup. Amazon DLM is allowed to add multiple tags on a single EBS volume, in case you want to run multiple policies on it.
An Amazon DLM lifecycle policy controls which resources you back up and when. It consists of three core settings:
- Resource type: Describes the type of resource that will be backed up. At the moment, the only supported value is “VOLUME.”
- Target tag: Key-value pair used to identify the volume you need to back up.
- Schedule: Specifies the interval in which the volume will be backed up.
A lifecycle policy is a JSON-formatted configuration file. Here is an example of a policy:
{
“ResourceTypes”: [
“VOLUME”
],
“TargetTags”: [
{
“Key”: “SnapshotVolume”,
“Value”: “true”
}
],
“Schedules”:[
{
“Name”: “DailySnapshots”,
“TagsToAdd”: [
{
“Key”: “type”,
“Value”: “DailySnapshot”
}
],
“CreateRule”: {
“Interval”: 24,
“IntervalUnit”: “HOURS”,
“Times”: [
“05:00”
]
},
“RetainRule”: {
“Count”:7
},
“CopyTags”: false
}
]
}
In this example, we targeted all volumes that have tag key “SnapshotVolume” and tag value “true.” The policy will run every day at 5 a.m., retain seven copies, and add tags with key “type” and value “DailySnapshot” to the newly created snapshot. To create an Amazon DLM lifecycle policy, run the following:
aws dlm create-lifecycle-policy --description “5AM backup” --state ENABLED --execution-role-arn arn:aws:iam:12345678910:role/aws-dlm-role --policy-details file://path_to_json
When working with Amazon DLM, you should consider the following limits:
- The number of lifecycle policies cannot be more than 100 per region.
- One lifecycle policy can have a single schedule time.
- Up to 50 tags are allowed per resource.
The Pros and Cons of Amazon Data Lifecycle Manager
Before getting started with Amazon DLM, you should consider its pros and cons.
Pros
- Easy configuration: To deploy an Amazon DLM policy, all you need is a JSON file which contains the three core settings mentioned earlier (resource type, target tag, and schedule) and an appropriate IAM role which allows Amazon DLM to create and delete snapshots.
- Incremental backup: By creating snapshots of EBS volumes, only data that has been changed is backed up.
- Monitoring: Using Amazon CloudWatch, you can create a rule which will monitor snapshot creation and the Amazon DLM policy state.
Cons
- A configuration change ignores previously created snapshots: If tags for targeting volumes are changed, all previously created volumes will no longer be affected by the policy. The same idea applies to the schedule name: if it is changed, all previously affected snapshots will not be affected by the new policy. This will result in old volumes and snapshots being left unmanaged, and can cause additional costs.
- Inaccurate backup time: The specified time for creating a snapshot is relative. Amazon DLM will create a snapshot within an hour of the time that is specified, meaning that if the “Times” property is set to “5:00,” snapshot creation can start as late as “5:59.”
- Missing cross-zone availability: As EBS volumes are available only in the Availability Zone in which they are created, you will have to automate copying volumes and snapshots to another Availability Zone or Region using custom-made scripts in order to perform disaster recovery (DR).
- The possibility of hitting limits: To perform multiple backups on a single EBS volume, you have to create multiple Amazon DLM policies. This increases the complexity of the setup, and can cause you to hit the limit of maximum available Amazon DLM lifecycle policies per region.
Introducing N2WS Backup & Recovery
N2WS offers a backup and recovery tool for your Amazon infrastructure called N2WS Backup & Recovery. The key features of this product are:
- Automated backup of Amazon Aurora, EC2 instances, Amazon EBS, Amazon RDS databases, and Amazon Redshift clusters.
- DR across multiple regions.
- File-level copy of your critical data.
- One-click recovery that allows you to restore single files or entire environments in less than 30 seconds.
- An easy-to-use web interface with monitoring, dashboards, alerting, and integration with other services.
N2WS Backup & Recovery stores all backups on Amazon S3, making it highly available and reachable from other regions or even other accounts.
Using the N2WS web interface, you can create schedules and policies for your Amazon resources. N2WS Backup & Recovery schedules describe the times when backups will be performed, while policies describe which schedules will be applied and the retention period for keeping the data.
When a schedule is created, you can apply a policy to start backing up your resources. Also, within policy options, you can choose whether this policy should create a DR. All you have to do is click on the DR option, enable DR, choose another region (or multiple regions) where backups will be stored, and apply the changes. You can also choose to create a cross-account DR.
Once you have created a policy and schedule, pick the resources that will be backed up. In the web interface, under your policy, choose “Backup targets” and then choose a resource. You can choose between EC2 instances, Amazon Aurora clusters, EBS volumes, and Amazon Redshift clusters.
Restoring an instance, volume, or even file is a really easy operation to perform. Simply choose your backup from the “Backup Monitor” tab and click “Recover.” From there, you will be able to restore your data to the same account, in a different region, or in a different account, depending on your policy setup.
To get started with N2WS Backup & Recovery, log in to your AWS account, and launch an EC2 instance with the N2WS Backup & Recovery 30 day trial AMI. You can find instructions for setting up N2WS Backup & Recovery here.
Summary
Amazon DLM is a fairly new service. However, it provides basic functionalities which allow you to avoid writing custom scripts for backing up Amazon EBS volumes. It’s still missing some key features, like DR and restoring from backup on the file level, but new features will likely be added to Amazon DLM in the future.
Third-party solutions like N2WS Backup & Recovery provide features that compensate for those missing from Amazon DLM. N2WS Backup & Recovery covers a wide range of resources that can be backed up, and with its easy-to-use web interface, you can create or restore a backup in just a few clicks.
