The previous post, Disk Array Backup on EC2 Part II: Consistency Issues, described the challenges in performing EBS snapshot-based backups for disk arrays on EC2. When performing a backup on an LVM data volume that spans multiple EBS volumes, one needs to make sure the snapshots are taken in a consistent state. AWS APIs do not currently allow snapshots of multiple volumes to be taken at the exact same point in time. LVM allows creating LVM snapshots. LVM snapshots reside in the volume group of the original volume, and stores all the changes made to the original volume since the snapshot’s point in time. LVM snapshots shouldn’t be used as a backup infrastructure because:
- LVM uses a copy-on-write mechanism to copy changes to the snapshot area. This consumes resources and may affect the logical volume’s IO performance.
- The snapshot consumes storage space from primary disks.
- The snapshot relies on primary data, so if there’s data loss, corruption or outage, snapshots will be as useless as the primary data.
#!/bin/bash ssh -i <ssh private key file> <user>@<address of instance> “lvcreate -n snapvolcpm -L 1G -s /dev/mapper/VolGroup00-MyVol00” if [ $? -gt 0 ]; then echo “Failed taking lvm snapshot” 1>&2 exit 1 else echo “lvm snapshot succeeded” 1>&2 fiThis script connects to the instance using SSH and performs a command to create an LVM snapshot with a pre-defined name. This version is a bit simplified; in a production script there may be additional error checking. If the backup policy contains more than one LVM volume group, there may be need to create more than one LVM snapshot. In the “after” script, the LVM snapshots will simply be removed:
#!/bin/bash if [ $1 -eq 0 ]; then echo “There was an issue running the first script” 1>&2 fi ssh -i <ssh private key file> <user>@<address of instance> “lvremove –force /dev/VolGroup00/snapvolcpm” if [ $? -gt 0 ]; then echo “Failed removing lvm snapshot” 1>&2 exit 1 else echo “remove lvm snapshot succeeded” 1>&2 fiAfter configuring these scripts, LVM doesn’t contain any snapshots most of the time, but EBS snapshots will contain LVM snapshots which can be used in case of recovery. When recovering the instance with all EBS volumes, the first thing that needs to be done is to revert to the LVM snapshots. Thus, after logging in to the newly created instance, type:
umount /dev/mapper/VolGroup00-MyVol00 lvconvert –merge /dev/VolGroup00/snapvolcpm mount /dev/mapper/VolGroup00-MyVol00 /volume_mountpointThese commands unmount the volume, merge the snapshot with the main volume, and then re-mount it. With this approach, you are able to perform backup using EBS snapshots, with all their advantages, and still make sure that the backup is consistent in the volume manager level.]]>