In part 1 of this series, we reviewed Cassandra node-based architecture and described how to perform backups and recoveries with Cassandra’s inherent mechanism. Given the importance of database backup, it’s good to know that hosting your database on AWS makes database backup much easier. In this post, we will set up a Cassandra 2.0 multi-node cluster on AWS EC2, then show you how to achieve consistent backup with N2WS. If you are planning on using a multi-node cluster, Cassandra provides an AMI for easy setup, but it only spans a single region. If you need to set up a multi-region cluster, you have to set up Cassandra manually. For the sake of this article, we are using Cassandra on a multi-node cluster and have inserted data by creating a keyspace, schema and table. If you have a production database, it’s imperative to maintain consistent database backups for optimal high availability and data recovery.
Cassandra Backup with N2WS
When a database is hosted on AWS EBS, it has the option of using EBS Snapshots to perform crash-consistent database backups. AWS snapshots offer a one-click backup and restore option. Crash-consistent backup means that a backup will be performed on the data that is written on the EBS volume. However, if any data is in the memory/cache due to the application’s specific configuration, it will not be included in the EBS snapshot. The best possible solution for data backup and recovery is to take application consistent snapshots at regular intervals.
Since AWS EBS snapshots provide incremental storage, it is best to run automated backups, both at frequent time intervals and on a regular basis. It is erroneous to take Amazon snapshots manually at regular intervals because they will always include more manual effort and may run the risk of missing critical backups. It is important to note that Cassandra does not recommend using EBS volumes for its clusters due to low disk and IO performance as well as EBS size limitations. However, this recommendation was written before major improvements were made to EBS. The C4 Instance family now offers better network performance, and EBS optimized instances and Provisioned IOPS provide better IO performance.
Additionally, high performing General Purpose volumes and EBS’ 16TB disk solve the sizing issue. N2WS Backup & Recovery comes in very handy for system administrators by offering a very simple and user-friendly web interface to manage automated EC2 and RDS backups. Additionally, N2WS provides a quick disaster recovery solution as well as the option to achieve application consistent backup, helping to achieve better data backup management and recovery. N2WS offers flexible backup policies that you create based on your specific requirements and allows you to perform backups at regular intervals.
N2WS also provides you with the option to copy EBS snapshots to other regions. You can restore complete backups with a single click. To achieve application consistent backup, it is recommended to run backup scripts first to inform the OS that an AWS EBS snapshot is being taken, then perform the backup. To achieve this, you should freeze your database temporarily by flushing all of your data to disks. As mentioned above, N2WS offers application consistent backup both for Windows and Linux. In Windows, it allows you to perform a VSS backup that is the same as the Windows VSS feature, whereas with Linux instances, it offers the option to run scripts on database instances before snapshots are taken.
An advantage of N2WS is that it offers scripts that can run before, as well as on completion of an EBS snapshot. We will now configure the Cassandra database that we set up on EC2 with N2WS for application consistent backups. Refer to our earlier guides about setting up a backup and restore schedule and policy with N2WS.
As mentioned in How-To Backup Your AWS Cloud Based PostgreSQL Database, N2WS can execute three different scripts: “Before”, “After” and “Complete”. It is mandatory to create each script, and if no command needs to be executed, leave it blank. For application consistent backups, it is recommended to flush all data from memory to disk and then freeze data writes during snapshots.
It is important to note that Cassandra flushes nodes before taking a snapshot, then takes the snapshot and stores the data in the snapshot directory for each keyspace data directory. We will use Cassandra’s built-in snapshot mechanism as part of N2WS’ ‘before’ script to take application consistent EBS snapshots using scripts. Since we have set up Cassandra with three nodes, Cassandra’s built-in snapshot (see part 1 of this post) should be executed simultaneously on all three nodes. We have configured a script in Node-1 that will run the push command and perform Cassandra’s inherent backup on all three nodes.
Cassandra Keyspace Backup Script(db_start_backup.sh)
#!/bin/bash echo "Taking snapshot..." 1>&2 pssh -i -h /home/ec2-user/hosts -x "-oStrictHostKeyChecking=no -i /home/ec2-user/n2ws.pem" 'hostname;nodetool snapshot;echo ------' 1>&2 echo "Snapshot completed" 1>&2N2WS Script
N2WS runs a ‘before’ script before it starts performing backups.
N2WS executes the following scripts:
“Before” script ->(before_<policyname>.sh)
#!/bin/bash ssh -i <location of pem file to SSH> <user-name>@<IP Address of PostgreSQL machine> "db_start_backup.sh" “After” script ->(after_<policyname>.sh)
#!/bin/bash“Complete” script ->(complete_<policyname>.sh)
#!/bin/bash Since we don’t need to run any ‘after’ or ‘complete’ commands, we have kept them blank. You can always run commands (e.g., deleting Cassandra’s backup from a local machine or copying data from one volume to another) in the ‘after’ or ‘complete’ scripts. If the backup is successful, it will show the output as follows: When you open the success log, you should see: A file will be created in Cassandra’s backup folder: ‘/var/lib/cassandra/demo1/users/snapshot’
If you have configured to capture the log output of your script, you can find the output that was sent to ‘std.err’ in the N2WS recovery panel:
Once you are sure that your snapshots are running successfully and your data is consistent with the application consistent snapshots, you can write scripts to remove the backup files and execute those scripts from N2WS’ ‘complete’ scripts. If you refer to Cassandra documentation, it says that a Cassandra snapshot first flushes all in-memory writes to disk, then makes a hard link of the SSTable files for each keyspace. This helps in achieving application consistent snapshots but also requires that you have enough free disk space on the node to accommodate new Cassandra snapshots of your data files.
A single snapshot requires little disk space. However, snapshots can cause your disk usage to grow quickly over time because a snapshot prevents old obsolete files from being deleted. It’s important to note that like AWS snapshots, Cassandra snapshots are also incremental. Therefore, if you delete the backup files, a full backup may be performed again at the same time that the inherent Cassandra backup occupies space. In this article, we have demonstrated how running scripts from our N2WS can help you achieve application consistent backup with Cassandra.
N2WS is an enterprise-class backup solution for EC2 based on EBS and RDS snapshots. If you are hosting production class databases such as MS-SQL, MySQL, MongoDB, or Oracle, N2WS can provide you with a best-in-class automated backup and recovery solution. N2WS supports consistent application backups on Linux as well as Windows servers. N2WS is sold in the AWS Marketplace. See pricing or try it for free.
