Werner Vogels: “Everything fails all the time”
As Amazon’s chief technology officer and vice president once famously said, “Everything fails all the time”. Bugs happen and customers get frustrated. Errors occur and loss of revenue results. Ransomware or weather, although less common, can shut down companies and even entire cities. In this blog, we’ll focus on how to properly back up your AWS Redshift cluster so any failure or data loss won’t have an impact. Being prepared will let you sleep at night knowing the impact of your Redshift cluster failure is minimal with the proper backup solution.
This is part of a series of articles on AWS backup.
What is this Amazon Redshift, per se?
Redshift, the AWS petabyte data warehouse solution, is designed to offer fast query performance with the use of columnar storage technology and is available to use over a wide range of SQL clients. There are two Redshift snapshot types: automated and manual, and both are stored in Amazon S3. In this article, I will elaborate more on the need to use and automate Redshift backups using manual snapshots.
Related post: AWS backup vs snapshot
1. Manual snapshots are crucial for longer retention
Redshift’s automated backups are held for a maximum retention period of up to 35 days, then are automatically deleted. Manual snapshots, on the other hand, can be saved for as long as needed, are not automatically deleted and can be taken at any desired point in time. Manual snapshots can be crucial when it comes to complying with strict regulation rules that generally require data to be saved for a number of years. Additionally, manual snapshots can be particularly useful if you load a large amount of data into your Redshift cluster, for example, and want to immediately back it up.
Furthermore, AWS limits the amount of manual snapshots you can take to 20 snapshots per account. If you have multiple databases, you might want to raise your snapshot limit. Contact AWS or carefully refine your backup policy to take snapshots at larger intervals.
2. Surviving an Altered or Terminated Cluster
According to this Stack Overflow discussion, automated Redshift backups are taken every eight hours, or every 5GB of inserted data, whichever happens first. If a database is altered, there is a good chance you won’t be able to restore it to its most recent changes. In which case, you should make sure that a manual snapshot has been taken. Redshift deletes automated snapshots when you delete the cluster. This means that your Redshift database cannot be restored because the snapshot is lost. At the moment of deletion, you are prompted to capture a manual snapshot, although the significance of this can sometimes be lost and this stage bypassed, meaning there will be no database backup.
Manual Redshift snapshots on the other hand, are stored separately, and if used, are the only way to save a deleted cluster. In both cases of automating your manual snapshots, you will be able to set your backup policy including high-frequency snapshots. Aside from controlling the policy, automating manual snapshots creates seamless and transparent backups, which increase confidence in Redshift operations, and allows you to perform restorations at any point in time.
3. Cross-Region Backup
In the event of an outage, AWS cross region disaster recovery plays a key role in making sure backups are safe and secure in case a region goes down.
Your retention periods set for automated snapshots in AWS destination regions might be different than those within the cluster’s source region. Each AWS region has its own specific retention period for automated snapshots. In addition, cross-region snapshots are limited to a retention period of seven days. If you wish to keep and maintain a comprehensive backup of your Redshift cluster (specifically for mission critical production environments) in another region for high availability purposes, manual snapshots are the better option.
N2WS Backup & Disaster Recovery offers a fail-safe solution that automatically configures the N2WS policy to schedule an automated backup however frequently you choose. It automatically copies the snapshot into a separate region as well as performs additional operations such as cross-account backup, copy to S3, and resource control which allows you to schedule the start/stop of your resources for cost savings.
- Combine S3 storage classes for snapshot archiving: For long-term manual snapshots, consider copying them to S3 and use storage classes like S3 Glacier or Glacier Deep Archive. This can significantly reduce costs while maintaining the ability to restore historical data.
- Optimize Redshift clusters before snapshots to save storage: Schedule VACUUM and ANALYZE operations before taking manual snapshots to minimize unused storage in Redshift clusters. This not only reduces snapshot size but can also improve cluster performance.
- Implement cross-account snapshot copying for added security: Instead of just relying on cross-region storage, consider copying snapshots to a separate AWS account. This approach adds an extra security layer.
- Monitor snapshot usage with CloudWatch metrics: Set up CloudWatch metrics to track snapshot storage size and usage. Alerts based on snapshot volume or cost thresholds help you adjust snapshot policies proactively, optimizing for both performance and budget.
- Regularly test snapshot recovery to validate integrity: Periodically restore a manual snapshot to a test cluster to validate the recovery process and ensure data integrity. This practice can reveal issues early and provide confidence in your recovery strategy.
Automated snapshots will not be enough
AWS automated snapshots offer great capabilities that might suit your needs. However, the catch is that you never know when the next recovery will happen, which is where manual snapshots come into play. Maintaining a comprehensive backup of your production environment and running a compliant environment can and needs to only be done by automating manual snapshots. They allow the flexibility and control that you need in order to support your specific custom backup policy and create robust and reliable cloud operations.
How to Fully Automate Your Redshift Manual Snapshots with no surprises
Redshift snapshots can be taken and managed through the AWS console or APIs. In order to define your backup policy and to automate your Redshift snapshots accordingly, you can use an AWS backup solution such as N2WS Backup & Recovery. N2WS Backup & Recovery is a native cloud backup and recovery solution for any Amazon EC2 instance, EBS volume, RDS database, Redshift Cluster, Amazon Aurora, Amazon DynamoDB or Amazon EFS.
Once you complete the initial setup, there is no need to worry about any failures as if anything goes wrong for any reason, N2WS will send alerts and daily summaries. Also AWS SNS can be integrated to send you a notification so you can troubleshoot the error.
Try N2WS Backup & Recovery for FREE
I salute you for making it this far! Here’s hoping your AWS Disaster Recovery goals are made a bit easier with this information.
