May 11, 2016: A “file integrity issue” at CRM-provider Salesforce led to a day-long outage. Five hours of data were lost—along with a whole lot of customer productivity—as the company scrambled to fix the disruption in service.
Jan 31, 2017: GitLab, the open-source Git repository backed by venture capitalists such as Y Combinator and Khosla Ventures, experienced an epic outage of its online services. All five of GitLab backup techniques failed to kick in and some production data was lost forever.
August 3, 2017: Due to a configuration mishap on the Meraki Cloud, networking giant Cisco Systems accidentally deleted a half day’s worth of uploaded user data. Programmers and engineers never intend on making errors.
But the harsh reality, as illustrated above, is that mistakes that wipe out data and backups can happen. And when they do, the results are often disastrous. In some cases, even the simplest of human errors can delete entire instances or databases. Other times, it’s difficult to understand how a company could have made so many mistakes along the way. Either way, the ramifications can be monumental.
It’s Not Just About the Money
Consider this: The average cost of one hour of downtime for enterprises can exceed $300,000. The massive 2016 outage and subsequent backup and data recovery deployment failure at Southwest Airlines stranded thousands of customers and triggered a loss of $80 million. Moreover, 43 percent of companies that suffer a major data loss never reopen, according to a University of Texas study.
Code Spaces, a former project-management startup, learned this lesson the hard way. After experiencing a loss of customer backups, the company was forced to close its doors—forever. While lost data is gone forever in many cases, it’s not necessarily “game over” if you find that your backups have been lost. In some scenarios, it may be possible to recover data with a whole lot of effort. And now—before a catastrophe occurs—is the time to make sure you know what to do when disaster strikes.
Tips for Recovering Lost Backups
OWN THE MISTAKE
Companies that take public and swift corrective action fare far better than companies that try to cover it up. Provide your users with as much detail as possible to show that you’re fully invested in solving the issue.
PIECE TOGETHER WHAT YOU CAN
Speak to your developers and engineers. Chances are that someone on your team may have at least a portion of the backups stored somewhere, perhaps in their own files or in code repositories. You can also begin to piece together what’s left, with reverse engineering.
CREATE A COMPLETE DISASTER RECOVERY PLAN
Once you have recovered whatever you can, it’s essential to build a disaster recovery plan. Without it, your organization has no way to ensure that during times of failure, all involved parties can work together in a fluid manner to fix the situation with as little fallout as possible. The plan should include:
- An up-to-date contact list to ensure that all critical players can be reached immediately.
- A communication plan to ensure that employees know their role in a situation of data loss.
- A current network map and inventory list to ensure that all areas are accounted for.
Most important, don’t forget to regularly test your DR plan to make sure it actually works—before a disaster happens. As stressful as it may be to simulate an outage, testing is the only way to make sure that your team is prepared for the real thing.
PREVENTING FUTURE LOSS
After a backup loss, the most important take-home lesson is to move forward in a more efficient and intelligent manner. You already know that mistakes can happen, so ensuring that you have a rock-solid backup solution in place is critical. N2WS Backup & Recovery lets you automatically backup your organization’s data as often as you need and allows for lightning-fast recovery when you need it the most.
With block-level and incremental-forever snapshots that can be scheduled as often or as infrequently as you want, and kept for as long as you want, you’ll get complete RPO flexibility. When it comes to restoration and recovery after one of these nightmare scenarios, N2WS has your organization covered. In a matter of seconds, it enables complete recovery for instances, specific volumes, and files. And N2WS makes it easy to backup data to another dedicated account as a backup vault. So, even when backups are gone from the original account, or are deleted completely, they are retained in the vault.
Final Notes
It goes without saying that losing your backups can become your worst organizational nightmare. And while there are measures you can take to mitigate the damage, it should really never come down to that. Having the right backup and disaster recovery plan in place before disaster strikes means never having to be the bearer of bad backup-related news. Try N2WS.
