May 11, 2016: A “file integrity issue” at CRM-provider Salesforce led to a day-long outage. Five hours of data were lost—along with a whole lot of customer productivity—as the company scrambled to fix the disruption in service. Jan 31, 2017: GitLab, the open-source Git repository backed by venture capitalists such as Y Combinator and Khosla Ventures, experienced an epic outage of its online services. All five of GitLab backup techniques failed to kick in and some production data was lost forever. August 3, 2017: Due to a configuration mishap on the Meraki Cloud, networking giant Cisco Systems accidentally deleted a half day’s worth of uploaded user data. Programmers and engineers never intend on making errors. But the harsh reality, as illustrated above, is that mistakes that wipe out data and backups can happen. And when they do, the results are often disastrous. In some cases, even the simplest of human errors can delete entire instances or databases. Other times, it’s difficult to understand how a company could have made so many mistakes along the way. Either way, the ramifications can be monumental.
It’s Not Just About the Money
Consider this: The average cost of one hour of downtime for enterprises can exceed $300,000. The massive 2016 outage and subsequent backup and data recovery deployment failure at Southwest Airlines stranded thousands of customers and triggered a loss of $80 million. Moreover, 43 percent of companies that suffer a major data loss never reopen, according to a University of Texas study. Code Spaces, a former project-management startup, learned this lesson the hard way. After experiencing a loss of customer backups, the company was forced to close its doors—forever. While lost data is gone forever in many cases, it’s not necessarily “game over” if you find that your backups have been lost. In some scenarios, it may be possible to recover data with a whole lot of effort. And now—before a catastrophe occurs—is the time to make sure you know what to do when disaster strikes.Tips for Recovering Lost Backups
Own the Mistake
Companies that take public and swift corrective action fare far better than companies that try to cover it up. Provide your users with as much detail as possible to show that you’re fully invested in solving the issue.Piece Together What You Can
Speak to your developers and engineers. Chances are that someone on your team may have at least a portion of the backups stored somewhere, perhaps in their own files or in code repositories. You can also begin to piece together what’s left, with reverse engineering.Create a Complete Disaster Recovery Plan
Once you have recovered whatever you can, it’s essential to build a disaster recovery plan. Without it, your organization has no way to ensure that during times of failure, all involved parties can work together in a fluid manner to fix the situation with as little fallout as possible. The plan should include:- An up-to-date contact list to ensure that all critical players can be reached immediately.
- A communication plan to ensure that employees know their role in a situation of data loss.
- A current network map and inventory list to ensure that all areas are accounted for.
