Disasters can happen. Consider:
- How will you recover if you lose your wallet?
- How will your business recover if ransom ware encrypts your server?
- What is your planning to survive a tornado if your business is in Oklahoma, a hurricane if you work on the east coast of Florida, or a flood if you live in low territory next to river that overflows once a decade?
Rumbi Bwerinofa-Petrozzello ponders these questions in her 7/2/17 post If Lost…Then What?
She tells of finding a wallet on the ground, walking into the adjacent restaurant looking for the owner by glancing between the patrons and the photo on the driver’s license in the wallet. No luck.
When she got home she was able to do a bit more research. She located the woman and returned the wallet.
Life hack tip: make sure you have a business card with a current phone number and email in your wallet so that if a kind-hearted person finds your lost wallet the nice person can reach you quickly.
From there she transitions to disaster recovery. A few questions for you to ponder:
- Do you have regular, automated backups of your data?
- Is your backup separate from your network so a virus in the network won’t take out your backup data too?
- Is a copy of your backup data stored off-site? (I recall an audit client that diligently swapped out a high-volume cassette from the backup machine every day and thus had a full backup of the last 7 days. The only backups were those 7 cassettes, all of which were on a shelf immediately above the server. If anything happened in the server room, all of the backups would have also been destroyed or stolen.)
- Have you checked to make sure you can actually retrieve the backup data? (I recall a small ministry that diligently made backups. When the hard drive crashed, they discovered something didn’t work in the backup program and every single one of their backups were garbage, including the copies taken offsite.)
One good and one sad story about backups
One of the educational stories I heard a long time ago was a company (don’t remember what type) located in Fargo, North Dakota, where floods happen every few years. The company was able to recover after their servers spent some time underwater.
They went to the safe deposit box at the bank. Even though the bank was itself underwater the company was able to recover quickly because their backup data was on a portable hard drive which was itself sealed in a watertight plastic baggie. All they had to do is get another server and load the data from a dry hard drive.
Another company, a CPA firm as I recall, was quite careful in setting up their automated backup process. When the flood water reached into their office and wiped out the server they were okay because they had a backup server. Unfortunately… the backup server was in … the basement. Underwater.
Address the highest risks in your company or ministry
The contingency planning most critical to you obviously depends on the nature of your company. Moments after I read the article by my friend Rumbi, I read of several companies who failed the most obvious first step of contingency planning given the nature of their operations.
Amy Dalrymple is one of my favorite reporters. I am amazed at the volume of informative, enjoyable reporting she turns out. She covers the oil patch in North Dakota, writing from Williston. At my other blog, Outrun Change, I have been talking about the oil boom a lot.
Back on 5/27/17, Ms. Dalrymple described Faces Of The Boom: Workers Adjust To Meet Changing Demands In ND’s Oil Fields.
The first person she described is the emergency manager of McKenzie County. In her first year in that position, on five different occasions she responded to an explosion at an oil facility. In four of those incidents, the company did not have any emergency plans.
Of five companies who suffered an explosion, four did not have any contingency plans.
Seems to me that if you are a CPA firm, the first and most critical step in any sort of disaster recovery or contingency planning would be to have a robust, systematic, and tested backup system. Without a backup of all those audit and tax files a CPA firm is dead in the water.
Also seems to me that if you are working with volatile, deadly, explosive fossil fuels it is critical to have some planning for disaster.
For example, it is imperative to train your people that if they hear the horn announcing presence of poisonous gas, they must instantly drop whatever they’re doing, glance at the direction the windsock is blowing, and immediately run up wind. (For those who don’t know, if hydrogen sulfide leaks out at a wellsite, the staff have a few moments to get to fresh air if they wish to live. As I understand one should not even take a moment to stop to pick up a coworker who has fallen. The gas, as I understand, at deadly concentrations does not have a scent.)
If you want to keep your workers alive and if you also want to salvage the millions of dollars of investment, disaster planning is really, really important.
Oh, since that first year on the job the emergency manager has developed a list of emergency contacts with all the companies in the area and has helped in training events for many of the companies. I will guess the level of contingency planning has skyrocketed since she started.
Consider what’s going on in your business, CPA firm, or ministry. Figure out the most likely disasters that could hit you and do some planning.
2 thoughts on “Things really can go terribly wrong. Do your backups work? How’s your disaster planning?”
I recently had a friend that lost (completely lost) an electronic file that had weeks of work in it. He was working from a cloud solution and accidentally hit a delete key for the file as a whole. The cloud service had no backup though he had previously been told it did. Not a happy camper.
That is frightening. One technique I use to avoid that very narrow risk (i.e. delete or corrupt a specific file) is to make many generations of backups. Such as workpaper-1, workpaper-2, up to workpaper-x, whatever x may be during the course of working on a major project. Still, quite scary.