On Saturday, February 20th at approximately 2:20 p.m. Mountain Time, the shared server that my Internet domain is on experienced a hardware failure as a result of an annual fire system inspection at WestHost’s Data Center (DC). An inadvertent release of Inergen (a fire suppression product) was triggered by an actuator that was not removed by the vendor as required in the fire suppression pre-test checklist. Not all servers were affected but mine was.
My website was disappearing a few pages at a time before it disappeared completely as the RAID disk systems were powered down followed by the server. I figured that this was not a big deal as I keep backups of my entire site including my email and WordPress blogging system. What I didn’t count on was that:
- email sent to me while my server was down was being bounced
- they weren’t answering trouble tickets nor picking up the phone
- I had no idea what server my account was on so the status board was useless to me
- When I finally talked to them they would not allow me to restore my account to another server
While bouncing email is not a good thing, at least I only had my website there and wasn’t running my actual business in the cloud. I also have a backup email account at a different ISP (thank you Apple MobileMe!). Now imagine this event happened at Salesforce, Amazon, GoogleDocs, or Azure where I was running my business applications and not just a website. Do you have a contingency plan for an outage of your cloud-based business applications?
I’m an all Macintosh shop and I would rather take my chances in-house where I can pop down the street to buy a new Xserve, MacPro, or iMac and restore my backups myself. If your business is in the cloud, you may want to be thinking about having a backup at another provider or in house, even if you need to recover selectively.
