I Got Burned And So Can You!

On Saturday, February 20th at approximately 2:20 p.m. Mountain Time, the shared server that my Internet domain is on experienced a hardware failure as a result of an annual fire system inspection at WestHost’s Data Center (DC). An inadvertent release of Inergen (a fire suppression product) was triggered by an actuator that was not removed by the vendor as required in the fire suppression pre-test checklist. Not all servers were affected but mine was.

My website was disappearing a few pages at a time before it disappeared completely as the RAID disk systems were powered down followed by the server. I figured that this was not a big deal as I keep backups of my entire site including my email and WordPress blogging system. What I didn’t count on was that:

  • email sent to me while my server was down was being bounced
  • they weren’t answering trouble tickets nor picking up the phone
  • I had no idea what server my account was on so the status board was useless to me
  • When I finally talked to them they would not allow me to restore my account to another server

While bouncing email is not a good thing, at least I only had my website there and wasn’t running my actual business in the cloud. I also have a backup email account at a different ISP (thank you Apple MobileMe!). Now imagine this event happened at Salesforce, Amazon, GoogleDocs, or Azure where I was running my business applications and not just a website. Do you have a contingency plan for an outage of your cloud-based business applications?

I’m an all Macintosh shop and I would rather take my chances in-house where I can pop down the street to buy a new Xserve, MacPro, or iMac and restore my backups myself. If your business is in the cloud, you may want to be thinking about having a backup at another provider or in house, even if you need to recover selectively.

1 Comment to “I Got Burned And So Can You!”

  • Lance says:

    I definately agree that it is important to make backups of your Web site and to have a contingency plan for when a disaster does happen. Even though WestHost does, the majority of shared hosting does not make backups of thier clients data for free, let alone keep them for multiple days, and so being prepared is always a must. When a disaster does happen to servers, there are multiple people calling in, sending support tickets, and flowing into online chat. It could take a long time for a technician to become available and get to your particular question or concern. In order to inform clients, WestHost posted updates of what the progress was on the server update page, along with keeping clients updated on Twitter and the WestHost forums. However, its not the same as speaking with a live person during a crisis such as what WestHost experienced. Thats why backups are good to have so that you can establish your site and email again somewhere else, such as the free temporary account WestHost offered thier clients until their server was back online and files restored from WestHost backups. So I think that you definately have the right idea to keep your site and email backed up and informing the public about it.

Post a Comment