In case you were trying to access any of the Thoughts Media sites (Pocket PC Thoughts, Digital Media Thoughts, Zune Thoughts, Smartphone Thoughts) or Mobility Site this weekend, you most likely noticed that you couldn’t. Evidently there was an explosion at The Planet’s Houston data center. According to a letter sent to The Planet’s customers, “three walls of the electrical equipment room on the first floor blew several feet from their original position, and the underground cabling that powers the first floor of H1 was destroyed.” But that’s not all.

Read the letter* I’ve posted below, and tell me if the idea of this type of catastrophe doesn’t totally freak you out. Of course there are a lot of smaller hosting companies who resell Planet server space; I think this may turn out to be an even larger problem than initially thought. :-/

I hope that the 9000 servers affected are soon up again, and I hope a bare minimum of data (if any) was lost; what a mess.

Dear Valued Customers:

As previously committed, I would like to provide an update on where we stand following yesterday’s explosion in our H1 data center. First, I would like to extend my sincere thanks for your patience during the past 28 hours.  We are acutely aware that uptime is critical to your business, and you have my personal commitment that The Planet team will continue to work around the clock to restore your service.

As you have read, we have begun receiving some of the equipment required to start repairs.  While no customer servers have been damaged or lost, we have new information that damage to our H1 data center is worse than initially expected. Three walls of the electrical equipment room on the first floor blew several feet from their original position, and the underground cabling that powers the first floor of H1 was destroyed.

There is some good news, however.  We have found a way to get power to Phase 2 (upstairs, second floor) of the data center and to restore network connectivity.  We will be powering up the air conditioning system and other necessary equipment within the next few hours.  Once these systems are tested, we will begin bringing the 6,000 servers online.  It will take four to five hours to get them all running.

We have brought in additional support from Dallas to have more hands and eyes on site to help with any servers that may experience problems.  The call center has also brought in double staff to handle the increase in tickets we’re expecting.  Hopefully by sunrise tomorrow Phase 2 will be well on its way to full production.

Let me next address Phase 1 (first floor) of the data center and the affected 3,000 servers.  The news is not as good, and we were not as lucky.  The damage there was far more extensive, and we have a bigger challenge that will require a two-step process. For the first step, we have designed a temporary method that we believe will bring power back to those servers sometime tomorrow evening, but the solution will be temporary.  We will use a generator to supply power through next weekend when the necessary gear will be delivered to permanently restore normal utility power and our battery backup system. During the upcoming week, we will be working with those customers to resolve issues.

We know this may not be a satisfactory solution for you and your business but at this time, it is the best we can do.

We understand that you will be due service credits based on our Service Level Agreement.  We will proactively begin providing those following the restoration of service, which is our number priority, so please bear with us until this has been completed.

I recognize that this is not all good news. I can only assure you we will continue to utilize every means possible to fully restore service.

I plan to have an audio update tomorrow evening.

Until then,

Douglas J. Erwin
Chairman & Chief Executive Officer

Link: The Planet

Link: Neowin - Explosion at theplanet knocks out 9000 servers

Link: The Planet Forum

*Letter was posted with permission from Chris Leckness, The Planet customer and owner of Mobility Site.

Update: I just received my first notification from a company hosted by The Planet; I wonder how many more of these type emails will be going out over the next day or so…

Hello,

Last night, there was an explosion and fire in one of our hosting provider’s data centers. No one was hurt and all is well with our servers, as they are not located in the affected facility.

However, the DNS server that hosts our domain is in the affected facility. This means that while our servers and website are fully operational, your computer can no longer “find” them on the Internet.

Because of the way the Internet works, this outage gradually spread throughout the day, and we were not aware of the problem until late in the day (we did not receive a notice from our provider). Likewise, when the DNS servers come back online, access will take time to reach everyone. Currently, there is no ETA.

We sincerely apologize for the inconvenience and will update you as soon as we have more information.

Sincerely,

Your Friends at BigCrumbs.com