– Michael Sigourney is president, CEO and founder of AVTECH Software, says:
Seasonal changes are a part of life. However, should environmental conditions outside impact your plan to protect the data center? Absolutely, even if it is multiple states away. It never snows inside the building although the influences of extreme temperature, rain, snow and ice can challenge even the best organization’s up-time strategy. After a month of extreme weather, AVTECH experienced several situations that demonstrate my point. Understanding what happened to us may help you prevent problems at your facility.
First, a one day “mini blizzard” caused a driver to lose control and crash into a pole supporting a power transformer. The result was a unique power outage where one of the three phases of power was lost for almost three hours. No breakers tripped, power just stopped here and there throughout the office, our mill and the surrounding town. Various UPS systems sprang into action although the random nature meant we needed to quickly check everywhere to gauge the impact. Think about it… do businesses map which power phase each outlet depends on? We will in the future.
This event happened about 5:00 a.m. and I had just arrived at work. This meant most offices were locked and it was tough to tell if equipment inside had power or not. About three hours later the power came on. Unfortunately, the primary server supporting our database went down when its UPS died. Our staff had checked that rack and everything was running fine. Unfortunately, the UPS supporting that critical server was located in a neighboring rack and powered mostly non-essential equipment so they moved on to check other hardware. We immediately changed the power source when it crashed although the pain point was the 45-50 minutes it took to bring the database back up and verify that it was not damaged. By that time, salespeople were at the office and unable to do business.
Another situation occurred during that same storm in Pennsylvania where we have a remote sales office. With VPN access used there by staff, they connect to systems in Rhode Island as if they were in the next room. The office was actually very well prepared, we thought, until a large tree soaked with ice and snow fell over to break the power and cable lines completely off the poles that feed the building. We found the office to be entirely “off the grid”. That is not the “going green” we often speak about and with over 800,000 properties there without power, we had no idea how long it would take to get the office back in operation. Four days later the power came on although by that time we had temporarily relocated the office across town. When power came on at 10:00 p.m. that Saturday night, I had to ask staff to spend their time Sunday moving everything back to the office so they could work on Monday. Needless to say, their productivity for about week was far below desired expectations and my popularity with their spouses declined temporarily.
About two weeks earlier, we had temperatures approach zero at the mill and a water sprinkler pipe in our entry tower froze and burst. We never knew there was a problem until the temperature there rose to thaw the pipes about a week later. As much as 400,000 gallons of water shot out of the pipe like a fire hose directly at an open window. The police called me at midnight to say the fire department was there and shutting down the entire sprinkler and building alarm systems. Water bounced off the brick walls in every direction and flooded the tower floor. The room looked like the ice castle in Disney’s new movie-Frozen. We were very lucky though as that area is non-essential space with almost no people traffic.
There was just one exception, the expensive Simplex master console box that monitors all the building sensors required by law for smoke, fire, emergency exits and pull stations. Alarms and beeps blended together to sound like a grade school band warming up. It is clear that water spraying out of a 3″ pipe at several thousand PSI will easily penetrate the Simplex box with their convenient air vents at the top of the unit for heat dissipation. The water was shorting out contacts to indicate alarms simultaneously throughout the mill. This drew multiple fire trucks and a full response due to the size of the mill. The fire chief shut down the entire system and demanded we have people on-site around the clock until the pipes and equipment were repaired or replaced. What’s worse is that I was three hours away in New Hampshire enjoying the Winter weather on the sea coast with my wife. Luckily, I have a great team and could call three staff members to help. They kept the ship sailing straight throughout the weekend until all repairs were completed. We were also fortunate that the tower drains into the 4′ tall 10,000 sq ft crawl space under the first floor of the mill where the water could later disperse into the ground. If this type of situation had happened almost anywhere else in the mill it could have easily cost us more than a million dollars in damage and disruption.
These experiences demonstrate that we can have great systems in place and prepare for the unexpected, however, the invisible hand of Mother Nature can reach into any data center. The best way to minimize impact, unexpected costs and downtime in a data center is to use “Proactive Monitoring” instead of relying on “Disaster Recovery”. Knowing immediately when things turn bad allows for the fastest response and reduced impact on the business.
