We are pleased to announce that we are back online!  Our office power – out since Monday – was restored this afternoon (Thursday), so all our local servers, facilities and services are fully operational again…

However – from an IT and life/business continuity perspective – Hurricane Sandy (and the widespread damage it left in its wake) have caused us to carefully review our internal emergency preparations, those of our clients, and made us ponder how we can all be better prepared in the future…  Prepare yourself – this is a long rant!

Image: debris deposited on land by Hurricane Sandy, along with an ironic, sardonic road-sign in a shoreline community area of Stamford, CT, near our office in Glenbrook Industrial Park.

The Imperfect Storm:
Beyond the general, dreadful impact of Hurricane Sandy on the entire region, this storm created some unexpected and unusual situations: 1) no power at the DreamIT office, but (thanks to recent improvements) power on at home, 2) the worst tide/storm surge level seen in 100 years and widespread shoreline damage, and in particular, 3) flooding in lower Manhattan, so severe that it caused a general 5-day (projected) power outage from 30th street to South Ferry, closed Wall Street for two days (first time since 1880’s), gridlocked (or closed) bridges and tunnels, and ground business to a halt.  And, with many of the big datacenters running in downtown Manhattan, even the best prepared facilities had to shut down or is running at partial capacity/functionality on a generator…  Are there some lessons to learn from this, in this electricity-hungry world?

First, we have to ask, how can this have happened? The response I’ve heard repeatedly the past few days regarding lower Manhattan is something like, “We never expected flooding like this, so everything’s in the basement!” or “We have no experience with floods, so we never prepared for them!”  Basically, boilers, generators, and all other heat, power and emergency equipment in all or most NYC buildings, are installed where floodwaters will naturally flow to – ground level or downstairs in the basement.  Most shoreline-area people and businesses know better, and have building codes, preparations and protocols to protect against seawater and high tides – to some degree – although this hurricane was far worse than expected. But it seems that someone (or many people) forgot that Manhattan IS ONE BIG SHORELINE and large portions of it are only a few feet above high tide level!  As the storm moved north, then inland and northwest, the rains and high winds did the rest of the damage.  Where our offices are in Connecticut – away from the shore – we suffered mostly from the high winds (there was not too much rain!), which ripped down so many trees, power and communications lines…

So, the real point is that this storm’s damage was far-reaching, and exceeded the worst-case scenarios anyone had imagined.  Everyone is now re-thinking their emergency planning, with 20-20 hindsight – there is no better teacher than (bad) experience!  Hurricane Sandy had a terrible impact on people and businesses. But this post is focused specifically on the impact from interruption of power and communications, which was the most stunning and unanticipated result of this natural disaster.  We have become so incredibly dependent on the internet, web, blog, e-commerce and communications networks, that when the electricity stops flowing, we are stopped in our tracks! Infrastructure broke down, cash and gas shortages began days before the storm, and are still going, but guess what – ATM’s and gas pumps all run on electricity!

People scrambled around – to friends, offices and public facilities with power – charging mobile devices, so they could make phone calls, check email, and maybe do some basic work. But with cell and Wi-Fi services interrupted or intermittent and unreliable at best this week, nothing was timely and urgency was meaningless. Text messages took hours (or days) to go through, cell phones were not ringing for inbound calls, emails were delayed, and many other strange events proved that when the networks and grids (power and communications) are hit this hard by weather, we cannot function. In the end, it seemed that more people were communicating via Facebook than any other way – somehow, people without power would get a little recharge, login and post a status update!  But we can’t run our businesses like that, can we?

In the case of RANDOMATRIX (formerly DreamIT), we could not have imagined power being interrupted more than briefly at our office, due to the size of the industrial park and building where they are located. Sure, we have UPS (uninterruptable power supplies) on our servers, and we only host internal software systems there (not email, etc.), so we should have been OK right? Well, after the power went out and we downed the servers, we discovered the huge trees and downed wires all around the area, which forced the power utility company to shut the entire power grid for the area, affecting everyone – even the biggest business – in that area. This is the same basic thing that happened to lower Manhattan: they had to shut the grid completely, to protect against worse problems (fire, etc.)…

What about the CLOUD?
All the buzz these days is about the “cloud”, “SAAS” (Software As A Service), Off-site vs. On-Premise and hosted stuff vs. local stuff. Mind you, this debate has been going on for years, the Cloud is in its sixth or seventh iteration, and all this hosted stuff is really a return to mainframes like the old days (only better), but that’s a different blog post/rant…  However, in the aftermath of Hurricane Sandy, especially in this densely populated region, we have to consider whether any of this disruption of power, communications, life and business could have been avoided or lessened by putting more systems and services into the Cloud, rather than having them onsite, on-premise and local in our offices/homes. Of course, the Cloud would not solve everything…if the power’s totally off, all devices accessing the Cloud would eventually run out of power, but…if we have all or most of our key servers, business-critical applications, and backup systems hosted in some weather-proofed, flood-resistant, multi-power-grid, and redundant backup datacenter, wouldn’t that greatly improve our ability to function through and recover after a natural disaster?

RANDOMATRIX (formerly DreamIT) was crippled for 4 days because our systems (which could not be easily moved to our homes) were in the office, and had no power.  Many people and businesses in the region have been similarly affected, but somehow, people have found places to grab electricity, or used that little gas generator they bought/rented that only makes enough juice to run a refrigerator and one lamp, to recharge their mobile devices enough to jump on Facebook to communicate!  And if you think about it Facebook is really, basically a “Cloud” server/application (like a big-iron mainframe of old) running somewhere out there on the world-wide-web, with multiple points of presence, redundant servers, and people found a way to get to it no matter what, even during the darkest hours!

How did (or will) people get to the Cloud?
Well, even though cell and Wi-Fi services suffered, and faltered, or performed poorly, they did not completely fail. The airwaves seemed to be the point-of-access for many people – one friend of mine posted to Facebook 20+ times per day, including “power still out” status updates entirely via his smartphone! I guess the key was that everyone had a friend with power, an office/public place, or a generator where they could recharge devices. The rest were getting into the Cloud via landlines: cable, phone/data networks from home, work, shelter, public facility or a friend’s house. The key is that it’s UNLIKELY both home and work/office power would be interrupted, right?  And, if power is interrupted in a more apocalyptic fashion, we’d all have worse things to worry about than accessing our favorite Cloud application…  🙂

Conclusion?
So, using the Facebook example, it seems reasonable, accurate and logical to surmise that if our key systems are out there in the Cloud, we will find a way to get to them.  And, in the next natural disaster, wouldn’t that help us all to function, both personally and professionally?

Our team thinks the Cloud is the way to go, even with its performance/speed issues, applications forced/re-designed to function in browsers, on smartphones and other less-than-perfect environments.  Sure, we LOVE a fast, functional application running on a high-powered computer sitting on a desktop, but Hurricane Sandy made us re-think what’s important and best for continuity and safety…

See you in the Cloud!  🙂

Randomatrix: Instagram | Twitter | Facebook | Tumblr | LinkedIn

From #random2results

info@randomatrix.com

rmx

(formerly DreamIT)

NOTE: with this post, we are not equating the relatively small impact on our operations, with the disastrous damage done to many other people and businesses in the region! We are simply using a few post-Sandy scenarios to tell a story from an business-disaster-preparedness perspective, and to explore ideas for improved business and information technology readiness in the future… We feel personally and professionally fortunate, knowing that many of our friends, neighbors and colleagues having suffered terribly, many of whom are still without power and have not begun to assess the damage… Our hearts and best wishes go out to all those affected and recovering from Hurricane Sandy…