Legal Technology Journal

The Legal IT Director of the Year tells his unique story of extreme disaster recovery following Hurricane Ivan.

When I accepted the job on the Island of Grand Cayman in 2002, I was aware that I was taking my family to live in a hurricane zone, but we’d experienced a ‘typhoon 10’ in Hong Kong and, although very wet and windy, it was nothing life threatening. Besides, I was reassured at the interview that ‘a direct hit by a major hurricane is an event that should occur less than once every 150 years’. Two years later I got to experience first hand just how disruptive a ‘once-in-150-years’ hit can be. Hurricane Ivan damaged 95% of homes on the island with many destroyed, shutting down power, water and communications. This places me within an unenviable group of legal IT directors who have truly tested their disaster recovery (DR) plans. Here I hope to share with you some of the lessons learnt, and give you an idea of what it is like to live and work through a disaster.

The Maples group, comprising the law firm Maples and Calder and its affiliate, Maples Finance, has over 750 staff worldwide and offices in the British Virgin Islands, the Cayman Islands, Dubai, Dublin, Hong Kong, Jersey, London and Luxembourg. Our head office is on Grand Cayman, a 76 square mile mountain peak barely protruding from the Caribbean Sea south of Cuba and west of Jamaica. Our capital, George Town, is a sophisticated financial centre – the world’s fifth largest, I am told. The waterfront displays a mixture of modern office buildings, tax-free jewellery shops and ocean-front cafés and bars to service the many cruise ships that call here most days. The financial services industry attracts top lawyers, bankers and accountants from the UK, Canada, Australia, South Africa and elsewhere, many of whom come here for the sun and tax-free lifestyle. Grand Cayman is more highly developed than other Caribbean Islands, mainly to meet the needs of the expatriate workforce as well as the numerous wealthy Americans who have second homes here.

As you would expect, it’s a great place to live and work, for which I feel privileged. The beaches are made of fine white coral sand, and the water is warm and the clearest I have ever seen. Balmy average winter highs of 85F mean it never gets cold and there is little winter rainfall. Family weekends can be spent boating, sailing, fishing, diving and snorkelling or skateboarding on the world’s largest concrete skate park, playing golf in shorts, taking your kids to karate/tennis/scuba/hockey practice, or just lazing by your pool. Idyllic perhaps, though there are downsides: summers can be humid, goods and services are expensive, with limited choice, and we frequently get hurricanes barrelling towards us. Indeed, as recently as August 2007 the Maples evacuation flight was called to take staff and their families safely away from Hurricane Dean, which thankfully missed us.

Disaster recovery planning

Prior to Ivan we had prepared a hurricane plan (our specific DR plan). One of the few advantages of hurricanes is that one is usually given ample warning of their approach. Unfortunately, the world’s most powerful supercomputers rarely agree on the projected path of these super-storms, so we have to factor frequent false alarms in our planning. Our plan in 2004 provided for:

• Protection of life through shelter procedures and provisions.
• Redundant power systems including a marine class 800Kw generator capable of powering not only our systems but the entire office including air-conditioning.
• Redundant capacity and remote access for continued IT services via other offices.
• All lawyers and senior staff equipped with laptops, BlackBerrys and second e-mail addresses.

On Sunday 5 September 2004, a week before hitting Grand Cayman, Ivan strengthened from a tropical storm into a major hurricane as it passed over Grenada. Initial computer models forecast Ivan to track north of Cayman over Cuba. For the next week we watched the internet tracking sites with interest, though without much trepidation. ‘They always miss’, ‘Cayman is not in the zone’, ‘It will never happen to us’, were common sentiments. By Thursday, Ivan’s path had moved south and was expected to be close enough for us to feel tropical storm winds. People began boarding up their homes and the more astute booked flights off the island.

Preparing for the worst

Our hurricane committee, of which I am a member, began regular meetings as we executed our plan. Many of our lawyers left the island, laptop and BlackBerry in hand (though too many forgot their chargers). Friday morning showed a more southerly path still, but the storm was expected to veer north after brushing Jamaica. It was, however, becoming clear that this was not an average storm; it was much bigger. As the island began to batten down, we continued our IT preparation, sending backups offsite, notifying staff in other offices and re-routing external e-mail to our DR accounts.

Our office building, Ugland House, is one of the strongest buildings in the Caribbean and had been designated as a staff shelter. Standing high above sea-level, our windows can withstand 100 mph flying coconuts and our double-layer roof can withstand the pull of 155mph winds. Staff were advised to come into the office if their homes were unsafe, bringing with them sufficient food and water to survive for a week. Having spent the afternoon boarding up my windows, my family and I spent a pleasant Friday evening at a friend and colleague’s BBQ, optimistically convincing ourselves that the chances of a direct hit were slim.

But on Saturday morning I checked the computer models and felt sick. The storm was huge and forecast to hit us head-on within 24 hours. After conferring with the other members of the hurricane committee, I gathered wife, kids, pets, food, water and bedding, along with a few clothes and irreplaceable documents, and headed for Ugland House. Even at over 150 miles away Ivan was whipping up powerful wind and rain, making driving hazardous. I settled my family into a third floor office and set about preparing systems and assisting staff to settle in for a long night.

A little-known fact about hurricanes is that the majority of their damage is caused not by the wind, but by the storm surge and wave action. The extreme low pressure in the centre of a hurricane literally sucks up the sea surface like an exceptionally high tide. Storm surges of 15ft or more are not uncommon. That said, 150mph winds alone are powerful enough to wreak incredible damage, as we were to find out.

Riding out the storm

On Saturday evening the power company shut down the island’s grid. As staff and families bedded down in Ugland House, we continued to run our generator to retain communication with the outside world. But around 8am the generator, the lynchpin of our DR plans, suffered catastrophic failure. We were later to find out that a head gasket had blown so badly that it had cracked the casing of the engine. With no lights or internet, a calm descended within Ugland House as parents settled children down and the storm raged outside. Radios quietly whispered recorded warning messages from candlelit offices and spaces under desks where people huddled. Some played cards, some stared through the windows at the blackest night they had ever seen and some slept; but few talked. Meanwhile Ivan had powered up and slowed down. The eye of the storm crept by 21 miles south of the island, bringing the ‘wall’, which packed the most powerful winds, right over us. For the next 36 hours the island was beaten by sustained 155mph winds and 220mph gusts within ‘micro’ tornadoes. A 10ft to 12ft storm surge began to flood parts of the island.

On Sunday morning we awoke to what I can only describe as the sound of an aircraft jet engine running full-thrust beside my window. I have seen 60mph winds in London during the great storm of 1987. They were a breeze. The ‘typhoon 10’ in Hong Kong was windy, but this was utterly inconceivable. Cars and even SUVs were blown sideways against one another in our car park, the roof of the building opposite had begun to peel back like a sardine tin and a river had formed down the road beside the office from rainwater alone. Throughout Sunday and Monday night all we could do was hunker down and wait as the building heaved and shuddered under the pressure of 200mphplus winds. There was no communication with the outside world and even the recorded messages on the radio stopped. It was too dangerous to venture outside. Virtually the entire island was underwater for several hours. Survival stories of people escaping the flood by punching holes in their ceilings to gain access to the rafters were common. On the south coast, houses over 300 yards inland had their window and door frames blown through their back yards by the pounding surf. As it finally moved to the west, Ivan’s encore was to throw 30ft-high waves at the south coast, throwing millions of tons of sand and rock inland and destroying roads and houses.

On Monday morning, the wind finally eased off and a few of us edged out into daylight. It was a different island to the one we had left outside on Saturday. Everywhere power lines were down with their poles snapped like toothpicks. All green vegetation had been stripped and sprayed like pea soup over the walls of buildings. Our office was an island surrounded by a brown lake of rainwater and the slowly receding storm surge. Dozens of boats, large and small, were smashed inland and thrown into piles like driftwood with cars, containers and debris. An entire condominium complex was flattened, and then washed out to sea. Roads were mostly impassable, some gone completely. People walked and drifted in a daze from shelters to devastated homes and back to shelters. My own home had been flooded and filled with mud and sand. Some of my storm shutters had blown through, taking the windows with them, and a large hole had appeared in part of my roof. But my house survived better than most. Some were completely destroyed. For the next 48 hours people pitched together to keep families safe from looters and disease while accounting for everyone we knew. As soon as the airport re-opened, my family, along with the many others, were evacuated.

Getting the systems back online

So what of the systems and Maples DR? Our data centre in Cayman was without power and worse, the main cabling trunk for data communication off the island had been broken in seven places by uprooted trees between George Town and the east of the island. When the generator failed, it caused a surge through the building’s electrics that shut all systems down hard, potentially losing data or damaging disks. The primary thrust of our DR was dead, but we still had contingencies in play. Our offisland Cayman lawyers were able to hold the fort using their redirected e-mail accounts, laptops, Citrix and BlackBerrys. My infrastructure manager set up a command centre in Miami to assist them just before Ivan hit and many of my team flew to Miami, Jersey or the UK to assist wherever needed. As one client later stated: ‘We noticed little, if any, disruption to the Maples service’. In a classic piece of British understatement, our website announced that ‘Due to inclement weather our Cayman office is currently closed’.

Within hours we were working to get all original systems, including our e-filing, billing, compliance and company management systems, back online. This required a parallel effort. In Cayman we worked to re-establish all original systems, power and communications while off-island the team worked to establish a fall-back data centre using the tapes we had shipped away at the last minute. Whichever came first, we would bring online to get everything back to normal. Five days later we had the luxury of both Cayman and fall-back systems available. We took the decision to bring the Cayman systems back online as lawyers began returning to the island to sort out their homes and possessions.

Re-establishing a sense of order

I would forgive you for thinking this was all reasonably straightforward, but it was not. For the first few days I was torn between sorting out my family, my home and the firm’s systems. Although I missed them dearly, I was thankful that my family was evacuated so quickly. Life without electricity is hard but without water it is unbearable. Without running water there are no flushing toilets, no showers, and little to drink and cook with. You pray it will rain so you can collect enough water to wash, or you wash directly in the rain. Seeing people lathering up their hair and washing under downpours was a common sight. Hygiene was important with the threat of disease a possibility. Nightly curfews were put in place to prevent looting and petty theft.

I moved back into the office. My home was wrecked and without a backup generator and unreliable grid power, the systems in the office needed babysitting 24-7. I spent weeks sleeping on the floor of my office or on an inflatable bed beside the computer room, setting up a laptop to play music every time grid power failed. Many of my team were outstanding during this period, also sleeping in the office and taking the time out from repairing their homes to getting everything back to normal.

As just getting a square meal was a job in itself, the chief operations officer, who was also sleeping on the floor of his office, having lost literally everything, organised some now-unemployed chefs to come in and BBQ food every lunchtime. This enabled us to tempt staff in with a rarity on the island, hot food. We set up a full-service launderette in the car park so that people would have clean clothes and we equipped a team of accountants with hammers and drills to secure the homes and possessions of staff no longer on the island. Once we had power and water to the office we had, luxury of luxuries, hot showers. Within the office we achieved a sense of order and normality, a necessity to combat the maddening devastation outside.

The island recovers

Soon some of the lawyers began to return to the island, but children were not allowed back due to the risk of disease and generally poor conditions. It was six weeks before the water was turned back on to my house, and three months before I got electricity. My wife and kids moved in with my wife’s parents in London and the kids enrolled in the local school there. As time went by, slowly the odd bar would open up with a BBQ and generator/fridge. Undoubtedly, many people here will have different memories of Ivan. I can remember the best cold beer I ever had on a sweltering evening, cheering the Canadian power company ‘line-men’ as they drove by on their way home. I remember standing in line for hours in the sun to get into a supermarket that had almost bare shelves. I remember the evening drone of hundreds of small generators running to keep lights and fans on across the island. I remember the shocked faces of people arriving back and seeing for the first time the desolation we no longer noticed.

The island recovered incredibly quickly. Within a year most of the tarpaulins were gone from roofs and a reconstruction boom was well under way. After two years most of the main tourist areas were back to normal, but even today, three years later, Ivan’s scars remain. It was an unbelievable experience that made me truly appreciate what I have, and really feel alive, but parts of it were so deeply unpleasant I never want to go through them again.

DR lessons learnt

So what did I learn? It was an enlightening experience. It is true that you see the best and worst of people under such extreme pressure, and you learn a lot about yourself. Here are the main lessons I have learnt relating to DR:

• Look after people’s needs first
  When in a disaster situation you have to look after people’s immediate needs first. Staff cannot work if they have to spend four hours a day queuing for food and fetching water. I was able to fully concentrate on Maple’s systems only when I no longer had to worry about my family. Establishing the BBQ lunches and launderette made a huge difference to morale and people’s ability and willingness to work. Make allowances for personal strife in your DR planning.
• You cannot plan for every eventuality – make flexible, general plans
  The bulk of our DR plan revolved around an indestructible generator. If we had not had contingency measures or a flexible plan our recovery would not have been as fast. Have plans that include data replication and resilience but also plans that assume this will fail. We had more than ample Citrix capacity and spare second e-mail addresses at the ready but we still thought on our feet and made adjustments right up to and directly after the hit.
• DR is your responsibility
  The firm’s business owners should decide the level of risk they can bear, but as IT managers we must advise them on those risks and provide options with which to address them. As an IT director, no one should understand the IT risk better than you. I spent the first two years at Maples campaigning to invest in laptops, Citrix, new networks, BlackBerrys and DR services. Had I not done so, I can handon- heart say that Maples and the firm’s clients would not have fared as well as they did through Ivan. Today, DR and risk management are still a very large part of my job. Help your firm grow while helping protect the business it already has. As an IT director, if you do nothing else you would do well to do this.

What have we changed since?

Since Ivan we have established a permanent replicated data centre on the other side of the Atlantic. We are now able to fail all systems over as a storm approaches and decide whether to bring our DR data centre or Cayman data centre online after the event. We call in evacuation flights before storms arrive to make sure we have all key personnel and sufficient lawyers off the island and families are safely out of harms way. We have implemented an e-mail archive that takes a copy of every e-mail sent or received for DR and general risk management. We have a detailed communication plan with comprehensive draft e-mails to avoid missing important facts during a crisis. And we have a new generator.

Andy Maclusky is the CIO at Maples and Calder.