Crisis over: Now what?

Crisis over: Now what?

The news just keeps getting worse: We now know that more than 45 million credit and debit card numbers were stolen in the TJX Companies data breach. Over the past few months, the scope of the problem seemed to grow with each announcement. The public didn't learn the (presumably) final toll until late March, when the US company filed the figures with the US Securities and Exchange Commission.

The news just keeps getting worse: We now know that more than 45 million credit and debit card numbers were stolen in the TJX Companies data breach. Over the past few months, the scope of the problem seemed to grow with each announcement. The public didn't learn the (presumably) final toll until late March, when the Framingham, Mass.-based company filed the figures with the US Securities and Exchange Commission. Even now, with the news out there, company officials aren't eager to talk: Calls seeking comment for this story went unreturned. And this kind of response is all too common. Coverage of last month's massive BlackBerry outage stressed that users were less annoyed by the failure of service than by the lack of communication from the vendor, Research In Motion Ltd.

Yet IT executives who have successfully handled data breaches and other incidents say communication is actually one of the most effective ways to contain a crisis.

"Transparency both inside and outside the organization is very important, and an important role that a CIO can play is communicator," says Darryl Lemecha, CIO and senior vice president of shared services at ChoicePoint Inc., a data aggregator in Alpharetta, Ga., that suffered a security breach in 2005 and learned firsthand the critical role that honest communication can play.

CIOs are making headlines these days, but not always for the right reasons. Security breaches, crashed Web sites and other public technical snafus create the kinds of crises that put IT leaders front and center.

Are you prepared?

You'd better be, because how you follow up in the immediate aftermath of a crisis can affect not only how the event is perceived, but also how successfully you'll avoid trouble in the future. It's not so much what occurred that matters, says Mike Tainter, IT service management practice director at Forsythe Solutions Group Inc. in Skokie, Ill. It's "how it was handled and communicated afterward. That's what really matters," he says.

As CIO, you can't leave crisis management to other executives, even if you're buried in the immediate task of solving the technical problem that precipitated the whole mess. You need to both lead the IT work and play a key role in the business's efforts to cope with the aftermath. Here's how:

-- Rely on your plan.

This is no time to wing it. "You shouldn't stand back and scratch your head and say, 'What should we do?'" Tainter says. Instead, get out your incident response plan and put it into action. As your IT people start running down the technical causes of the crisis, you should start implementing the plan that lays out your business responses, your key contacts, and your public and regulatory obligations.

-- Work with the right people.

"We're operating in two courts at the same time -- the court of public relations and the court of law," says Joe Brennan, executive director of communication and marketing at Ohio University, which suffered a series of data breaches in 2006. "The CIO has to know that what the organization says and does can expose the company to legal risks."

To minimize such risks, reach out fast to the nontechnical folks who can help you, says Janice Malaszenko, who has served as CIO and chief technology officer at several Fortune 1,000 companies. Those people include human resources staffers, who can help deal with employee-related issues; public relations people, who might need to field questions from the media; and legal staffers, who will help craft responses to public and legal inquiries.

You also might need the chief financial officer to authorize emergency spending, accountants to track spending for insurance claims, or operations folks to work overtime to make adjustments as IT gets everything back up and running, Brennan says.

It's also important to touch base with executives from your vendor companies, says Dennis Fishback, CIO at Calpine Corp., an energy producer in San Jose. That way, you can reach them quickly if you can't get what you need through the normal chain of command.

-- Identify the problem, then dig deeper.

The recovery process must include a root-cause analysis, Malaszenko says. So while your IT team is containing the situation to prevent further damage, you and your staff must be analyzing the underlying problem to prevent it from happening again.

That's just the start, though. "Look around at the environment and ask what other scenarios or situations could happen," says George McBride, director of IT risk consulting at Aon Consulting Worldwide in Chicago.

If your firewall was breached, for example, look for other vulnerabilities that hackers could exploit. If your server crashed because of a bad patch, check whether other servers are using similar patches. "It's easy to forget this, because you're so focused on the problem at hand," McBride says.

Also, examine why the crisis wasn't averted in the first place by your early-warning processes and systems.

-- Communicate with all stakeholders.

After experiencing a technical meltdown, people want answers. It's your job to help supply them, Lemecha says.

Following ChoicePoint's 2005 breach, Lemecha made sure that various stakeholders -- the board, customers, consumers, the IT organization, all ChoicePoint associates, the news media and regulatory bodies -- knew what had happened and what was being done to fix it.

"Initially, I spent time on the phone with various business unit leaders, discussing with customers what actually happened and how the customers' data and our systems were secure," says Lemecha, who recalls speaking to one upset customer for two hours.

"He had a large number of misconceptions about ChoicePoint, the incident and the data that we maintained on consumers," Lemecha says. "At the end of the call, he was so pleased with what ChoicePoint was doing that he sent [CEO Derek Smith] a note congratulating us on our steps."

Lemecha talked with corporate information officers and their senior staffers and explained the steps IT was taking to protect information, such as the removal and truncation of sensitive data.

He also assisted in the preparation of board presentations and worked with others to create a Web site that offers consumers up-to-the-minute information on privacy efforts.

Eventually, "everybody participated in communicating to customers," Lemecha says. The business unit general managers coordinated the customer communication efforts, first by calling customers and later by distributing documents that explained ChoicePoint's privacy and security practices.

"If we were to do it all over again, I would have all these materials in place upfront so that we could have distributed information to the media, regulatory bodies and customers -- and posted information on our Web sites simultaneous with any consumer notices," Lemecha says. "This would reduce the amount of false information that [proliferates] on the Web in times of crisis."

After a crisis, he says, the CIO is uniquely qualified to communicate in virtually every direction. "The CIO is in the best situation to really understand all the technical issues, the business process issues and how they all come together," Lemecha says. He can "talk up to the executives and talk down to the technologists" about the direction the company is headed, as well as convey information outward to customers.

-- Connect with affected colleagues.

When hardware problems affected his company's energy trading operations, Fishback flew from his San Jose office to Houston to meet with his senior management team -- a dozen or so officers and directors who are based there.

"It's important for them to see that the highest level of management is involved in trying to fix it," Fishback says, adding that face-to-face meetings are the only way to effectively convey that message and build the credibility and trust needed to move forward.

"They've got to know you've got some skin in the game and that your attention is fully focused on resolving the issue," he says.

-- Support your people.

Stress will be running high in the weeks after a crisis, with some employees even wondering if their jobs are on the line. But you won't get the best out of your workers if you let those doubts fester. Instead, back up your team.

"Acknowledge to the staff that you're not interested in boxing them around the head on why you're having a problem, but you're more interested in helping them solve the problem," says Fishback.

Shawn Ostermann, who served as interim CIO at Ohio University following the data security breach there, says he went to various meetings to show solidarity with staff but didn't call "all-hands meetings" that employees might see as wasting valuable work time.

On the other hand, he says he respected his staff's need for downtime, so despite the continuing crisis, he sometimes sent them out to grab lunch or to play volleyball to work off the stress.

"You have to be an advocate for [your] people," Ostermann says.

-- Move the organization ahead.

It might be tempting to take a break once the crisis has passed, but it's smarter to use that time while everyone is still engaged and energized to push through the changes that will prevent a repeat incident. The time after a crisis "is often a chance to polish up on policies," says Aon's McBride.

That means examining not only the organization's technology, but its people and procedures as well.

ChoicePoint did just that. Lemecha worked with other executives to conduct a process review following the 2005 security breach. As a result, they created or enhanced 90 policies and procedures to help prevent such a breach from happening again and added the new role of vice president of consumer advocacy.

-- Take a final look back.

Documenting your reaction to a crisis and holding a postmortem that examines your responses are crucial to learning from the event, McBride says. The postmortem needs to happen soon, while the incident is fresh in everyone's mind. This kind of analysis, he says, can help organizations develop "better and more efficient ways to respond to a crisis."

Don't compound the problem

When you're coping with a crisis, sometimes what you don't do is as important as what you do. Here's some advice from CIOs who have been there:

Don't create a power vacuum. You or a designee should be available to make decisions as your workers try to identify the technical fixes needed to contain the crisis. Also, make it clear when and how workers should escalate disagreements, disputes or any other problems that they can't handle.

Don't promise anything you're not positive you can deliver. If you fail to deliver on promises, you can hurt your credibility and damage morale even further.

Don't push through too much change too fast. Your staff can probably handle a few weeks of round-the-clock work, particularly if they are rallied around fixing a problem. But they'll quickly burn out if they're forced to implement numerous improvements in the immediate aftermath of a crisis.

Don't be too hands-on. It's important to show solidarity with your IT staff, but you shouldn't spend too much time in the trenches. Instead, balance your time among all the constituencies you serve -- your workers, your colleagues, the public, the CEO and the board.

The incident-response plan

Here are some points that all IT-specific incident-response plans should include:

Scenarios: Pick incidents that are representative of the various crises that could hit, and then outline basic strategies and tactics to handle them.

The team: Identify which departments have roles to play, and name specific people as team members and backups. Go outside the company, too, and identify the key suppliers and service groups most likely to play a part during a crisis.

Communication: Appoint a communications czar to get everyone talking during an emergency. Ensure that each team member has a list of bridge lines, conference call numbers and intranet sites for collaboration from diverse locations. Include on that list the home phone numbers, nonwork e-mail addresses and cell phone numbers for team members and backups.

Responsibilities: Detail team members' responsibilities and authority so they can get right to work without having to seek approvals from higher-ups. Think about roles in layers -- like customer communications, internal communications, partner communications and media communications. Be sure to include people to track spending and document the response. Also include guidelines that spell out the values that will guide the response. For example, getting accurate information to customers as quickly as possible could be an overriding principle for one company; safety could be the top concern for another.

Safekeeping: Consider how and where the plan and accompanying documentation should be stored to guarantee both security and access during all sorts of scenarios. Some companies keep multiple copies on various media in several locations.

Management: Appoint someone to manage the document. That includes updating the plan and training new team members as employees come and go; organizing drills, tests and simulations; and improving the plan based on experience as incidents occur.

Testing: Test your plan. Start with a walk-through to identify any glaring problems. Then simulate an actual event, watch what works and what doesn't, and learn and revise.

Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Tags disaster recoveryBusiness Continuity Management

Show Comments