News from the Red Zone

News from the Red Zone

The Press published an edition the day after Christchurch's February quake. With communications compromised, residents were even more reliant on news, but the paper's IT team has its own stories of resilience in the face of adversity.--

However stirring the stories of Christchurch companies continuing business post-earthquake, they pale in comparison with the loss felt by the city's residents and workers.

The 22 February quake critically damaged the Christchurch Press's Cathedral Square building. On that day, delivering news and public information became more crucial and Fairfax Media NZ CEO Allen Williams said; "We will be publishing an edition tomorrow so that our readers can stay in touch with what is happening at a time when communication channels are erratic."

The quake trapped Press staff in the building, some were seriously injured. And there was worse to come; the next day a tweet read: "It is with regret that we can confirm the death of a colleague from @pressnewsroom. It is a dark day and our thoughts are with staff."

Those staff had been weeks away from moving into new premises. Fairfax Media IT had begun decommissioning old systems. In the weeks to come, they learned they would be prohibited from recovering servers and desktop devices from the Cathedral Square building before its demolition.

However, because the group's main systems were centralised, staff located at other offices around the country were able to provide operational support. As such, instead of a catastrophic loss of data and production systems resulting in the paper being unable to publish, The Press was back on newsstands the next day.

Nigel Bailey is Fairfax Media's group operations manager, a role incorporating ICT for the company's New Zealand publishing operations. "Given the circumstances and the scale, we could have been in a far worse shape if we had been in one of those other buildings, and it is horrific to think that could have been the case. Having been able to come out of it the other side, still producing papers, I think is amazing. From a business continuity perspective it was a phenomenal story."

Bailey says many readers were surprised to see The Press printing again so soon. "It gave people a sense of normality but also, importantly, it was the only source of news they were getting at that time because mobile networks were overloaded and people had battery power issues with cellphones. A lot of people had no power, so they weren't able to get onto the internet."

Centralised systems enabled sub-editing, most editorial production and page-building to be done from the Dominion Post, advertising makeup from the Waikato, sales and marketing from Wellington and credit control from Auckland and elsewhere. "We were able to have other people just log on from wherever they were in the group and continue those operations."

Dubbed 'Overdrive', the IT-change project proved even more transformational than Bailey could have foreseen. It mitigated damage to Christchurch operations.

Transformation projects can overwhelm organisations; especially those in industries undergoing profound change -- in Fairfax's case from a legacy mentality to platform-agnostic multimedia. It is inevitable, Bailey says, for transformation to encounter hurdles: legacy technologies, inflexible mind-sets and ingrained work practices.

"Although it is largely underpinned by the technology, it's effectively a cultural change."

As an indication of the scale of change, Overdrive reduced Fairfax Media's application count by at least 55 percent, eliminating legacy applications no longer supported by vendors. It cut physical servers by around 45 percent (with virtualisation also resulting in savings) and slashed supported desktop models by 60 percent. Group server room numbers were decreased by 87 percent, consolidating into a datacentre model. IT staff numbers were reduced by 35 percent.

Functional expertise, not silos

In late 2008 Bailey presented his vision for transformation to then chief executive David Kirk, based on Bailey's premise that the New Zealand company had a siloed, geographically structured IT environment: every paper served by its own IT shop and systems, data stored on local servers and desktops.

"That's an extremely costly, high-risk and inefficient environment. You had lots of really good people who worked hard and did a good job, but weren't necessarily experts at anything. You end up with a relatively mediocre outcome. It was also a really costly model."

His vision entailed implementing a new enterprise architecture platform, but not before demonstrating its benefit to the company. "The drivers were cost, reducing complexity, risk and delivering enterprise architecture that allows the various business units to adapt and grow when market opportunities present themselves. Cost was largely around headcount reduction, using our staff more efficiently and moving from that geographical spread to a functional structure -- so we had functional expertise rather than local jacks-of-all-trades."

Delivering applications largely through Citrix thin client technology would allow the life of desktop devices to be extended and lower-end devices procured.

Fairfax Media engaged HP to design the architecture and project-manage, while HP engaged Gen-i for delivery of specific components. "Some organisations don't make step-change easily when it's their own IT shop making those decisions -- they'll more likely make more conservative or incremental change," he says.

"While we have tremendous people in IT, it was important that we had a completely different perspective, that we engaged organisations that do this for a living, experts in enterprise architecture and transformational projects."

The main metric is a reduction of ICT operating costs, which has been achieved largely through staff reductions and capital avoidance.

Post-quake lessons

Having begun his media career as a cadet reporter on Tasmania's Launceston Examiner, Bailey went on to consult to Australian media organisations on enterprise production systems and organisational change. "I've spent a lot of time in newsrooms, in projects that involve workflow changes and structural changes. I have a real affinity not just for newsrooms, but other production areas."

Had he not sold Overdrive to Fairfax management, several editions of the Christchurch Press would have been lost in the course of the quakes and aftershocks, incurring significant revenue loss and damage to the brand.

Bailey praises Fairfax Media's general manager for the southern region, Andrew Boyle, and his team. "They managed the business continuity largely themselves."

The quakes tested Overdrive in extremis; although as he says, "It was a hell of a way to prove a point -- you wouldn't really want to do it that way."

He has been trying to educate management and staff when they ask why the group isn't working in the cloud. "We actually are: we run a private cloud and also run services in the public cloud. Our private cloud is based out of the Gen-i datacentre in Auckland, and then we have a 'hot disaster recovery' down in Wellington. It's scaled down because it's DR, but there is huge redundancy within the primary datacentre. If we have to go to DR it provides all our systems at a reduced capacity."

Having moved into an optimisation and maintenance phase, his division of time is likely to shift over the next 12 months towards business strategy and staff development.

One of Overdrive's goals was reversing the 80/20 split of 'keeping the lights on' versus more value-adding activities and investment. "While we're not quite there -- it's more like 40/60 -- we are well down the track and will continue to focus on this aspirational goal in the next year."

No one can prevent natural disasters. Events in Christchurch were a more rigorous test of project Overdrive than anyone inside or outside Fairfax Media had anticipated.

Sidebar: Hold the front page

ON 1 AUGUST a computer failure affected production of the Dominion Post, Southland Times, Timaru Herald and Christchurch Press, the latter failing to publish an edition for only the second time in recent years. Staff worked throughout the night to fix the outage, which was traced to a failover of Fairfax Media's Genera database. CIO asked Nigel Bailey for pointers on dealing with a critical outage.

Although Fairfax Media's IT team had sufficient internal resources to recover from the outage, given its unprecedented nature Fairfax Media resorted to external vendors to confirm the integrity of the impacted system prior to returning to full operations. "External involvement is continuing as the root cause is being investigated," Bailey says.

The Taranaki Daily News published its 2 August edition using a different system, and Bailey expects business continuity plans for each business to be reassessed as a result of the event, to see whether other publications in the group can be given similar production alternatives.

"There's always plenty to learn from any major outage and this is no different. An independent review of the outage is being completed currently, and we'd expect further recommendations to follow."

With hindsight, a key disaster recovery lesson is to question what your organisation would do should its primary, secondary and DR technology contingencies fail simultaneously.

"We'll be reframing our baseline assumptions to develop this scenario for all business-critical systems," says Bailey.

Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.

Tags IT management

More about Andrew Corporation (Australia)Citrix Systems Asia PacificHewlett-Packard AustraliaHP

Show Comments