But since, you’re expected to roll out a series of complex, simultaneous projects in weeks rather than months or years, you’d better get on with it.
None of the above business problems were insurmountable, but the real world outcomes are unique. The Lord of the Rings (LOTR) trilogy has made history, winning a total of 19 Academy Awards and the Oscar for best visual effects in a motion picture three years running. But it is easy to overlook the fact the LOTR trilogy could never have been delivered without world-class project management skills.
Scott Houston, Weta Digital’s chief technology officer; Milton Ngan, digital operations manager; Paul Gunn, systems manager; and Duncan Nimmo, IT manager of Jackson’s production company, 3foot6, are satisfied with a small place in filmmaking history. “The credits list is very long and you wait a long time to see the IT department, but it’s got us all there,” Gunn says proudly.
The dictatorial power a film director carries is unique to the movie world. Jackson personifies the notion of a ‘benevolent dictator’ – and a far more effective one than most corporate CEOs or political leaders. The loyalty of his cast and production team and abilities to get each job done well, to his specifications, suggests corporations and government departments would have much to gain from a Jacksonesque leadership style – in spite of corporate governance failure being the business fashion phrase of the moment.
On LOTR, failure – in an IT mega-project sense – was simply not an option. “There’s no chance of any of our projects turning into an INCIS, because we don’t have the time. We do it straight through to the end in one go. We don’t expand it like ‘bloatware’,” Gunn says.
The scope of Weta Digital’s projects ensures a classic ‘short timeline, many milestones’ model. “Because we do everything so fast, with such short delivery times,” Gunn says, “one of the things we’re defended against, by definition, is mission creep.”
“The big difference, I think, is that in film there is a much shorter time to think and plan than in the corporate world, so one has to anticipate and act smartly,” Nimmo says.
During the making of The Fellowship of the Ring, the first part of the trilogy, under the IT leadership of Jon Labrie, satellite technology connected Jackson remotely so he was able to direct the action on three locations simultaneously using real-time audio and video links. Later, this ‘virtual studio’ enabled Jackson to be in a London studio with the orchestra recording the film score, while new special effects sequences were screened over the broadband link from Wellington.
There is now a 10Gbit link to Weta Digital’s 1000 processor renderwall and, across the Miramar network, there is a single gigabit connection between the various offices and facilities.
Film Net, the network developed with Telecom Advanced Solutions, quickly became mission critical. It cut out the need to courier footage back and forth and saved Jackson some of the time he had lost collecting awards at ceremonies around the world. “Nobody’s really tried transmitting synchronised sound and picture in realtime across the world and actually working with it,” Ngan says.
Some of these remote collaboration tools have enabled Weta Digital to set world class benchmarks, says Houston, raising issues of competitive advantage. “The feedback we’ve had since the end of the project has been, ‘How can we get access to it?’ Some of the discussions we’ve been having internally are that these are actually key differentiators, allowing us to compete with any other post-production studio in the world without the tyranny of distance.”
Raising the bar
By the middle of 2003, it had become clear yet more processing power would be needed. There were around 800 digital special effects in the second film, for the third, 1500. The team still had not yet begun work on the big battle scenes for The Return of the King but the facility was already up to 95 per cent of its processing capacity.
“We knew the complex shots in high-resolution were coming,” says Houston. “We had a meeting with the producers and said there were two choices: ‘Tell Peter we can’t have all those soldiers on the battlefield, or we need a little bit more money.’ As Milton said, both to New Line and to Peter’s credit, they didn’t ask for sacrifices from Weta Digital. The difficulty is, this was in August, and we said, ‘You’ve got about 24 hours to make the call and then we’ve got about two to three weeks to make it happen.’”
Earlier in the year, Weta Digital had an installed base of 2300 processors for the digital rendering of the third part of the trilogy. Around another 1000 would be needed. A network that could handle this processing farm’s output, of up to 10TB a day, would also be required. There was just one problem: They had less than three months to go before film production was completed. So the business case for a new hosting facility came together fast.
“An 80 per cent decision now is better than a perfect decision four months down the track,” Ngan says. “If we deliver late, that’s huge amounts of lost revenue. It’s much simpler to throw a bit more money at the problem to get it done faster and as an insurance to guarantee your timely delivery. The only thing that would bend was the guys with the chequebook. Peter would not compromise on the vision.”
Corporate IT directors might view jealously the way the Weta Digital team made such decisions. Upfront, says Ngan, at times when urgent action is needed to get a project underway, “There’s not much in the way of written documentation, most of it’s verbal, face-to-face discussions. We have to produce evidence that we’ve got a requirement, but we keep track of that through our monitoring systems.” However, as in all production houses, Weta Digital’s accountants, production team, directors and New Line Cinema (in the case of LOTR) keep tight records of budgets and expenses.
The hosting centre project, situated in Telecom’s central Wellington exchange, was scheduled to take three weeks, but completed in an astonishing two with Telecom Advanced Solutions overseeing project management and integration. The processing cluster comprised 504 Blade Servers (1008 Processors), each containing dual 2.8Gigaherz Xeon Intel processors and 6Gb of RAM, representing an 83 per cent increase in processor capacity. The long-haul 10Gbit Ethernet link between the hosting centre and Weta Digital’s studios took care of the network connection.
Gunn agrees most corporations would be operating to a different timeline. “They might start having unofficial chats nine months in advance, a year in advance; whereas the phrase that comes to mind for our facility has been ‘wild growth’.”
The films couldn’t have been made without conflict and challenges; sometimes, the IT team pushed even its impressive inventory of hardware beyond its operational limits. “We’ve had our set of challenges,” Houston admits, “some real nightmares – switches spring to mind. But nothing I would feel we haven’t succeeded on. We just overloaded it. We’ve gone into the unknown, somewhat.”
Before the hosting facility came online in 2003, Weta added 20Tb of storage, resulting in 60 straight hours of work for Ngan. “We were working all the way through Sunday, right into Monday, and then back again on Monday night and we finished it by Tuesday. The challenge we have is, by the time we get the budget cash and implement the expansion, we’re quite heavily into production. And that’s when we need to have the system up, 24x7.”
Windows within which to make changes are narrow and difficult to justify. “What we ended up doing is cramming a lot of changes into one window, and we knew it was a bad idea,” Ngan acknowledges. “Something just had to happen to highlight to management and production that it was going to cause a problem. Ultimately, it took a 60-hour outage for them to realise that perhaps we should slow down, give these guys a bit more opportunity. There were hints of a problem, but we weren’t quite sure what it was. We tried tracking it down and it wasn’t until we put the extra load on the system that it really showed up.” What appeared to have been caused by a fault in two file servers turned out to be in one of the new switches, and it brought down the network.
Another in a seemingly unending line of firsts for LOTR was the delivery of the second and third films to the studio in Hollywood via broadband, gaining Weta Digital the two or three days it would have taken to courier them to the States and enabling them to make last-minute adjustments.
MIS put it to Nimmo that, with everything riding on the arrival of that final cut, the idea of sending the film into the ether must have required a suspension of disbelief. “We did test it first,” he assures me.
The first time they tried such an intercontinental transfer of the film back to Wellington, Nimmo, his laptop and a FireWire drive were in a studio in LA. He had 48 hours to transmit the files; an experience he describes as one of the most terrifying moments of his life.
“Actually being the person who had to hand the print master of the film to the studio, knowing that there were thousands of people back in Wellington who had put their hearts and souls into getting it there, on time, for that moment, being the final link was terrifying.”
Nimmo had another moment of filmic terror while in London, where he was almost mugged. “I had an iPod in my pocket at 5am in Soho and there just happened to be some QuickTime files of the film in my pocket. I was on my cellphone and these two big blokes saw me and turned around and started following me. I walked faster and they walked faster, then I broke into a jog and they started running. I could not have the iPod with the film taken off me, so I ran and got away.”
No time to waste
Before Weta Digital’s processing power is needed for Jackson’s next production, King Kong, an interesting commercial opportunity presents itself. Talks are underway to see if big consumers of computing cycles outside the film industry will be allowed to use Weta’s huge server farm. “At the moment, we’re talking to Telecom and we hope that we can get to a partnership agreement,” Houston explains.
Potential customers include the oil and gas industries, universities, genetic research and Crown Research. Houston believes 70 per cent of external users of this surplus processing power would come from offshore, making a potential partnership with Telecom Advanced Solutions particularly attractive.
“The important thing for us, though, is that as we start to work with other organisations that either have or need processing power, it can become more of a utility-type computer model.”
Houston reckons there will be surplus processing power and storage available at Weta Digital for nine months of the year. “We’ve been talking to Ian Foster [Professor in Computer Science at the University of Chicago], who’s being called the ‘Gridfather’. He’s a Kiwi, born in Wellington, who went to Wellington college. He came and visited, and grid computing is the direction we want to have a look at.”
Never say die
Looking back over his experiences on LOTR, Houston believes too much has been made of the technology, the huge budget and the partnerships Weta Digital has wrought. “Ninety per cent of the project’s success is the ‘never say die’ attitude of Milton, Paul and Duncan and the IT team. They really haven’t known failure and we don’t know what we can’t do, so we just do it. We’ve attempted things most mature organisations wouldn’t tackle – probably because they know it would be too difficult, or their careers would be at risk. This attitude really comes down to Peter: ‘Give it your best shot – and by the way, it has to work.’”
The incentives appreciated by IT people tend to differ from those for other employees and there were many non-financial benefits to be had from working on this fantasy IT project, says Nimmo. “No one else gets to play with the coolest gear or fix Liv Tyler’s email.”
Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.