For most Auckland businesses, 12 June 2006 was a day of reckoning. On that day most parts of Auckland suffered from a protracted power outage, and exposed organisations that were unprepared for such emergencies. At the New Zealand Lotteries Commission’s new office in Mount Eden, that day was doubly significant. “It was the first day we
moved the Auckland staff here and it was the first day of the new chief executive Todd McLeay,” says Quinton Hall, chief technology officer.
“We were probably one of the few places on Khyber Pass Road with power that day,” he recalls. “Everything worked perfectly but for me it was a stressful day.
“It was just a matter of keeping a close watch, watching the network and making sure everything worked as it should. Everything worked perfectly.”
Fortunately, Hall had designed the new centre with high availability and high redundancy to cope with any eventuality. This included two separate battery strings, with two uninterruptible power supplies, two generators, and four fuel tanks. The diesel tanks together hold 2000 litres of fuel, enough to last the centre for up to two days, and contracts are held with suppliers to top up almost instantly.
Further security measures were also taken. The building has its own transformer so Lotteries need not worry about neighbouring power problems.
The front of the building is protected with secure glass. Staff are usually onsite 24x7 and there are cameras everywhere.
Inside are separate computer systems in their own individual cages. Combined biometric and card swipe systems restrict access only to authorised personnel.
Related “environmentals” like power and air conditioning are sourced from outside the cages, so they can be repaired without having engineers going near the systems. The floors also contain glass floor panels for easy checking of contaminants and to ensure the cablers are “kept honest” with layouts looking neat and tidy.
Systems live or on standby
The gaming system features racks of IBM PSeries eservers, with Cisco networking. There is an X and Y system to mirror each other, coupled with a Z system as a back-up.
“When you go into a shop and buy a ticket, we write to three systems before the ticket is printed. All systems are live or on full hot standby,” Hall explains.
Furthermore, in a neighbouring cage is the old Digital server rack containing the old gaming system, which will be kept a year for any old ticket claims.
Another room, the test room, mirrors everything in the gaming room. Next to that is the corporate room, racks of IBM servers running the commission’s new SAP MySap management information system and corporate systems.
This features virtualised servers for email, file and print, with everything connected to a large storage area network as a significant back-up solution for the entire infrastructure.
Furthermore, is a fireproof safe which keeps the storage tapes, which are then taken offsite. Fire protection is strengthened further with gas fire suppression throughout the centre. Finally, in a room where access is strictly controlled, is the random number generator, sealed on all four corners for security, which is audited by Audit New Zealand.
At the adjoining data centre, IT staffers monitor the network, troubleshoot terminal issues from retailers, and manage the IT component of the draws. Here, infrared beams on each side of the data centre can warn Lotteries of anyone trying to gain access. Vibration sensors on the walls can also tell if anyone is present.
“It’s not often as an IT professional you get to build a whole new environment. We have to build for the future. Integrity and security represents our business and is very important,” says Hall.
In December 2003, the commission executive decided to relocate from Wellington to Auckland and give the technology at the Lotteries Commission a complete overhaul.
There were many reasons behind the move. Engineers’ reports said Wellington had too many risks from earthquakes and landslips for the commission to house its primary data centre in the capital.
The commission also wanted to con-solidate its marketing, sales and IT, and Auckland had the office space as well as readily available labour.
However, there is irony in an organisation that leaves earthquake-threatened Wellington for a city with power blackouts. But as an expert in risk management, Hall was prepared for that.
Indeed, after two years of 60 to 70 hour weeks dealing with a complete overhaul of its systems and building up a new IT team, Hall is now looking forward to “enjoying weekends again”.
The new environment
New Zealand Lotteries has carried out at least 10 projects since December 2003, with each effectively following on from the other. These include replacing the gaming system and its 1100 shop terminals, a new SAP management information system, replacing the old DDS retail WAN system with an IP network, installing a VoIP telephony system, upgrading corporate systems, including migrating corporate services to virtualisation, building a new back-up data centre and creating a new IT team in Auckland.
The commission’s website was relaunched to move to a portal-type system in preparation for new services. New terminals were purchased to allow lottery payments using Eftpos.
“There were a lot of decisions. There had been a lack of investment over time. To a certain degree, IT had been left to its own devices and not incorporated into a long-term strategy. We have just come out of the program of work. There were a few sleepless nights along the way.”.
Hall gained responsibility for IT operations, providing strategic direction and centralised management of staff and budgets. His management team consists of Wayne Pickup as IT projects director, whose new position focuses on gaming and innovation projects, with overall responsibility for designing, planning and implementing projects. Gary Safe as IT manager production is responsible for the day to day running of the systems and Nathaniel Pearce as IT manager quality assurance is responsible for the quality assurance and testing of new systems and software.
“I was obviously responsible for the entire program of work. I had to build a team, a project office. I had to find good people, have an over-arching governance role, be involved in contract negotiations, and provide an overview. I have been pretty much hands on across the board,” Hall explains.
The most significant project was the gaming system as that was the foundation of other projects. Hall wrote an RFP, which was a closed process as applicants had to demonstrate experience of running a lottery.
Six suppliers took part and a number of evaluation committees had to review over 10,000 pages of response. Travel overseas to reference sites in the USA, Greece and Germany helped determine the winners.
Working closely with Pickup, a shortlist of three was then cut to two and finally one, with incumbent supplier GTECH kept on for the new system, with them having to define the requirements of the new system.
“Being the incumbent can’t have an influence on the procurement process but it certainly gives them an advantage. They obviously understand our business from the perspective they have. They did an excellent job,” says Hall.
The new system went live on June 5 after a three-month rollout process, which the company called “big whisper” as opposed to a “big bang” of doing it all at once. “An IP network from Telecom and
Gen-i was rolled out early this year and when that was in place we rolled out the new terminals. The new terminals were able to talk to the old gaming system through an emulator, essentially a protocol converter.”
As that progressed, Hall and his IS team of 37 further tested and developed the new system and stabilised the WAN and retail network. “On June 5, after customer acceptance, we switched over from the old system to the new system, which was pretty much a non-event for us. We were pleasantly surprised. The retailers carried on the next day none the wiser. They have had the new terminals for up to two-and-a-half months and the switchover from old to new was seamless, on time and on budget.”
“Considering the complexity of the project we were very happy with its components. If the IP networks had not worked, if the data centre had not been ready... There were so many factors that could have impacted the go-live.”
Hall credits the success of the project on picking good vendors, working closely with them to ensure they had good teams and working with them as partners, not mere suppliers.
“We took a lot of time to talk to our partners GTECH, SAP, IBM, Telecom. We took a lot of time to ensure they appreciated their part in the project and the impact of them not delivering on time.”
Looking back, Hall says one major issue of the project was its size. But Lotteries considered it a corporate project so he was able to pull people out from other departments and create a long-term cross-functional team to help the business deal with the change management processes involved.
The replacement of the proprietary corporate financials system with the MySAP management information system followed a similar procurement process but involved short-listed suppliers to come and meet NZ Lotteries staff who would be using it. That way, they were better able to see what system was needed. A first phase delivered late last year provided financials and procurement functionality. A recently-delivered phase two offers customer relationship management, business intelligence and data warehousing.
The switch to IP was essential as Telecom would no longer support the old network and IP allows easy and remote monitoring with drill down capabilities to individual networks. Thus, the team in Auckland can now remotely monitor and diagnose a terminal, for instance, in Invercargill.
Smaller projects include sales teams being issued with Blackberrys so they can instantly provide marketing feedback from storekeepers to head office.
Last year, Big Wednesday was one of the biggest consumer products New Zealand Lotteries had launched for some years. “For us, we had to go through the processes of developing and testing process changes. It’s a new draw, done differently to that on Saturday, which involved developing its specifications from scratch, adjusting financial systems and linking them with Teletext, TVNZ and NZPA, for publishing results.”
Now, online gaming is on the horizon. The law allows NZ Lotteries to launch a product but Hall won’t say what is forthcoming.
“It is fair to say we are looking at interactive channels. At this point it is too early to say what the strategy will look like but it is important for us to make a move, even if it is just another channel for customers to buy their tickets on. More and more New Zealanders use the internet and we need to move towards serving them online,” he says.
The Lotteries Commission will work closely with the Department of Internal Affairs on future online activities, with further input from Audit New Zealand, though any new services could be up to18 months away.
“We had to have the new gaming system in place for it. SAP, the new gaming system and IP, allows us to look at new channels. We built the system to give us flexibility to integrate new systems using SOA methods. What we have insisted on throughout all these processes is that we have standard industry interfaces. Thus we use components that have the ability to connect. We do the integration ourselves, between the systems.”
Hall says the company’s marketing team is always looking at new projects and enhancements, along with efficiencies around distribution. Thus, his IT department in particular is set up for constant change as change is essential to maintain the business.
“You cannot have the same product for five years. Thus, we have a three-year roadmap of new products, enhancements, initiatives, new products, new channels, new efficiencies.”
Riding to the top
Hall was born-and-bred in Christchurch and started his career in banking as a futures broker and assistant settlements manager for Bankers Trust New Zealand.
However, after five years, he moved from the banking sector to the Department of Internal Affairs in 1995, where he was appointed a risk manager in its casinos division. “I was very interested in the gaming technology aspect, especially risk management.”
In 1999, Hall moved to Australia to be general manager for Global Gaming Systems, an online casino pioneer. “Risk management is certainly a focus (of my career). It has to be top of mind. You always look at what is going to impact on a project, where the roadblocks are, how you are going to pass them,” he says.
Hall also says the risk management focus helps him understand how IT fits together. And having worked at various sides of the IT fence also gives him a meaningful insight into what drives his business partners.
He says New Zealand Lotteries is an exciting place to work. Interestingly, IS plays a role each time a winning number is picked.
“We identify the store, ensure the winners are well-looked after and check and validate the ticket, the security aspects to ensure integrity is maintained,” says Hall.
Regardless of any jackpots, a high profile organisation with a million customers and a nationwide network of retailers means that if anything fails, the IT department is soon alerted.
“From an IT perspective, the IT systems is the business. No systems, no business. These draws are critical, but now things are starting to ease up,” says Hall.
“I’m looking forward to having some weekends, drinking coffee, riding a Vespa. I have had one since I was 17 and have three now. They are not the sort of machines for everyday use. They are like old gaming systems. You treat them gently.”