Employees at retail tax refund specialist Global Refund used to frequently endure multiple bugs from the new off-the-shelf software. According to CIO Waleed Hanafi, his users encountered software features that are documented to be available, but ended up having errors. "We have this large enterprise suite for project management where we are continuously running into bugs," says Hanafi.
This is the case when off-the-shelf software is working on paper, but plenty of bugs emerge when users start exercising the features. What Hanafi realised was that while the products have a huge amount of functions, many simply run only on the surface.
"I think a lot of companies get by with 60 to 70 per cent of their customers never fully using the software they paid for," he adds.
As a solution to prevent his end-users from having to cope with the bugs, Hanafi's IT teams make sure that the applications are thoroughly tested with an extensive programme of acceptance testing through internal pilots. Though time-consuming, Hanafi's team was able to work with the vendor on feature improvements, making steady progress with the suite.
Competing with time
Global Refund is not the only company to encounter such a problem. Driven by the rush to launch applications, developers are often forced to compromise software quality. "Organisations are under increasing pressure and rapid cycles of change on the business front. And enterprises in a bid to be competitive, often perceive testing as a low to no value-add differentiator," says Asheesh Raina, principal research analyst on software, Gartner.
In addition, software quality activity such as application testing is often perceived as a form of "low value-add" exercise. Companies view it as focused on catching mistakes rather than inherently improving functionality, according to Raina.
One common problem with software fault is buffer overflow, according to Hong Kong Productivity Council's principal consultant Roy Ko, who promotes software process improvement. He says buffer overflow occurs when extra information spills out of the capacity of data fields, corrupting the software.
This problem can be averted if testing on exceptional scenarios are carried out but, "you have to spend a lot of time designing different test scenarios", he says. Developers end up conducting only one test to ensure it works in normal conditions.
This results in bug-ridden applications that not only upset business processes, but there are also strategic implications such as the credibility of the company.
"In a few years, IT organisations and service providers will be distinguished by their quality capabilities," says Raina.
Their position as the top-tier providers will depend substantially on their process and service management. "Failure in that area will lead to a downtrend," she says.
Also, the cost of fixing a software bug after the project has been delivered is more than doing it during the development stages. "Your user reports an error on the system after it was delivered and you have to trace it and test it, go through the whole development cycle again," says Ko.
The challenge for developers to ensure software quality, however, is often the lack of resources.
"Either they don't have a lot of people to work with or they do not have enough time," he adds. He attributes this to the pressure of having to meet business objectives on time. "By meeting tight deadlines, you have to sacrifice something. You either have to use more people to help out in the project or you have to sacrifice the quality of the project."
Nevertheless, companies' awareness of software quality has been encouraging with the growing expenditure on software testing systems in recent times, says Raina.
Spending on testing software has been comparatively higher than in previous years. According to Gartner's data, the market for testing software was US$69 million in 2006. With an annual growth rate of 18.2 per cent, the market is expected to reach US$159 million by 2011.
Today, awareness among CIOs on the importance of software quality has been improving.
"This is primarily because of the software deployment maturity of these companies. They are beginning to see a lot of efficiency that can be developed at the application deployment level," says Wilvin Chee, research director for software research in Asia Pacific in IDC.
The role of the CIO
On top of spending on testing applications, the quality of software remains the responsibility of the CIO. The ability to balance business users' demands on time to market and software quality is very much like walking a tightrope.
"People always want things fast and cheap," says Hanafi. The consequences, which come much later, are constant update and maintenance. "Whatever happens in this year's project is next year's maintenance."
To prevent such situations, he says, CIOs should step in to minimise shortcuts in the software development process, but, at the same time, ensure business requirements are met.
However, there is no silver bullet in managing the fine line between business users' requirements and IT development. "I wish I had a single prescription for how to handle it," says Hanafi. The CIO has to persuade the business leader to understand the importance of raising their funding for development, or to tolerate delays to ensure the quality is good enough.
Standards make perfect
Meanwhile, CIOs should also take a leading role in formulating software quality policy by setting the development process and minimum tolerance levels.
"We have to ensure that all the processes and standards are complied with to ensure quality, because defects detected at the later stages will be more costly to retrofit," says Daniel Lai, head of IT at Hong Kong's train transport provider MTR. "That's why we have the philosophy of building in quality from the start till the end, even after it [the software] goes live."
Poor software quality could result in a miscalculation in the automatic fare collection system and MTR's revenue. Lai notes the miscalculation could also damage passengers' satisfaction and experience, damaging the organisation's reputation.
The company thus places a high priority on entrenching software quality throughout the stages of software development. The project manager is accountable for the ultimate quality of the software. But a quality manager is also assigned to work hand-in-hand with the project manager to set application standards. The quality managers are expected to review the software based on these standards throughout the development process.
MTR sets three levels of testing to ensure the functionality and performance of the application is up to standard "First, we have unit tests conducted by the developer. We also have system integration tests led by the project team leader and project manager. At the final stage, we have the user acceptance test," says Lai.
In addition to the functionalities at the final deliverable stage, the review also covers the intellectual property (IP) status. It aims to ensure there is no IP infringement or make sure the MTR's IP is protected.
In addition, the review also audits the software development process to ensure the developers follow the standard procedures.MTR has an offshore software development centre, where compliance checks are performed. The centre also collects matrices for measuring the number of faults at different development stages. These matrices, according to Lai, aim to identify a trend and detect faults as early as possible to prevent the additional cost in fixing bugs at the later stage of development.
"On top of these, we have adopted a lot of standardised processes," says Lai. "We are certified on ISO 9001 and CMM Level 2 and so is our offshore team. All these help ensure software quality."
Users take responsibility
At Global Refund, Hanafi set up a pilot team for testing software quality as well as collecting business process requirements from end-users. In each project, a business user is assigned a champion role in driving the project forward. The user is assigned from the department with functions that are related to the project. "For [implementing] the project management suite, the role was assigned to an individual with a background on project management and also financial analysis," says Hanafi. "In another project, the head of the client's support is the centre of excellence for the service desk's software."
Even after the software has been implemented, the person is still responsible for end-user education and modifications to meet with changing business requirements.
To ensure the quality of off-the-shelf software, Hanafi's teams also conducted software pass quality tests. "We actually go through the full build process [of testings] until we can demonstrate that it passes quality assurance on a functional software basis," he says.
Most importantly, it is about ensuring the software that the organisation will be using is fully functional and error-free. "I would rather spend more time upfront getting it to work properly rather than prematurely unleashing it," says Hanafi. "This would avoid a huge backlash from users if it does not work."
© Fairfax Business Media
Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.