ROI: Reasons why bashing it isn’t enough

ROI: Reasons why bashing it isn’t enough

MIS asks IT bosses to put the return on investment metric into business perspective.

Only a couple of years ago, return on investment (ROI) was hailed at management level as the way to bring IT to heel in the aftermath of the dotcom crash. Many IT directors welcomed this reinforcement of financial accountability. It brought project planning power, which was being hijacked by business managers, back to IT departments.

But the idea of determining IT expenditure through ROI was doomed from the outset. ROI calculations require assumptions, and could never be the be-all and end-all of complex decisions with human implications. Attempts to use it as a tool across just about every IT scenario created many misgivings.

But today there’s a consensus among IT practitioners. ROI may be a policy requirement, or needed for specific situations, but the driving decision-making approach is to minimise its importance, and promote the primacy of other factors.

Globally in the 1990s, IT understood and acknowledged its own cost was too high. And it took on board that, so as to deliver real productivity increases, IT had to be cut back and controlled.

And cut it did, using new technology to slash the cost of management and service delivery, outsource non-core functions, implement new procurement practices, go server-centric (again) plus a range of other strategies.

But, in the face of tougher economic times and in the spirit of IT scepticism after the dotcom collapse, CEOs and CFOs demanded more. The metrics IT instigated were increasingly co-opted by a troop of other executives intent on implementing them across the board, applying them to both strategic and infrastructure investment to “bring IT under control”.

In some cases, the dependence on metrics got out of control, was demanded in quite inappropriate areas, and elaborate return on investment (ROI) models were built for projects that simply had to be done to ensure continued service delivery. Projects had to be documented, justified and analysed like never before.

But, while IT directors accepted accountability a long time ago, they object to being measured inappropriately. And in the smaller firms found locally, it appears common sense has generally prevailed.

For this special report, MIS interviewed the following IT directors for their insights on ROI:

Neil Andrew, chief technology officer, TVNZ;

Tony Darby, director of information services, Auckland Regional Council;

Iain Graham, IT manager, Contact Energy;

Egon Guttke, group manager IT, Foodstuffs;

Billy Michels, information services manager, South Waikato District Council;

Barry Miller, managing director, Information Technology Systems.

IT bosses interviewed by MIS talk of working with management teams to keep ROI in perspective; to gain the involvement and support of business managers; and ensure buy-in on longer-term strategic-related investments.

IT managers talk fervently about the other reasons for IT decision-making that revolve around less tangible, but no less important, benefits. To build a bulwark of policies and processes providing a documented and broad justification for IT projects.

They factor in the strategy risks that different IT approaches can create, and they point to the manipulability of financial figures, where ROI can be distorted to justify projects.

IT bosses spoken to by MIS agree that ROI is far too narrow a consideration for IT spending but differ on what weight it deserves.

At one extreme is the feeling that if ROI isn’t obvious and simple to calculate, the project is too marginal to consider.

At the other is a love of the process amongst those who see it as a useful check. In the middle are regimes where ROI is a requirement of any IT decision, regardless of the suitability.

No one thinks it is the be-all and end-all of IT

Meeting mandatory requirements

There is little point in doing complex analysis of IT projects if you are going to have to do them regardless.

Regulation or changes to external infrastructure (such as telecommunications), or as a requirement dictated from elsewhere within the firm can result in an unnecessary doubling of effort.

Egon Guttke, group manager IT for Foodstuffs, says ROI is an important tool but is of no use in mandatory circumstances.

He points to the external example of the introduction of food labelling requirements. “That was introduced by the government and we just have to comply. Or if ETSL [an Eftpos network] changes its standards for something, so you have to think about how you meet those requirements.”

Contact energy IT manager Iain Graham points to an internal example.

“For example, we’ll get warnings that our sales and marketing retail arm are going to try and acquire customers. We’ll make an investment to ensure that is possible.”

Non-financial business goals

Many firms prioritise service delivery over strict financial considerations.

Graham says the first thing Contact Energy considers in any IT investment is “how does it support our corporate objective”. Then it looks at the other factors.

“If you first align to corporate goals, that can exclude certain projects without a great deal of consideration.”

He says the firm uses the same project management approach across IT as it does for other work such as investing in power stations.

“Some things we want to build an ROI calculation on are intangible, so you need to be flexible. Our process tends to take a broad look at the investment.”

State-run TVNZ now has a charter. Chief technology officer Neil Andrew says some decisions may not meet an ROI hurdle but may well deliver to the charter.

“Quite often, you’ll have businesses or non- profit organisations where the ROI is hard to use except for comparison of alternatives,” he says.

Andrew says TVNZ is tough on investment decisions but finance is not the gatekeeper. The CAPEX committee, including finance alongside other management, undertakes that role.

The need to factor-in risk

It can be difficult to take account of the risk in a technology decision if it has to be expressed as numbers.

TVNZ’s Andrew says, done properly, ROI is powerful because you can start to assess some of the risk, but TVNZ then uses ‘scenario modelling’ to compare different options. This allows the firm to test the sensitivity of assumptions that are being made.

For example, “If we had a 20 per cent overrun on budget how would that effect the ROI? And what is the worst case on the benefits?” Or for a project that is marginal: “You can look at sensitivity – at what point would it be worth investing? Sometimes, we’ve been able to go back to suppliers and say, ‘If the pricing was at this level we could justify it.’”

TVNZ recently managed to negotiate a more suitable deal on a storage area network by using this technique, he said.

Strategy takes priority

An assessment of ROI won’t tell you if the project is what is really needed in the long term by the market or the community the organisation serves.

Billy Michels, information services manager, South Waikato District Council (SWDC), says he’s had questions from his management team, who ask, “Where are the savings from spending on IT?”

“The reality is that any system we put in might save only minutes here and there. Finding big gains in ROI is getting harder and harder.”

Luckily for Michels, ROI is not a necessity at the council, which will consider projects supporting its strategic vision.

“When we put in internet access into the libraries it was an investment, but was seen as a service to the public.”

In another case, the council changed network infrastructure on the basis it was just too hard to manage, says Michels.

He says there were advantages to the council standardising its operating systems.

“I did some cost comparisons and the costs of changing the system were not much different from continuing with what we had.”

The benefits can’t be quantified

Not everything can be counted, despite the fad of accountability.

Director of information services for Auckland Regional Council (ARC), Tony Darby, asks: “How much quality is enough? How much customer service is enough? They are intangible components, which you and I can agree on as a customer, but how would you attribute a dollar value to it?”

ARC uses a methodology internally for its IT decisions which it calls the ‘Effective Design Process’.

“It has a whole lot of things you need to input into your decisions to change processes or implement systems. The ROI is one component of that, because we also build into account the design component, the customer component, and the innovation component.

“ROI is too narrow and can’t hope to take into account every variable or service improvement. If you did go to that level of detail you’d never start the project. There has got to be a line.”

Darby also points to the inappropriateness of something like ROI for basic productivity packages.

“How would you do an ROI for the word processing package? When you’re doing significant systems acquisition it’s fine, but how do you evaluate the ongoing benefit of using it efficiently?”

He says ROI is one of the tools you can use, but you have to be able to balance it with common sense.

Basic IT is well proven

Quantifying efficiency and productivity is notoriously difficult, especially amongst white-collar workers and management.

A lot of basic IT infrastructure and usage has come to be understood to be ‘worthwhile’ or simply essential to be in the game, in spite of difficulties quantifying it within the business.

Barry Miller is managing director of Information Technology Systems (ITS), an outsourcer of system administration and IT management, effectively acting as an IT management team for a number of firms.

Miller says IT has become a basic necessity and it’s no longer about seeking advantage. The biggest drivers for IT decision making are productivity and reliability and reducing expenses, rather than an ROI calculation.

“In a sense it is ROI. But ROI is not measured in dollar figures, it’s measured in productivity, reliability, getting things done on time, customer service.”

Miller says the requirement for ROI came from higher up. “From the finance side, they hated seeing this thing they couldn’t control.”

Now the industry is matured and understands the basics of IT better.

“They’re starting to take a much broader view and to realise that there are things that have just got to be done. The smaller companies look a bit closer, but bigger companies see this is the tool they need and that is all there is to it. There are tools they know they have to have. They need to be reliable and you have to maintain them.”

You can say anything you want with ROI

Figures can be adjusted to deliver the results you want.

Often, the only people able to prepare an ROI business case are the same people who are either advocating or opposing the project internally; therefore, the analysis often lacks independence.

Many business cases are built on a series of assumptions, and minor adjustments to these can have major impacts on the bottom line. Many of these assumptions cannot be properly tested before commencement. ROI analysis provides the impression of precision, but often this is false precision.

“Most people know you can come up with a number with attractive returns on investment. You can calculate different results depending on assumptions. I wouldn’t believe every ROI figure because it comes off a spreadsheet. You can manipulate those returns, and so you have to be careful as to what you produce and what you believe,” says Foodstuffs’ Guttke.

It is all meaningless!

The causes of IT project failure often have nothing to do with whether a sound business case has been developed.

Lack of executive sponsorship is one common cause of failure. Integration issues are another and user resistance another.

So even if you complete a full ROI analysis, you then have to temper those results with a realistic look at the threats the project faces in your organisation.

For some types of projects, like CRM, the benefits of the system cannot be measured in normal accounting terms. In other words, the CFO may not be able to see the return coming through on their periodic statements or may not be able to clearly link improved performance to the project. For this reason, organisations have to come up with proxy measures for success, such as customer value.

And while many commentators talk about the need for a consistent ROI methodology across the organisation, different types of projects often demand different kinds of analysis and produce different benefits – hard, soft, quantitative and qualitative.

To impose a template across all projects, especially in a complex organisation, is to ignore these differences. But the alternative – to allow many ROI/business case variants – makes project comparison and evaluation fraught.

Where ROI shines

ROI is a useful tool, provided it is used correctly – that was the general feeling from our interviewees.

Contact energy IT manager Iain Graham says business is an art and you intuitively seek value. ROI is a useful tool for testing your assumptions. “I love the tools, because they help me hone my skills. I like the fact that business is intuitive, but then you can apply some discipline to measure yourself and help yourself improve.”

Auckland Regional Council’s director of IS, Tony Darby, says they’ll run ROI analysis over anything significant they do. “It’s done later on in the process. We like to understand where the payback areas are going to be first and then put the numbers behind them.”

He says a focus on ROI is not a bad thing. “You’ve got to be financially prudent, although it’s only part of the whole scenario. You have to look at the state of the industry. When things get lean, people tend to focus on the bottom line. It’s not a particularly nice time in the market, and I guess the ROI argument is being flamed by there being limited money.”

Comparing competitive options is regularly touted as an appropriate use for ROI, including where the option is to forego using IT. TVNZ chief technology officer Neil Andrew says ROI comes into its own when comparing different vendor options for a specific project.

“One costs more and we look at the lowest cost option that fits and then turn around and say that anything over that price has to be justified on an ROI.”

And ROI can also provide a check on replacement policy, he said. “I have had situations where ROI has shown it’s worth replacing just for the support savings,” says Andrew.

Darby says ROI is simply a part of the way you do things and is one of the tools you use to demonstrate you’re actually delivering value for money.

“The latest one that we did is quite a large outsourced arrangement, and we had to look at all of the options of doing it in-house or doing it as outsourced arrangement. Where a lot of organisations come unstuck is they don’t go back to their original justification to see if they did deliver that return.”

Richard Wood

Towards better ROI

MIS looks at the case for honing your return on investment analysis and methodologies.

Return on investment analysis is an important tool in any manager’s kit, but often that tool is blunt due to inappropriate use, lack of follow-up analysis or the application of incorrect assumptions. That doesn’t mean you throw it out – it can be sharpened and made more effective.

Nevertheless, for one ROI apostle, “The only metrics that matter in a business are ROI metrics”.

Mark Heath, managing director of management consultancy MBH Management, says dissatisfaction with ROI is caused by a lack of understanding about what metrics to use – even among CFOs. The result, he says, is people being held accountable for fundamentally unrealistic business cases.

Heath says standard project management disciplines do not incorporate the management of benefits realisation. Benefits management needs to be built into the process and updated regularly.

Lack of CFO understanding of the correct methods of ROI measurement and management, he says, can lead to other executives, such as the chief information officer, being left high and dry when they seek advice and guidance. They are often told to “go away and make it work”.

Methodology matters

Within any ROI model there are several different standard components and a few that are more theoretical. Choosing the right methodologies to apply is important. Internal rate of return and net present value (NPV) are standard.

The first measures the return expected from an IT investment compared with what could be achieved from simply investing the money elsewhere. NPV uses a discount rate, often higher than the rate used above due to project risk to rate different projects. The formula: NPV = Cost + Benefits in year one / (1 + discount rate) + benefits in year two, and so forth.

There are other more arcane models such as economic value added analysis, EVA, which strives to calculate the economic value of an investment by taking all capital costs into account. The idea is to value an investment the way an external shareholder would – by the value it adds to the business. The formula: EVA=Net operating profit after tax – (capital x cost of capital). Read more at:

Heath, along with backing NPV strongly, also offers another approach; option pricing, a method adopted from financial markets in the early 1990s to cater for and evaluate multiple future investments. (It appears the jury remains out on this method’s value.




The value of ROI analysis

To some degree, your choice of the right model will be determined by the philosophy of your organisation. Some, for instance, have deployed EVA and associated bonus concepts widely. If this is the case, you will probably have to think in those terms as that is your business environment.

On a more fundamental level, the value of ROI analysis can be increased but it requires application and sometimes courage.

Follow up

MIS’s Australasian survey confirms overseas results showing business cases and ROI analyses are inadequately followed up. It is vital that project performance is tracked post implementation.

Are the promised returns being realised? If not, why not? How can this be fixed? What can you learn from this experience to plug back into your ROI model to make it better? It’s about accountability and you, the IT director, can lead from the front.

Discount the benefits

Soft benefits such as increased customer satisfaction and productivity gains need to be viewed with some scepticism in any ROI model. In your analysis this is a risk and, when a risk factor is introduced, you need to reflect that in your model.

Discount these benefits by an appropriate percentage when you plug them in – and make sure you flag this and the reasons for it in your report. This is an area where a lot of ROI fudging happens. The figures don’t work out – change them. You have to fight this, from yourself and from other executives.


If the project you are promoting is simply necessary – is simply the cost of staying in business – resist ROI. Determine what you need to do and what the desired outcome is and then seek to achieve that for the minimum cost to your organisation. You may still have a battle on your hands with the CEO or CFO.

If you believe you are being forced to use ROI inappropriately, make that clear, and present them with the best numbers you can. But if you feel the analysis is bogus, say so. Inappropriate use of ROI is one of the major reasons why so much of this analysis is bad analysis and why so many experienced managers have grown to distrust it.

Put it in context

ROI is one thing, but how does a particular project fit into overall IT and business strategy? What are the interfaces and impacts of this project with other projects that may already be underway? How does it deliver on corporate objectives as defined, say, in your mission statement? Is there a way you can quantify these ‘mission’ benefits and plug them into your ROI analysis?</ol>

Achieving good ROI analysis is about applying high levels of intellectual honesty and rigour. There is an aspect of corporate responsibility here. At times, intellectual honesty may not be what your organisation is after.

It gets political and it gets difficult, but you have to continuously defend and extend your own credibility as a manager and try and take the high ground. That means embracing ROI, when it is appropriate, improving your ROI methodologies at every opportunity and being very clear (and calm) about why it shouldn’t be used if you feel it is inappropriate.

For more information and links, see

Join the CIO New Zealand group on LinkedIn. The group is open to CIOs, IT Directors, COOs, CTOs and senior IT managers.

Join the newsletter!


Sign up to gain exclusive access to email subscriptions, event invitations, competitions, giveaways, and much more.

Membership is free, and your security and privacy remain protected. View our privacy policy before signing up.

Error: Please check your email address.
Show Comments