http://www.itsmwatch.com/itil/article.php/3660826/There-is-No-Evidence-for-ITIL.htm
Back to Article
|
|
|
|
By The IT Skeptic Feb 20, 2007 Where is the evidence for the benefits of ITIL? There isn’t any. Not the kind of hard empirical evidence that would stand up in, say, clinical trials. There is more evidence for quack alternative medicines than there is for ITIL. Granted there is some research around the benefits of aligning IT with the business but not around quantification of ROI and nothing that I can find specific to ITIL.
The itSMF themselves make a few outrageously unsubstantiated claims in their pocketbook An Introductory Overview of ITIL:
Send no money now!! Before it mysteriously disappeared, the Best Practice Research Unit (BPRU) was a website claiming to be associated with the ITIL v3 Refresh (After twenty years of ITIL, it is high time there was such a unit.) It is a shame there is no such initiative from either OGC or itSMF (at least itSMF USA is doing something, in fact several things focused on research). The BPRU website explicitly recognised the evidence problem: "Much of the material published on IT management, including IT service management, has been normative or prescriptive in flavour. Few rigorous, academic studies have been undertaken to evaluate how tools, techniques, methods and management approaches have been selected, adapted, implemented and measurable benefits achieved. There is a danger that new approaches arise out of the practitioner community with little empirical validation." “Few rigorous, academic studies … ?” Don’t be kind. The solitary piece of academic research I can find carries a bold and, I think, unproven title Evidence that use of the ITIL framework is effective . It opens by saying “Very little academic material exists on ICT Service Management Best Practice …” and concludes its own research with: "We found that both customer satisfaction and operational performance improve as the activities in the ITIL framework increases. Increased use of the ITIL framework is therefore likely to result in improvements to customer satisfaction and operational performance. Although the study was limited to a single research site, claims made by executive management of the research site and OCG as to the contribution the ITIL framework seems to be justified. More definitive research delineating the nature of these “relationships” is however needed, especially regarding each process in the ITIL framework." The database is poor: “… research site was a large service unit of ICT in a provincial government in South Africa during 2002/3” — one site. More importantly, the two things measured to support this brave conclusion were (1) customer satisfaction (the three surveys they conducted only included management in the final survey so all we can say is that non-managerial staff were happier) and (2) “objective service improvement” by measuring “the number of calls logged at the Help Desk” because “we can rather safely conclude that the number of problems logged would be a good reflection of objective service levels." Granted there is some research around the benefits of aligning IT with the business but not around quantification of ROI and nothing that I can find specific to ITIL.
The itSMF themselves make a few outrageously unsubstantiated claims in their pocketbook An Introductory Overview of ITIL:
Send no money now!! Before it mysteriously disappeared, the Best Practice Research Unit (BPRU) was a website claiming to be associated with the ITIL v3 Refresh (After twenty years of ITIL, it is high time there was such a unit.) It is a shame there is no such initiative from either OGC or itSMF (at least itSMF USA is doing something, in fact several things focused on research). The BPRU website explicitly recognised the evidence problem: "Much of the material published on IT management, including IT service management, has been normative or prescriptive in flavour. Few rigorous, academic studies have been undertaken to evaluate how tools, techniques, methods and management approaches have been selected, adapted, implemented and measurable benefits achieved. There is a danger that new approaches arise out of the practitioner community with little empirical validation." “Few rigorous, academic studies … ?” Don’t be kind. The solitary piece of academic research I can find carries a bold and, I think, unproven title Evidence that use of the ITIL framework is effective . It opens by saying “Very little academic material exists on ICT Service Management Best Practice …” and concludes its own research with: "We found that both customer satisfaction and operational performance improve as the activities in the ITIL framework increases. Increased use of the ITIL framework is therefore likely to result in improvements to customer satisfaction and operational performance. Although the study was limited to a single research site, claims made by executive management of the research site and OCG as to the contribution the ITIL framework seems to be justified. More definitive research delineating the nature of these “relationships” is however needed, especially regarding each process in the ITIL framework." The database is poor: “… research site was a large service unit of ICT in a provincial government in South Africa during 2002/3” — one site. More importantly, the two things measured to support this brave conclusion were (1) customer satisfaction (the three surveys they conducted only included management in the final survey so all we can say is that non-managerial staff were happier) and (2) “objective service improvement” by measuring “the number of calls logged at the Help Desk” because “we can rather safely conclude that the number of problems logged would be a good reflection of objective service levels." Where is the evidence for the benefits of ITIL? There isn’t any. Not the kind of hard empirical evidence that would stand up in, say, clinical trials. There is more evidence for quack alternative medicines than there is for ITIL. Granted there is some research around the benefits of aligning IT with the business but not around quantification of ROI and nothing that I can find specific to ITIL.
The itSMF themselves make a few outrageously unsubstantiated claims in their pocketbook An Introductory Overview of ITIL:
Send no money now!! Before it mysteriously disappeared, the Best Practice Research Unit (BPRU) was a website claiming to be associated with the ITIL v3 Refresh (After twenty years of ITIL, it is high time there was such a unit.) It is a shame there is no such initiative from either OGC or itSMF (at least itSMF USA is doing something, in fact several things focused on research). The BPRU website explicitly recognised the evidence problem: "Much of the material published on IT management, including IT service management, has been normative or prescriptive in flavour. Few rigorous, academic studies have been undertaken to evaluate how tools, techniques, methods and management approaches have been selected, adapted, implemented and measurable benefits achieved. There is a danger that new approaches arise out of the practitioner community with little empirical validation." “Few rigorous, academic studies … ?” Don’t be kind. The solitary piece of academic research I can find carries a bold and, I think, unproven title Evidence that use of the ITIL framework is effective . It opens by saying “Very little academic material exists on ICT Service Management Best Practice …” and concludes its own research with: "We found that both customer satisfaction and operational performance improve as the activities in the ITIL framework increases. Increased use of the ITIL framework is therefore likely to result in improvements to customer satisfaction and operational performance. Although the study was limited to a single research site, claims made by executive management of the research site and OCG as to the contribution the ITIL framework seems to be justified. More definitive research delineating the nature of these “relationships” is however needed, especially regarding each process in the ITIL framework." The database is poor: “… research site was a large service unit of ICT in a provincial government in South Africa during 2002/3” — one site. More importantly, the two things measured to support this brave conclusion were (1) customer satisfaction (the three surveys they conducted only included management in the final survey so all we can say is that non-managerial staff were happier) and (2) “objective service improvement” by measuring “the number of calls logged at the Help Desk” because “we can rather safely conclude that the number of problems logged would be a good reflection of objective service levels." I expect this last statement leaves this research with zero credibility with anyone who understands ITIL and ITSM. No cost/benefit analysis. Not a single valid objective metric. Sure if you throw enough government money at anything and launch an aggressive enough PR campaign you can make the users happier. That proves nothing. And the fact that calls to the service desk went down over an initial nine-month period would to me be a cause for concern not celebration. But BE WARNED: this paper will be quoted all over the place as evidence of the effectiveness of ITIL. Pink Elephant have finally extended the number of anecdotal stories beyond the tired old Proctor and Gamble, Caterpillar and the internationally famous Ontario Justice Enterprise. They now have a few more unsubstantiated statistics, this time for PEMCO, Zurich Life, Hospital Corporation of America, Nationwide Insurance, Ontario Ministry of Transportation (what is it with Ontario?), and Capital One. I’ve been in vendorland and I have generated this kind of case study. These amount to no more than selective quotes from managers justifying their decision after the fact. HP is one vendor putting numbers where their hype is, though this pertains to a service desk product not ITIL: "IDC found that, for the companies surveyed, IT productivity increased by an average of 14% … contributing an average cost savings over three years of almost $4.2 million annually. When normalized for company size, these savings amounted to $17,235 per 100 users … Based on these savings, the three-year hard ROI … averaged 130%, giving an average payback period of 13.5 months." University researchers are you listening? Those are results. That’s what we want to see. Unfortunately, paid analysts doing surveys for a vendor are about the least useful sort of research. It is not subject to the same transparency, peer review or statistical rigour as academic research, and the results tend to be selectively quoted by the vendor. But since we’ve started, here is another analyst survey: “Did you make a business case before decision? (Based on 62 European firms): No - 68%” Fully two thirds had no business case! But wait. There’s more: “Did you observe the expected ROI? (Based on 20 European firms). No - 50%; Don’t know - 30%; Yes - 20%.” If less than a third built a business case, one would guess the ones that did build one represented a sample biased towards those who had a good case, and yet only one fifth of them achieved the expected ROI. This is best business practice? I think I need to go lie down. Before I do though, here is more. If you are willing to make major decisions based on informal research by vendors (as everyone adopting ITIL does), ponder this: In a survey carried out by Noel Bruton of 400 sites, about half of the 125 organizations which were found to have adopted ITIL made no measured improvement in terms of their service performance or the rate at which they were able to close helpdesk calls. Finally, the piece de resistance from none other than the itSMF USA, earlier this year: "Compass then asked the companies how well they actually measure their ITIL process maturity. Only 4 percent of respondents felt able to say that all of their ITIL processes were fully measured for maturity and fewer than one third of respondents had maturity measures for all ITIL processes. Compass also asked people to define how well their organization is able to relate process maturity to performance improvement based on measurement. Only 9 percent of respondents felt able to say that the relationship was based on full measures, fully linking process maturity with performance." A staggering 72% felt unable to acknowledge any linkage at all between process maturity and performance improvement. How on earth do they get the money? And how do their managers keep their jobs? Not only do we have no rigorous evidence for ITIL, but the apocryphal and casual data we do have indicates there is real cause for doubt and hence a crying need for objective investigation. The IT Skeptic would like to see some solid scientific research on: The delicious irony in all this is ITIL’s own emphasis on the importance of a business case and ROI. But the fact is few organisations even bother to examine the business case before embarking on ITIL; even fewer measure results; and the few that do are building their business case in the absence of any solid research to justify their estimates, and in the face of conflicting informal evidence. The IT Skeptic is an ITIL professional and active itSMF member who, for obvious reasons, prefers to remain anonymous. More thoughts from the IT Skeptic can be found at IT Skeptic.org. |
