So What are You Measuring?And is the information really worth anything? ITSM Watch columnist George Spafford of Pepperweed Consulting weighs in.
Metrics are great. We all know that we should be measuring things and generating reports with pretty charts right? There are numerous books on the topic of metrics for IT, tools that can generate reports with graphs and so on. The problem is that once all of that work is done and the reports are created, are they really worth anything?
One of my favorite stories is of a consultant working with an organization who questioned the need for a thick management report produced each month on high quality paper, bound and presented to management. There were several people working to compile the report and then the costs of printing, binding and so on. It definitely looked impressive and had a cost to match.
Nobody who produced the report was sure if the intended audience of managers were actually reviewing it. To subtly answer the question, they decided to stop publishing the report and see if anyone spoke up. After several weeks they received an inquiry from a senior manager about the report. Of course, everyone was excited that someone had missed the report because surely it must be adding value and they had not been wasting their time and the organizations money.
They set up a meeting with the senior manager and asked him how he used the report. He sheepishly admitted that he didnt read the report but his daughter really enjoyed drawing on the fancy paper. Needless to say, they stopped printing the report after that.
Its a great story and where I originally encountered it I no longer remember but the message is very importantproduce reports to meet the requirements of stakeholders, not because you can.
When a manager or executive reads a report, they need to have the information they require to make concise decisions. These people are typically very pressed for time so long reports with answers hidden somewhere in the pages are less likely to be read or even resented. However, a focused report that is just a page or two long that has the proper analysis and presentation is a welcome relief.
What to Measure
Asking managers for requirements can be a challenge and people are sometimes asked to recommend metrics. What is useful to bear in mind is that process adoption tends to follow a lifecycle. Metrics arent just for reporting as they drive behavior as well. Once employees understand how they are being measured, they typically adjust their activities accordingly.
When a process is first introduced, it helps to track metrics that demonstrate compliance to the process. What you are seeking to understand and influence is adoption of the new process. Metrics might revolve around process exceptions detected during audits, detective controls in production that demonstrate processes were not followed, etc.
Another class of metrics to look at early on relate to effectiveness. In other words, are the outcomes of the process what were expected? If a process is supposed to increase availability, then is it really doing that? The metrics here revolve around the desired outcome. Look for metrics that are meaningful not just to IT but the business as well. A good example is availability. This is an opportunity to look not just at uptime percentages but also sales lost due to unplanned downtime, and so on.
As the process adoption increases and matures, then efficiency becomes more relevant. In this case we are looking at the ratio of inputs to outputs. A good example is cost versus benefit. Relative to the costs we are incurring for the process, are the benefits worthwhile?
As the process stabilizes economy enters in. In this case, we are measuring the costs over time and looking for trends. We may also compare the costs of the current solution to other potential solutions to determine if there is a more economical means of achieving an objective.