I recently sat the ITIL v3 Foundation exam. Despite expounding about ITIL for years and building a pretty good following on my websitehttp://www.itskeptic.org/, I haven’t certified in ITIL since my v2 Foundation exam so many years ago. I argued that I would certify when the lack of it prevented me getting work. Well, it is 2009 and times are tough and off I went to Prometric to sit v3 Foundation. (I can’t imagine any recession persuading me to invest what ITIL v3 Expert demands, but that is another article).

Before sitting the exam, I ran through a number of practice exams available on the Web (personally I liked ITIL Campus and Taruu). I’ve been critical of the concept of multi-choice testing all along, and we’ve pulled apart a few examples on my blog, but as I was doing question after question on the practice exams (and the final real exam) it struck me how bad most of them were.

The Foundation exams should test an understanding of the fundamentals of ITSM, and a grasp of the key principles of ITIL. If someone has passed Foundation that should tell us they understand IT is all about delivery of services; that a catalogue defines what we do in users’ terms; that modifications to everything that impacts a service should go through Change; that an Incident is distinct from a Problem; that if what our suppliers promise doesn’t line up with we promise the business, we are screwed, etc., etc.

Who gives a rodent’s posterior exactly how many steps there are in the ITIL CSI process? (Especially since that process is different from every best practice CSI process already in existence―that too is another article), or what the “seven Rs” of Change are for? More precisely, why do candidates need to know these things by heart? These are not things that trainees should have absorbed as learned principles, they are things they can look up if they need to. Many of the questions test whether the candidate has memorised the sacred books rather than whether they have a useful knowledge of ITSM/ITIL.

In my satirical book Introduction to Real ITSM, I made fun of these exams with a little exam of my own. I’ll reproduce a couple of the Real ITSM questions as we go along to illustrate my points. We can start with:

1) On page 49 of Real ITSM what is the first word on line 10 (counting page headers as lines as well) a) the b) and c) This d) By

Testing Multiple Choice Skills

The second group of ITIL Foundation questions that really tick me off are the ones that test the candidate’s logic skills.

The satirical Real ITSM question that spoofed the logic tests was:

2) Which of the following is not the opposite of not being unresponding to a user’s failure to call the Service Desk?

a) not calling the user

b) not failing to call the user

c) not calling not the user

d) not the opposite of not calling someone other than the user

If that seems over the top, consider this question (reportedly from the official “mock” exams but we can’t say for sure as the mock exams to the best of my knowledge are not published – True!):

Which of the following are not included in Access Management?

1) Verifying the identity of users requesting access to services

2) Setting the rights or privileges of systems to allow access to authorised users

3) Defining security policies for system access

4) Monitoring the availability of systems that users should have access to

a) 2 and 4 only b) 1 and 4 only c) 1 and 3 only d) 2 and 3 only

Reportedly one of the live exam questions looks something like this:

Order the following CSI implementation steps in the correct order of the PDCA (Plan-Do-Check-Act) model:

1) Assign roles and responsibilities to work on CSI activities

2) Measure and review CSI plan objectives are being met

3) Identify CSI requirements, objectives and scope

4) Implement CSI enhancement

a) 1-3-2-4 b) 2-3-4-1 c) 3-1-2-4 d) 3-4-2-1

I recently sat the ITIL v3 Foundation exam. Despite expounding about ITIL for years and building a pretty good following on my websitehttp://www.itskeptic.org/, I haven’t certified in ITIL since my v2 Foundation exam so many years ago. I argued that I would certify when the lack of it prevented me getting work. Well, it is 2009 and times are tough and off I went to Prometric to sit v3 Foundation. (I can’t imagine any recession persuading me to invest what ITIL v3 Expert demands, but that is another article).

Before sitting the exam, I ran through a number of practice exams available on the Web (personally I liked ITIL Campus and Taruu). I’ve been critical of the concept of multi-choice testing all along, and we’ve pulled apart a few examples on my blog, but as I was doing question after question on the practice exams (and the final real exam) it struck me how bad most of them were.

The Foundation exams should test an understanding of the fundamentals of ITSM, and a grasp of the key principles of ITIL. If someone has passed Foundation that should tell us they understand IT is all about delivery of services; that a catalogue defines what we do in users’ terms; that modifications to everything that impacts a service should go through Change; that an Incident is distinct from a Problem; that if what our suppliers promise doesn’t line up with we promise the business, we are screwed, etc., etc.

Who gives a rodent’s posterior exactly how many steps there are in the ITIL CSI process? (Especially since that process is different from every best practice CSI process already in existence―that too is another article), or what the “seven Rs” of Change are for? More precisely, why do candidates need to know these things by heart? These are not things that trainees should have absorbed as learned principles, they are things they can look up if they need to. Many of the questions test whether the candidate has memorised the sacred books rather than whether they have a useful knowledge of ITSM/ITIL.

In my satirical book Introduction to Real ITSM, I made fun of these exams with a little exam of my own. I’ll reproduce a couple of the Real ITSM questions as we go along to illustrate my points. We can start with:

1) On page 49 of Real ITSM what is the first word on line 10 (counting page headers as lines as well) a) the b) and c) This d) By

Testing Multiple Choice Skills

The second group of ITIL Foundation questions that really tick me off are the ones that test the candidate’s logic skills.

The satirical Real ITSM question that spoofed the logic tests was:

2) Which of the following is not the opposite of not being unresponding to a user’s failure to call the Service Desk?

a) not calling the user

b) not failing to call the user

c) not calling not the user

d) not the opposite of not calling someone other than the user

If that seems over the top, consider this question (reportedly from the official “mock” exams but we can’t say for sure as the mock exams to the best of my knowledge are not published – True!):

Which of the following are not included in Access Management?

1) Verifying the identity of users requesting access to services

2) Setting the rights or privileges of systems to allow access to authorised users

3) Defining security policies for system access

4) Monitoring the availability of systems that users should have access to

a) 2 and 4 only b) 1 and 4 only c) 1 and 3 only d) 2 and 3 only

Reportedly one of the live exam questions looks something like this:

Order the following CSI implementation steps in the correct order of the PDCA (Plan-Do-Check-Act) model:

1) Assign roles and responsibilities to work on CSI activities

2) Measure and review CSI plan objectives are being met

3) Identify CSI requirements, objectives and scope

4) Implement CSI enhancement

a) 1-3-2-4 b) 2-3-4-1 c) 3-1-2-4 d) 3-4-2-1

I recently sat the ITIL v3 Foundation exam. Despite expounding about ITIL for years and building a pretty good following on my websitehttp://www.itskeptic.org/, I haven’t certified in ITIL since my v2 Foundation exam so many years ago. I argued that I would certify when the lack of it prevented me getting work. Well, it is 2009 and times are tough and off I went to Prometric to sit v3 Foundation. (I can’t imagine any recession persuading me to invest what ITIL v3 Expert demands, but that is another article).

Before sitting the exam, I ran through a number of practice exams available on the Web (personally I liked ITIL Campus and Taruu). I’ve been critical of the concept of multi-choice testing all along, and we’ve pulled apart a few examples on my blog, but as I was doing question after question on the practice exams (and the final real exam) it struck me how bad most of them were.

The Foundation exams should test an understanding of the fundamentals of ITSM, and a grasp of the key principles of ITIL. If someone has passed Foundation that should tell us they understand IT is all about delivery of services; that a catalogue defines what we do in users’ terms; that modifications to everything that impacts a service should go through Change; that an Incident is distinct from a Problem; that if what our suppliers promise doesn’t line up with we promise the business, we are screwed, etc., etc.

Who gives a rodent’s posterior exactly how many steps there are in the ITIL CSI process? (Especially since that process is different from every best practice CSI process already in existence―that too is another article), or what the “seven Rs” of Change are for? More precisely, why do candidates need to know these things by heart? These are not things that trainees should have absorbed as learned principles, they are things they can look up if they need to. Many of the questions test whether the candidate has memorised the sacred books rather than whether they have a useful knowledge of ITSM/ITIL.

In my satirical book Introduction to Real ITSM, I made fun of these exams with a little exam of my own. I’ll reproduce a couple of the Real ITSM questions as we go along to illustrate my points. We can start with:

1) On page 49 of Real ITSM what is the first word on line 10 (counting page headers as lines as well) a) the b) and c) This d) By

Testing Multiple Choice Skills

The second group of ITIL Foundation questions that really tick me off are the ones that test the candidate’s logic skills.

The satirical Real ITSM question that spoofed the logic tests was:

2) Which of the following is not the opposite of not being unresponding to a user’s failure to call the Service Desk?

a) not calling the user

b) not failing to call the user

c) not calling not the user

d) not the opposite of not calling someone other than the user

If that seems over the top, consider this question (reportedly from the official “mock” exams but we can’t say for sure as the mock exams to the best of my knowledge are not published – True!):

Which of the following are not included in Access Management?

1) Verifying the identity of users requesting access to services

2) Setting the rights or privileges of systems to allow access to authorised users

3) Defining security policies for system access

4) Monitoring the availability of systems that users should have access to

a) 2 and 4 only b) 1 and 4 only c) 1 and 3 only d) 2 and 3 only

Reportedly one of the live exam questions looks something like this:

Order the following CSI implementation steps in the correct order of the PDCA (Plan-Do-Check-Act) model:

1) Assign roles and responsibilities to work on CSI activities

2) Measure and review CSI plan objectives are being met

3) Identify CSI requirements, objectives and scope

4) Implement CSI enhancement

a) 1-3-2-4 b) 2-3-4-1 c) 3-1-2-4 d) 3-4-2-1

If it is a test of basic principles then I have to say I could place (1) in either Plan or Do or Act. I would eventually get the one the examiners want by a process of elimination. If it is a test of rote learning, then the sacred book on page 113 says:

PLAN: “Framework of management roles and responsibilities"

DO: “Documenting roles and responsibilities” “Allocation of roles and responsibilities to work on CSI initiatives”

ACT: “Updating… roles and responsibilities”

So, yes the answer is there but any new student will get confused trying to recall that in a closed-book exam. And all the ITIL exams are closed book―an odd approach from book vendors. Second, what an over-complicated question! Are we testing knowledge of PDCA or the ability to mentally map sequences to each other?

Here's my rewrite:

Assign the following CSI implementation steps to the correct steps of the PDCA (Plan-Do-Check-Act) model:

- Assign roles and responsibilities to work on CSI activities

- Measure and review CSI plan objectives are being met

- Identify CSI requirements, objectives and scope

- Implement CSI enhancement

a) PLAN-CHECK-ACT-DO

b) ACT-PLAN-DO-CHECK

c) CHECK-PLAN-DO-ACT

d) ACT-CHECK-PLAN-DO

Why make it a test of mental gymnastics?

Implications

The implications of bad exams are several. First, executives and management get chased along to Foundation courses. The feedback on my blog is that many of them are underwhelmed by the exam questions. Nor will it escape their notice that most Foundation training courses are training on how to pass the exam. This undermines ITIL’s credibility.

Second, it undermines ITIL certification credibility. If senior people learn that getting Foundation certified is about learning a couple of hundred sample questions so one can pass a badly-crafted multiple choice exam, what value does that certification have? If we don’t fix this, watch the certification’s prestige decline with time as more and more companies wise up.

As Aiden Lawes, a leading figure in the ITIL world, said recently

"Value is subjective. If, for any reason, candidates place less value on certain qualifications than on others, that view will gradually be transmitted to those responsible for purchasing training programmes or defining criteria for particular job roles. Once the value of the qualifications is brought into question, the whole scheme is in jeopardy."

Third, it does little to evangelise ITIL. We all know Foundation doesn’t teach enough to be useful. It is a once-over-lightly sometimes referred to as sheep-dipping. There is value in getting the trainees speaking a common language but the other real purpose is to get them enthused, onside. Bad exams will have the opposite effect.

This debate has been raging for years now―the syllabus keeps changing, the exams keep changing, and delivery has dragged on. It still is not fixed and it needs to be fixed. This is something we will look at in part II of this article.

Rob England is an IT industry commentator and consultant, best known for his blog The IT Skeptic.