Change Management Do's and Don'ts

By Kevin Behr, Gene Kim and George Spafford

Stabilize the Patient and Modify First Response
The goal of a successful Change Management process implementation is to reduce the amount of unplanned work as a percentage of total work done. Organizations that are in a constant firefighting mode can have this percentage at 65 percent or even higher.

The first phase of a Change Management implementation as outlined in the Visible Ops Handbook (a useful guidance and prescriptive roadmap for organizations beginning or continuing their IT process improvement journey) resembles the triage system used by hospitals to allocate scarce medical resources. In a similar fashion, IT must identify the most critical systems generating the most unplanned work and take appropriate action to gain control.

The primary goal of this phase is to stabilize the environment, allowing work to shift from perpetual firefighting to more proactive work that addresses the root causes of problems. Start by identifying the systems and business processes that generate the greatest amount of firefighting. When problems are escalated to IT operations, which servers, networking devices, infrastructure or services are constantly being revisited each week (or worse, each day)?

These items are your list of "most fragile patient" which are generating the most unplanned work. These are the patients that must be protected from uncontrolled changes, both to curb firefighting and to free up enough cycles to start building a safer and more controlled route for change.

For each fragile patient (i.e. server, networking device, asset, etc.), do the following:

1. Reduce or eliminate access: Clear everyone away from the asset unless they are formally authorized to make changes. Because these assets have low change success rates, you must reduce the number of times the dice are rolled.
2. Document the new change policy: keep the change policy simple: "Absolutely no changes to this asset unless authorized by me." This policy is the preventive control and creates an expectation of behavior.
3. Notify stakeholders: After the initial change policy is established, notify all of the stakeholders about the new process. Make sure the entire staff sees it: Email it to the team, print it out, and add it to login banners.
4. Create change windows: Work with stakeholders to identify periods of time when changes to production systems can be made. Your goal will be to eventually schedule all changes into these maintenance windows. Amend the change policy accordingly. For example, Once I authorize the changes, I will schedule the work to be performed during one of the defined maintenance windows on either Saturday or Sunday between 3 and 5 pm.
5. Reinforce the process: By now, you have defined a clear expectation of how changes are to be made. Make sure that people are aware of the new process and reinforce it constantly. For example; "Team, let me be clear on this: These processes are here to enable the success of the entire team, not just individuals. Anyone making a change without getting authorization undermines the success of the team, and we'll have to deal with that. At a minimum, you'll have to explain why you made your cowboy change to the entire team. If it keeps happening, you may get the day off, and eventually, it may prevent you from being a part of this team."

Electrify the Fence
Put a fence around the systems where unauthorized changes are causing the most carnage. Next, you will electrify the fence in order to keep everyone accountable and responsible for playing by the rules.The goal is to start creating a culture of change management.

To do this, proper change monitoring must be in place so you can trust, but verify. You will use this instrumentation to detect and verify that changes are happening within the specified change management process, and also to negatively reinforce and deter changes that are not.

You must be aware of changes on all infrastructure that you are managing: servers, routers, network devices, databases and so forth. Each detected change must either map to authorized work, or it must be flagged for investigation.

Critical questions that need to be answered are:

• Who made the change?
• What did they change?
• Should it be rolled back? If so, then how?
• How do we prevent it from happening again in the future?

The key to creating a successful culture of change management is accountability. If the change process is repeatedly bypassed, management must be willing to take appropriate disciplinary action.

Create the Change Team
In this next step, you will continue to develop the change management process by creating a Change Advisory Board (CAB), comprised of the relevant stakeholders of each critical IT service. These stakeholders are the people who can best make decisions about changes because of their understanding of the business goals, as well as technical and operational risks.

Create a Change Request Tracking System
A prerequisite for any effective Change Management process is the ability to track requests for changes (RFCs) through the authorization, implementation, and verification processes.

Paper-based manual systems quickly become impractical when the organization is large or complex, or when the number of changes is high. Because of this, most groups use some computerized means to track RFCs and assign work order numbers. Some refer to these applications as ticketing systems or change workflow systems. The primary goals of a change request tracking system are to document and track changes through their lifecycle and to automate the authorization process.

Secondarily, the system can generate reports with metrics for later analysis. Each change requester should gather all the information the Change Manager needs to decide whether the change should be approved. In general, the more risky the proposed change is, the more information that is required.

For instance, a business as usual (BAU) change, such as rebooting a server or rotating a log file, may require very little data and oversight prior to approval. On the other hand, a high-risk change such as applying a large and complex security patch on a critical production server may not only require good documentation of the proposed change, but also extensive testing before it can even be considered for authorized deployment.

Start Weekly Change Management Meetings (to Authorize Change) and Daily Change Briefings (to Announce Changes)
Now that you have identified the change stakeholders by creating the CAB, the next step is to create a forum for them to make decisions on requested changes.

The CAB will authorize, deny, or negotiate a change with the requester. Authorized changes will be scheduled, implemented, and finally verified. The goal is to create a process that enables the highest successful change throughout the organization with the least amount of bureaucracy possible.

While they may seem unnatural at first, with practice, weekly 15 minute change management meetings are possible. Take special care to avoid an attitude of "just get it done," which allows people to make changes that circumvent the change approval process.

If you make it easy for all changes to flow through your process, it will soon be easier to use the process than to circumvent it, even during emergencies. CABs must meet on a regular published schedule that all stakeholders understand. To start, each CAB should meet weekly.

Miscellaneous Change Management Do's and Don'ts
Here are some tips for change management.

Items to do:

• Do post-implementation reviews to determine whether the change succeeded or not.
• Do track the change success rate.
• Do use the change success rate to learn and avoid making historically risky changes.
• Do make sure everyone attends the meetings, otherwise auditors have a good case that this is a nonfunctioning control.
• Do categorize the disposition of all changes. In other words, all outcomes must be documented once a change is approved. Three potential outcomes are:
  • Change withdraw - the change requester rescinds the change request along with the reason why. This should not be flagged as a failed change in change metrics.
  • Abort - the change failed, accompanied by documentation of what went wrong.
  • Completed successfully - the change was implemented and is functioning appropriately.

  Items not to do:
  • Do not authorize changes without rollback plans that everybody reviews. Changes do fail, so be proactive and think ahead about how to recover from a problem rather than attempting to do so during the heat of firefighting.
  • Do not allow rubber stamping approval of changes.
  • Do not let any system changes off the hook - someone made it, so understand what caused it.
  • Do not send mixed messages. Bear in mind that the first time the process is circumvented, incredible damage can be done to the process. "Well heck, we did it last time" or the boss said, "just do it" - both send the wrong messages.

  Do not expect to be doing closed loop Change Management from the start. Awareness is better than being oblivious, and managed is better than unmanaged. Start with a particular class of changes and constantly refine the process.

  The Spectrum of Change:
  The management of change is an evolutionary process. Groups should not become discouraged as they start developing their change management processes. The solutions may require changing people, processes, and technology. The following illustrates the stages of change management:

  1. Oblivious to change - Hey, did the switch just reboot?
  2. Aware of change - Hey, who just rebooted the switch?
  3. Announcing change - Hey, I'm rebooting the switch. Let me know if that will cause a problem.
  4. Authorizing change - Hey, I need to reboot the switch. Who needs to authorize this?
  5. Scheduling change - When is the next maintenance window? I'd like to reboot the switch then.
  6. Verifying change - Looking at the fault manager logs, I can see that the switch rebooted as scheduled.
  7. Managing change - Let's schedule the switch reboot to week 45 so we can do the maintenance upgrade and reboot at the same time.

  The Change Management goal is to reduce the amount of time spent on unplanned work down, by reducing the number of self-inflicted problems and modifying how problems are solved so that change is ruled out early in the repair cycle. By increasing the change success rate and reducing MTTR, you not only decrease the amount of unplanned work, but also increase the number of changes that can be successfully implemented by the organization.

  Excerpted from - The Visible Ops Handbook, Starting ITIL in 4 Practical Steps by Kevin Behr, Gene Kim and George Spafford. Copyright 2004 by the IT Process Institute (ITPI); all rights reserved.