Emergency Change ManagementOrganizations need to establish and follow formal emergency change processes to ensure that risks are properly managed.
In doing so, some organizations bypass their standard change management processes, make the change and then document what was done later. While this is one approach, it may leave the organization vulnerable to certain risks.
The IT Infrastructure Library (ITIL) recognizes this need and outlines suggestions for emergency changes and the Change Advisory Board/Emergency Committee (CAB/EC). For each system, there should exist an emergency-fix policy that identifies the procedure to follow and who is to sit on the CAB/EC. Typically, this is established during release management for new systems or retroactively planned for existing systems.
At the beginning of an emergency change, there needs to be a decision made that the request is of an emergency nature and that it warrants the accelerated model. This decision should not be made lightly as the emergency change process must be for exceptions and not become a shortcut to getting changes into production for whatever reason.
In some organizations, this decision may be made by the formally empowered change manager or a person acting in that capacity. In others, an escalation list is created that identifies the people duly authorized to make that decision plus their contact information. Furthermore, it is prioritized in terms of who to try first, second, third and so on.
Document, Document, Document
The decision of whether something is an emergency should follow defined criteria. Once a change request is cleared for emergency processing, it then enters the appropriate emergency change model. The exact process can vary but at the least the change should be discussed with the CAB/EC. In the event that there isnt time for a full CAB meeting, the CAB/EC is comprised of a minimum core group of people that will need to review and approve an emergency change before it can go to production. If there isnt a change manager, make it the responsibility of the senior person in the CAB/EC to notify the CAB of the emergency change so they are aware and so the emergency change can be formally reviewed.
Once the emergency change has been implemented, the responsible party or parties need to generate a formal RFC and document the results for the standard CAB to review. There are a number of reasons why this is very important:
1. The CAB needs to understand the change in order to communicate it to others who may question what is different.
2. The CAB needs to review the change and make sure it doesnt affect other changes that may be in process, future direction, etc.
3. After an emergency, there needs to be an understanding of lessons learned. How could this type of emergency be prevented or better handled in the future? This crosses over into problem management, but the question is the same How can we improve? To be clear, dont redundantly investigate the root cause in both the emergency change and problem management processes. If both are defined, have the groups collaboratively review the change.
4. The CAB needs to ensure the change was truly of an emergency nature. If it wasnt, then corrective action is needed, ranging from a warning to rolling back the change to formal disciplinary action. Again, the emergency change process must be reserved for emergencies. Because it is likely easier than the standard processes there will be the temptation to use it push otherwise normal changes through to production and this must be guarded against.
5. Lastly, the formal documentation is needed to ensure that there is always knowledge of what is in production in order to support the configuration, release, incident and problem management processes. People must know what the production builds are in order to do their jobs.
Depending on the emergency change procedure, there may be additional steps prior to the CAB review, such as formalized testing to see if the emergency change breaks any functionality, interfaces, security, etc. In addition, groups may identify the standard change model that would have been used if the change were not an emergency and then follow that process to identify required tasks and documentation for the CAB to review. There will always be changes, incidents and the need to implement emergency changes. The key is to understand that emergencies must follow a defined process and not simply bypass standard processes. The intent of an emergency change process is to manage risks to an acceptable level while responding to the needs of the business in a timely manner.