The Importance of System Change Controls

I had the pleasure recently of attending a webinar from BetterManagement.com in which Gene Kim, CTO of Tripwire, gave a very interesting presentation on the need for effective change management controls in today's network devices. This article is a brief discussion of three categories of controls that Gene terms "IT Controls 101" as they relate to system configurations.

The Need for Controls

The need for effective change management is driven by two factors -- compliance with Sarbanes-Oxley Section 404 as well as the need to drive towards operational excellence.

First, it is clear to auditors that financial applications reside on top of infrastructure systems. Indications of poor service levels are an immediate flag that there are control issues. Think of it this way, the ERP system may be running flawlessly, but if nobody can reach it due to a network failure, failed system upgrade, or improperly tested business rule change, then there are issues. For today's organizations to operate effectively, access to systems must adhere to defined service levels.

Second, companies exhibiting best practices have a server-to-administrator ratio of 120:1 and are moving quickly beyond that number. Stellar ratios such as this can only be achieved through the careful management of IT resources. Whether we are talking about servers, switches, firewalls, or whatever else, each time the device is changed, there is a risk of an error creating an outage.

Many groups carefully build, test and deploy only to then enter a phase of applying patches and then praying that there isn't a failure. Best-in-class organizations push as many changes as possible back into the build and test phases. In other words, only approved emergency change requests are applied to production systems. All others must be applied to test servers and go through a formal test program prior to being deployed to production systems. These organizations recognize the fact that as the number of patches increases, the inherent reliability in the system decreases.

Categories of Controls

Without a doubt, changes are constant. New business rules appear, new security holes need to be fixed, flaws with hardware drivers that need to be upgraded and so on are all pressures that make administrators update servers.

The issue is that change is an inherent part of IT and changes to devices and software must be part of a managed process. The methods to control change can be generalized into three broad categories: Preventive, Detective and Corrective. Let's discuss each of these three areas:

Preventive

These are controls aimed at preventing unauthorized changes to systems. These controls include authentication, authorization, separation of duties, as well as a formal change management process that includes effective testing, technical review, business owner review, and documentation.

Detective

These are methods aimed at identifying unauthorized changes. For example, not all unauthorized changes are malicious. In fact, well-intentioned people who either ignore the change management process or forget to document their change after it has been implemented do the overwhelming majority of unauthorized changes!

Regardless, there must be a control wherein changes are identified and tied back to authorized change requests. Any unauthorized changes must be investigated and appropriate action taken to prevent a reoccurrence. Obviously, an automated method is preferable and there are tools from Tripwire and Ecora to help facilitate both the scheduled downloading of configuration files plus scanning for changes.

Corrective

Corrective controls are aimed at the methods used to restore systems to the prior state. For example, if a configuration is updated, how will it be restored in the event that issues necessitate rolling back to it? Ideally, automated tools are used to periodically back up configurations. As mentioned, Tripwire and Ecora can assist with this plus there are numerous utilities and scripts available to help with this effort.

Getting Started

There are two ways to get started. First, you can read up on ITIL (Information Technology Infrastructure Library) by purchasing the books and visiting various web resources. Second, you be audited by a third party who is helping organizations put IT controls in place and this is a hot topic these days.

I strongly recommend using a standard, such as ITIL, as a roadmap to identify what needs to be done as opposed to attempting an ad hoc method that incorporates only what a few people know.

Summary

Change management is critical to have highly reliable systems that meet the defined service levels of the organization. To this end, best-practice organizations are pushing all changes back into the build and test phases such that only rare emergency changes are actually performed on production systems.

The whole network device change process must become formalized and incorporate security, testing, and documentation. The organization must ensure that appropriate preventive, detective and corrective controls exist in order to meet the challenges of Sarbanes-Oxley as well as to improve operational efficiencies.

Additional Resources

As a starting point, the following websites have additional information available:

Gene's article at BetterManagement.com
Ecora
Information Technology Infrastructure Library (ITIL)
IT Service Management Forum
Pink Elephant
Tripwire