Thursday, June 3, 2010
Over the years, I've heard many argue about what makes a test successful or not, and how to best present test results relative to business objectives when management, auditors and regulators are pushing for 'successful tests'. I know of several large organizations that base employee performance evaluations, at least partially, on the outcomes of disaster recovery tests.
I think much of the confusion comes from the very sloppy way we, as practitioners, use terminology that has meaning in common speech outside our specialty. The following describes how I distinguish testing, exercises and practice, and will hopefully serve to help others in the field to explain the differences and how success and failure in each can mean very different things for the organization.
Tests
One activity that takes place in organizations involves validating that a proposed recovery or business resumption process works under ideal circumstances. This can take the form of working from a "plan" or, although discouraged, letting an employee do what he knows how to do without any documented guidance. The key here is that at this stage, the recovery process is somewhat experimental and untried.
Validating whether the process works can be accurately described as a 'test'. The result is usually a pass or fail outcome, as it either works or it doesn't. And I stress that tests are performed under ideal (or at least consistent) circumstances, because test results in a controlled environment should always be repeatable.
Exercises
Another activity that businesses should undertake, though few actually do, is the exercise. As an analogy, consider the body builder. A body builder knows and can successfully perform his routines with proper form. But simply going through the motions is not what gives the body builder his size or strength. To grow and improve, the body builder must push his body to its limits by constantly increasing the difficulty or intensity of his program. Frequently, he 'lifts to failure', meaning he lifts very heavy weight enough times in a single set that he just cannot lift any more.
The same concept applies to disaster recovery planning. Once the recovery process is tested and proven to work under ideal circumstances, organizations must increase the intensity of their program to keep it strong and continuing to grow stronger. Increasing intensity involves introducing variables into an exercise scenario that requires participants to think a bit outside the plan, executing recovery under other-than-ideal circumstances. By making each exercise more demanding than the last and exercising 'to failure', the organization grows more resilient in the face of a broader range of disaster sources and scales of business disruptions.
Practice
The last activity I want to address that occurs most frequently and is incorrectly called a 'test' is practice. To use another sports analogy, imagine a little league baseball player. At some point in his short baseball career he was given training in the basic mechanics of how to swing the bat to hit the ball. Through trial and error and 'testing' various grips and postures he learned the most effective way to do so. Later, to improve and expand his abilities, he 'exercised' his batting technique by introducing variation in pitches and learned how to hit an in-field grounder versus a pop-fly. And while there is a very fuzzy line between training, testing and exercise, the player does not neglect going back and practicing the fundamentals of simply hitting the ball.
In disaster recovery, just as in baseball, to repeat the same process over and over to get better and more comfortable with it is 'practice'. Practice should nearly always result in successful execution, but can perhaps be graded on a scale of success. In my experience, due to the demand for successful test execution, organizations often conduct 'practice', call it a test and present those successful 'test' results to management and stakeholders. However, if an organization is only practicing execution of their plans, and they are nearly always successful in doing so, that organization is sure to have developed a false sense of security in their apparent ability to recover while not prepared to manage through the variables that will inevitably occur during a real incident.
Conclusion
A business continuity / disaster recovery test and exercise program must consist of all three of these components. Further, objectives for each type will be different and must be defined within the overall program.
The objective of a 'test' is to validate that the basic, fundamental process actually works under ideal circumstances. The test fails if the process does not work and a new process should be developed.
The objective of 'practice' is continuous improvement. Practice should always result in successful execution, though it could be along a spectrum of success based on how a particular organization measures it. For example, recovering faster (closer to meeting RTO's) may indicate a higher level of success. Failure during practice would indicate something significant was missed during testing and require revisiting the recovery process.
The objective of an 'exercise' is to evaluate the limits of the recovery process and the people involved. In an exercise, people and processes are pushed to failure. Not successfully executing during an exercise does not necessarily imply failure, and should in fact be anticipated. An exercise as a whole, even if recovery failed, can be successful if information is obtained, lessons are learned and corrective actions are taken to improve performance next time.
---------
Disagree or have something to add? Comments are certainly welcome!
I think much of the confusion comes from the very sloppy way we, as practitioners, use terminology that has meaning in common speech outside our specialty. The following describes how I distinguish testing, exercises and practice, and will hopefully serve to help others in the field to explain the differences and how success and failure in each can mean very different things for the organization.
Tests
One activity that takes place in organizations involves validating that a proposed recovery or business resumption process works under ideal circumstances. This can take the form of working from a "plan" or, although discouraged, letting an employee do what he knows how to do without any documented guidance. The key here is that at this stage, the recovery process is somewhat experimental and untried.
Validating whether the process works can be accurately described as a 'test'. The result is usually a pass or fail outcome, as it either works or it doesn't. And I stress that tests are performed under ideal (or at least consistent) circumstances, because test results in a controlled environment should always be repeatable.
Exercises
Another activity that businesses should undertake, though few actually do, is the exercise. As an analogy, consider the body builder. A body builder knows and can successfully perform his routines with proper form. But simply going through the motions is not what gives the body builder his size or strength. To grow and improve, the body builder must push his body to its limits by constantly increasing the difficulty or intensity of his program. Frequently, he 'lifts to failure', meaning he lifts very heavy weight enough times in a single set that he just cannot lift any more.
The same concept applies to disaster recovery planning. Once the recovery process is tested and proven to work under ideal circumstances, organizations must increase the intensity of their program to keep it strong and continuing to grow stronger. Increasing intensity involves introducing variables into an exercise scenario that requires participants to think a bit outside the plan, executing recovery under other-than-ideal circumstances. By making each exercise more demanding than the last and exercising 'to failure', the organization grows more resilient in the face of a broader range of disaster sources and scales of business disruptions.
Practice
The last activity I want to address that occurs most frequently and is incorrectly called a 'test' is practice. To use another sports analogy, imagine a little league baseball player. At some point in his short baseball career he was given training in the basic mechanics of how to swing the bat to hit the ball. Through trial and error and 'testing' various grips and postures he learned the most effective way to do so. Later, to improve and expand his abilities, he 'exercised' his batting technique by introducing variation in pitches and learned how to hit an in-field grounder versus a pop-fly. And while there is a very fuzzy line between training, testing and exercise, the player does not neglect going back and practicing the fundamentals of simply hitting the ball.
In disaster recovery, just as in baseball, to repeat the same process over and over to get better and more comfortable with it is 'practice'. Practice should nearly always result in successful execution, but can perhaps be graded on a scale of success. In my experience, due to the demand for successful test execution, organizations often conduct 'practice', call it a test and present those successful 'test' results to management and stakeholders. However, if an organization is only practicing execution of their plans, and they are nearly always successful in doing so, that organization is sure to have developed a false sense of security in their apparent ability to recover while not prepared to manage through the variables that will inevitably occur during a real incident.
Conclusion
A business continuity / disaster recovery test and exercise program must consist of all three of these components. Further, objectives for each type will be different and must be defined within the overall program.
The objective of a 'test' is to validate that the basic, fundamental process actually works under ideal circumstances. The test fails if the process does not work and a new process should be developed.
The objective of 'practice' is continuous improvement. Practice should always result in successful execution, though it could be along a spectrum of success based on how a particular organization measures it. For example, recovering faster (closer to meeting RTO's) may indicate a higher level of success. Failure during practice would indicate something significant was missed during testing and require revisiting the recovery process.
The objective of an 'exercise' is to evaluate the limits of the recovery process and the people involved. In an exercise, people and processes are pushed to failure. Not successfully executing during an exercise does not necessarily imply failure, and should in fact be anticipated. An exercise as a whole, even if recovery failed, can be successful if information is obtained, lessons are learned and corrective actions are taken to improve performance next time.
---------
Disagree or have something to add? Comments are certainly welcome!
