Roles and Activities > Tester Role Set > Tester > Execute Test Suite

Purpose
  • To execute the appropriate collections of tests required to validate the product quality
  • To capture test results that facilitate the assessment of the product
Steps
Input Artifacts: Resulting Artifacts:
Frequency: This activity is typically conducted multiple times per iteration. .
Role: Tester
Tool Mentors:
More Information:

Workflow Details:

Setup test environment to known state To top of page

Purpose: To accurately establish the test environment in preparation for Test Suite.

Setup the test environment to ensure that all the needed components (hardware, software, tools, data, etc.) have been implemented and are in the test environment, ready in the correct state to enable the tests to be conducted. Typically this will involve some form of basic environment reset (e.g. Registry and other configuration files), restoration of underlying databases to known state, and so forth in addition to tasks such as loading paper into printers. While some tasks can be performed automatically, some aspects typically require human attention.

Tools such as those that enable hard-disk image capture and restoration are extremely valuable to manage this effort effectively.

Set execution tool options To top of page

Purpose: To appropriately configure the tools used in Test Suite execution.

Set the execution options of the supporting tools. Depending on the sophistication of the tool, this may be many options to consider. Failing to set these options appropriately may reduce the usefulness and value of the resulting Test Logs and other outputs. Where possible, you should try to store these tool options and settings so that they can be reloaded easily based on one or more predetermined profiles.

In the case of automated test execution tools, there may be many different settings to be considered. In the case of manual testing, it may be a simple matter of partitioning a new entry in a support system for logging results, signing into issue and changes request logging systems. Important concerns include the name, location and state of the Test Log to be written to, and the speed at which execution should be performed.

Schedule Test Suite execution To top of page

Purpose: To determine the appropriate time for test execution to begin.

In many cases where test execution can be attended, the Test Suite can be executed relatively on demand. In these cases, scheduling might need to consider environment resets and the work of other testers, as well as different test teams and other team members that share the test environment.

However, in cases where unattended execution of automated tests is required, or where the execution of many tests running concurrently on different machines must be coordinated, some form of automated scheduling mechanism may be required. Either use the features of your automated test execution tool or develop your own utility functions to enable the required scheduling.

Execute Test Suite To top of page

Purpose: To conduct the tests enclosed in the Test Suite and to monitor and support their completion.

Execute the Test Suite(s). Note: executing the Test Suites and Test Scripts will vary dependent upon whether testing is automated or manual.

  • Automated testing: The test scripts created during the Implement Test activity are executed.
  • Manual execution: The structured Test Scripts developed during the Design Test activity are used to manually execute test.

Evaluate execution of Test Suite To top of page

Purpose: To determine whether the Test Suite executed to completion or halted abnormally, and make an assessment whether corrective action is required.

The execution of testing ends or terminates in one of two conditions:

  • Normal: all the Test Scripts execute as intended to completion. 
  • Abnormal or premature: the Test Scripts did not execute completely as intended. When testing ends abnormally, the Test Logs from which subsequent Test Results are derived may be unreliable. The cause of the abnormal termination needs to be identified, and if necessary, the fault corrected and the tests re-executed.

Recover from halted tests To top of page

Purpose: To determine the appropriate corrective action to recover from a halted Test Suite execution, and if required correct the problem, recover, and re-execute the Test Suite.

To recover from halted tests, do the following:

Inspect the Test Logs and other output To top of page

Inspect the Test Logs and other output for completeness and accuracy. Identify where errors have occurred and inspect them. There are two major types of halted tests:

  • Fatal errors - the system fails (network failures, hardware crashes, etc.)
  • Test Script failures - this is when some part of a Test Script within a Test Suite cannot be executed as intended.

Both types of abnormal occurrence during test execution may exhibit the following symptoms:

  • a large number of or ongoing occurrence of unexpected actions, windows, or events occur while the Test Suite is executing
  • the test environment appears unresponsive, is slow or is in an undesirable state (such as hung or crashed).

Work through the symptoms until you can determine the root cause of the problem.

Restore test environment and tool options To top of page

It's often necessary to adjust the test environment as part of correctly the problem. As such, it's a good idea to restore the environment again from a known state, before making any necessary corrections. If you use the reset environment to investigate the problems, remember that you may need to reset the environment again before applying any permanent changes.

Correct errors To top of page

Correct the Test Script, Test Data and test environment or other aspect of the test. After making the necessary changes, save the Test Script and backup or save the test environment as required.

Schedule and execute Test Suite again To top of page

Reschedule and re-execute the Test Suite. Depending on what recovery process is available (if any), this step may differ from that performed originally.

Reevaluate execution of Test Suite To top of page

Confirm the Test Suite now runs to completion. In some cases where you have created test execution recovery procedures, you may be able to recover test execution from a point part-way through the test run.

If there are still problems, work through the subsections that make up Recover from halted tests until the problem is resolved.

Inspect the Test Logs for completeness and accuracy To top of page

Purpose: To determine if the Test Suite execution generated worthwhile test information and if not, to identify appropriate corrective action.

When test execution initially completes, the Test Logs should be reviewed to ensure that the logs are reliable and reported failures, warnings, or unexpected results were not caused by external influences (to the target-of-test), such as improper environment setup or invalid Test Data.

For GUI-driven automated Test Scripts, the most common failures reported when Test Scripts fail are given below:

  • Test verification failures - this occurs when the actual result and the expected result do not match. Verify that the verification method(s) used focus only on the essential items and / or properties and modify if necessary.
  • Unexpected GUI windows - this occurs for several reasons. The most common is when a GUI window other than the expected one is active or the number of displayed GUI windows is greater than expected. Ensure that the test environment has been setup and initialized as intended for proper test execution.
  • Missing GUI windows - this failure is noted when a GUI window is expected to be available (but not necessarily active) and is not. Ensure that the test environment has been setup and initialized as intended for proper test execution. Verify that the actual missing windows are / were removed from the target-of-test.

If the reported failures are due to errors identified in the test artifacts, or due to problems with the test environment, the appropriate corrective action should be taken and the testing re-executed. See: Activity: Analyze Test Failure.

If the Test Log indicates the failures are genuinely due to the Target Test Items, then the execution portion of the activity is complete.

Restore test environment to known state To top of page

Purpose: To ensure the environment is properly cleaned up after Test Script execution.

(See the first step) Next you should restore the environment back to it's original state. Typically this will involve some form of basic environment reset (e.g. Registry and other configuration files), restoration of underlying databases to known state, and so forth in addition to tasks such as loading paper into printers. While some tasks can be performed automatically, some aspects typically require human attention.

Maintain traceability relationships To top of page

Purpose: To enable impact analysis and assessment reporting to be performed on the traced items.

Using the Traceability requirements outlined in the Test Plan, update the traceability relationships as required. Test Suites might be traced to defined Test Cases or to Test Ideas. Optionally, they may be traced to Use Cases, software specification elements, Implementation Model elements and to one or more measures of Test Coverage. Update the status of the relationships that were established during implementation of the Test Suite.

Evaluate and verify your results To top of page

Purpose: To verify that the activity has been completed appropriately and that the resulting artifacts are acceptable.

Now that you have completed the work, it is beneficial to verify that the work was of sufficient value, and that you did not simply consume vast quantities of paper. You should evaluate whether your work is of appropriate quality, and that it is complete enough to be useful to those team members who will make subsequent use of it as input to their work. Where possible, use the checklists provided in RUP to verify that quality and completeness are "good enough".

Have the people performing the downstream activities that rely on your work as input take part in reviewing your interim work. Do this while you still have time available to take action to address their concerns. You should also evaluate your work against the key input artifacts to make sure you have represented them accurately and sufficiently. It may be useful to have the author of the input artifact review your work on this basis.

Try to remember that that RUP is an iterative process and that in many cases artifacts evolve over time. As such, it is not usually necessary—and is often counterproductive—to fully-form an artifact that will only be partially used or will not be used at all in immediately subsequent work. This is because there is a high probability that the situation surrounding the artifact will change—and the assumptions made when the artifact was created proven incorrect—before the artifact is used, resulting in wasted effort and costly rework. Also avoid the trap of spending too many cycles on presentation to the detriment of content value. In project environments where presentation has importance and economic value as a project deliverable, you might want to consider using an administrative resource to perform presentation tasks.



Copyright  1987 - 2001 Rational Software Corporation


Display Rational Unified Process using frames

Rational Unified Process