Roles and Activities > Analyst Role Set > Test Analyst > Determine Test Results

Purpose
  • To make ongoing summary evaluations of the quality of the product
  • To determination the detailed Test Results
  • To propose appropriate corrective actions to resolve failures in quality
Steps
Input Artifacts: Resulting Artifacts:
Frequency: This activity is typically conducted multiple times per iteration. .
Role: Test Analyst
Tool Mentors:
More Information:

Workflow Details:

Examine all test incidents and failures To top of page

Purpose: To investigate each incident and obtain detailed understanding of the resulting problems.

In this activity, the Test Logs are analyzed to determine the meaningful Test Results, regarding the differences between the expected results and the actual results of each test. Identify and analyze each incident and failure in turn. Learn as much as you can about each occurrence.

Check for duplicate incidents, common symptoms and other relationships between incidents. These conditions often provide valuable insight into the root cause of a group of the incidents.

Create and maintain Change Requests To top of page

Purpose: To enter change request information into a tracking tool for assessment, management, and resolution.

Differences indicate potential defects in the Target Test Items and should be entered into a tracking system as incidents or Change Requests, with an indication of the appropriate corrective actions that could be taken.

Verify incident facts To top of page

Verify that there is accurate, supporting data available. Collate the data for attachment directly to the Change Request, or reference where the data can be obtained separately.

Whenever possible, verify that the problem is reproducible. Reproducible problems have much more likelihood of receiving developer attention and being subsequently fixed; a problem that cannot be reproduced both frustrates development staff and will waste valuable programming resources in fruitless research. We recommend that you still log these incidents, but that you consider identifying unreproducable incidents separately from the reproducible ones.

Clarify Change Request details To top of page

It's important for Change Requests to be understandable, especially the headline. Make sure the headline is crisp and concise, articulating clearly the specific issue. A brief headline is useful for summary reports and discussion in CAB status meetings.

It's important that the detailed description of the Change Request is unambiguous and can be easily interpreted. It's a good idea to log your Change Requests as soon as possible, but take time to go back and improve and expand on your descriptions before they are viewed by development staff.

Provide candidate solutions, as many as practical. This helps to reduce any remaining ambiguity in the description, often helping to clarify. It also ensures increases the likelihood that the solution will be close to your exceptions. Furthermore, it shows that the test team is not only prepared to find the problems, but also to help identify appropriate solutions.

Other details to include are screen image captures, Test Data files, automated Test Scripts, output from diagnostic utilities and any other information that would be useful to the developers in isolating and correcting the underlying fault.

Indicate relative impact severity and resolution priority To top of page

Provide an indication to the management and development staff of the severity of the problem. In larger teams the actual resolution priority is normally left for the management team to determine, however you might allow individuals to indicate their preferred resolution priority and subsequently adjust as necessary. As a general rule, we recommend you assign Change Requests an average resolution priority by default, and raise or lower that priority on a case-by-case basis as necessary.

You may need to differentiate between the impact the Change Request will have on the production environment if it isn't addressed and the impact the Change Request will have on the test effort if it isn't addressed; It's just as important for the management team to know when a defect is impacting the testing effort as it is to be aware of severity to end users.

Sometimes it's difficult to see in advance why you need both attributes. It's possible that an incident may be really severe, such as a system crash, but the actions required to reproduce it very unlikely to occur. In this case the team may indicate it's severity as high, but indicate a very low resolution priority.

Log additional Change Requests separately To top of page

Incidents often bare out the old adage"Where there's smoke, there's fire"; as you identify and log one Change Request, you quite often identify other issues that need to be addressed. Avoid the temptation to simply add these additional findings to the existing Change Request: if the information is directly related and helps to solve the existing issue better, then that's OK. If the other issues are different, identifying them against an existing CR may result in those issues not being actioned, not getting appropriate priority in their own right, or impacting the speed at which other issues are addressed.

Analyze and evaluate status To top of page

Purpose: To calculate and deliver the key measures and indicators of test.

Incident distribution To top of page

Analyze the incidents based on where they are distributed, such as functional area, quality risk, assigned tester and assigned developer.

Look for patterns in the distribution, such as functional areas that appear to have above average defects count. Also look for both developers and testers that may be overworked and where their quality of work is slipping

Test execution coverage To top of page

To evaluate test execution coverage, you need to review the Test Logs and determine:

  • The ratio between how many tests (Test Scripts or Test Cases) have been performed in this Test Cycle and a total number of tests for all Target Test Items.
  • The ratio of successfully performed test cases.

The objective is to ensure that a sufficient number of the tests targeted for this Test Cycle have been executed usefully. If this is not possible, or to augment that execution data, one or more additional test coverage criteria can be identified, based upon:

  • Quality Risk or priority
  • Specification-based coverage (Requirements etc.)
  • Business need or priority
  • Code-based coverage

See "Concepts: Key Measures of Test, Requirements-based test coverage".

Record an present the Test Results in an Test Evaluation Report for this Test Cycle.

Change Requests statistics To top of page

To analyze defects, you need to review and analyze the measures chosen as part of your defect analysis strategy. The most common defect measures used include the following different measures (often displayed in the form of a graph):

  • Defect Density - the number of defects are shown as a function of one or two defect attributes (such as distribution over functional area or quality risk compared to status or severity).
  • Defect Trend - the defect count is shown as a function over time.
  • Defect Aging - a special defect density report in which the defect counts are shown as a function of the length of time a defect remained in a given status (open, new, waiting-for-verification, etc.)

Compare the measures from this Test Cycle to the running totals for the current Iteration and those from the analysis of previous iterations, to better understand the emerging trends over time.

It is recommended you present the results in diagram form with supporting findings on request.

Make an assessment of the current quality experience To top of page

Purpose: To give feedback on the current perceived or experienced quality in the software product.

Formulate a summary of the current quality experience, highlighting both good and bad aspects of the software products quality.

Make an assessment of outstanding quality risks To top of page

Purpose: To provide feedback on what remaining areas of risk provide the most potential exposure to the project.

Identify and explain those areas that have not yet been addressed in terms of quality risks and indicate what impact and exposure this leaves the team.

Provide an indication of what priority you consider each outstanding quality risk to have, and use the priority to indicate the order in which these issues should be addressed.

Make an assessment of test coverage To top of page

Purpose: To make a summary assessment of the test coverage analysis.

Based on the work in step test execution coverage, provide a brief summary statement of the status and information the data represents.

Draft the Test Evaluation Summary To top of page

Purpose: To communicate the results of testing to stakeholders and make an objective assessment of quality and test status.

This step in the activity is to generate the Test Evaluation Summary.  This is accomplished by assembling the above information into a single report and distributing it to the appropriate stakeholders for review.

Record and present the Test Results in an Test Evaluation Summary for this Test Cycle.

Advise stakeholders of key findings To top of page

Purpose: To promote and publicize the Evaluation Summary as appropriate.

Using whatever means is appropriate, publicize this information. We recommend you consider posting these on a centralized project site, or present them in regularly held status meetings to enable feedback to be gathered and next actions to be determined.

Evaluate and verify your results To top of page

Purpose: To verify that the activity has been completed appropriately and that the resulting artifacts are acceptable.

Now that you have completed the work, it is beneficial to verify that the work was of sufficient value, and that you did not simply consume vast quantities of paper. You should evaluate whether your work is of appropriate quality, and that it is complete enough to be useful to those team members who will make subsequent use of it as input to their work. Where possible, use the checklists provided in RUP to verify that quality and completeness are "good enough".

Have the people performing the downstream activities that rely on your work as input take part in reviewing your interim work. Do this while you still have time available to take action to address their concerns. You should also evaluate your work against the key input artifacts to make sure you have represented them accurately and sufficiently. It may be useful to have the author of the input artifact review your work on this basis.

Try to remember that that RUP is an iterative process and that in many cases artifacts evolve over time. As such, it is not usually necessary—and is often counterproductive—to fully-form an artifact that will only be partially used or will not be used at all in immediately subsequent work. This is because there is a high probability that the situation surrounding the artifact will change—and the assumptions made when the artifact was created proven incorrect—before the artifact is used, resulting in wasted effort and costly rework. Also avoid the trap of spending too many cycles on presentation to the detriment of content value. In project environments where presentation has importance and economic value as a project deliverable, you might want to consider using an administrative resource to perform presentation tasks.



Copyright  1987 - 2001 Rational Software Corporation


Display Rational Unified Process using frames

Rational Unified Process