IEEE Std 829 Software Test Plan (IEEE Standard for Software and System Test Documentation)
This article will examine the Test Plan in detail. We shall refer mainly to the IEEE Std 829-2008 (IEEE Standard for Software and System Test Documentation) in this article.
1. Test Plan definition
The IEEE Std 829 defines a Test Plan as, "(A) A document describing the scope, approach, resources, and schedule of intended test activities. It identifies test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency planning. (B) A document that describes the technical and management approach to be followed for testing a system or component. Typical contents identify the items to be tested, tasks to be performed, responsibilities, schedules, and required resources for the testing activity. (adopted from IEEE Std 610.12-1990 [B2])"
2. Why is a Test Plan required ?
The Test Plan is the fundamental document for testing. The purpose of the Master Test Plan, as stated by the IEEE Std 829 is to provide an overall test planning and test management document for multiple levels of test (either within one project or across multiple projects).
The Test Plan is a medium of communication, used by the testing team to communicate to the other project participants about the team's intent, expectations and understanding of the test activity to be performed. It is important that all the stakeholders of the project review and sign-off on the test plan. The Test Plan helps the testing team to set the right expectations about the intended test activities. It makes for sound business sense to have the review and sign-off happen before starting the test campaign. This will help in avoiding any mis-understanding around the testing effort as well as protect the testing team from potential complaints about the testing performed later in the project.
The Test Plan is a product of the test planning process. The Test Plan may be referred to both as a map and a blueprint for testing. It is a comprehensive document that offers clarity to all the project's stakeholders about testing. It addresses the how, what, when, who, where and why of testing and helps the testing group to focus.
3. What constitutes a Test Plan ?
The IEEE Std 829 provides an outline of the Master Test Plan (MTP). MTP involves selecting the constituent parts of the project's test effort, setting the objectives for each part, setting the division of labor in terms of time and resources, and the interrelationships between the parts, identifying the risks, assumptions, and standards of workmanship to be considered and accounted for by the parts, defining the test effort's controls and confirming the applicable objectives set by quality assurance planning. It also identifies the number of levels of test, the overall tasks to be performed and the documentation requirements.
Master Test Plan outlines1. Introduction
This section identifies the document and describes the entire test effort, including the test organization, test schedule and the system characteristics (such as complexity, risk, safety level, security level, desired performance, reliability, and/or cost) selected as important to stakeholders, and arranged into discrete levels of performance or compliance, to help define the level of quality control to be applied in developing and/or delivering the software. A summary of required resources, responsibilities, tools and techniques may also be included here.Identifier: This uniquely identifies the version of the document by including information such as the date of issue, author(s), approver signatures, status (e.g. draft, reviewed, corrected or final), reviewers and managers. This information may be placed either at the beginning or end of the document. Scope: This describes the purpose, goals and scope of the test effort. Identify the project(s) for which this plan is written and the specific processes and products covered by the test effort. Describe the inclusions, exclusions, assumptions and limitations. Test tasks will reflect the overall test approach and the development methodology. For example, if the development is based on the waterfall methodology, then each level of testing will be executed once. However, if the development is based on an iterative methodology, then there will be multiple iterations of each level of testing. The test approach identifies what will be tested and in what order for the gamut of testing levels such as component, component integration, system and acceptance. The test approach identifies the rationale for testing or not testing as also the selected order of testing. The test approach may also identify the type of testing performed at the different levels listed earlier.
References: This lists all applicable reference documents, both internal and external.
System overview and key features: This describes the purpose and key features of the system or software product under test or reference where the information can be found.Test overview: This describes the test organization, test schedule, system characteristics (such as complexity, risk, safety level, security level, desired performance, reliability, and/or cost) selected as important to stakeholders, test resources, responsibilities, tools, techniques and methods necessary to perform testing.
While describing the test organization, an organization chart may be included to clarify the reporting structure. Include information on the authority for resolving issues raised by testing, and the authority for approving test products and processes.
The test schedule describes the test activities within the project lifecycle and milestones. Summarize the overall schedule of testing tasks, identifying where the task results feed back to the development and related process such as quality assurance and configuration management. Also, describe the task iteration policy for re-execution of test tasks and any dependencies.
The test resources requirement should be summarized to include, staffing, facilities, tools and any special procedural requirements such as security, access rights, etc.
Include information on the responsibilities for testing tasks. Responsibilities may be primary (task owner / leader) or secondary (providing support) in relation to specific test related responsibilities.
Describe the hardware, software, test tools, techniques, methods and test environment to be used in testing. Include information pertaining to acquisition, training and support for each tool, technology and method. Include the metrics to be used by the testing effort.
2. Details
This section describes the test processes, test documentation and test reporting requirements.
Test processes and definition of test levels: Describe the test activities and tasks for all development lifecycle processes. List the number and sequence of levels of test. Levels of test may include component, integration, system, acceptance, security, usability, performance, stress, interoperability, regression, etc. Not all projects will have all the levels of test. Some projects may have fewer levels of test and could combine multiple levels. The test processes may either be described here or reference to already defined standards may be provided.
In addition, the IEEE Std 829 recommends that for each test activity, the following topics be addressed.
1. Test Plan definition
The IEEE Std 829 defines a Test Plan as, "(A) A document describing the scope, approach, resources, and schedule of intended test activities. It identifies test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency planning. (B) A document that describes the technical and management approach to be followed for testing a system or component. Typical contents identify the items to be tested, tasks to be performed, responsibilities, schedules, and required resources for the testing activity. (adopted from IEEE Std 610.12-1990 [B2])"
2. Why is a Test Plan required ?
The Test Plan is the fundamental document for testing. The purpose of the Master Test Plan, as stated by the IEEE Std 829 is to provide an overall test planning and test management document for multiple levels of test (either within one project or across multiple projects).
The Test Plan is a medium of communication, used by the testing team to communicate to the other project participants about the team's intent, expectations and understanding of the test activity to be performed. It is important that all the stakeholders of the project review and sign-off on the test plan. The Test Plan helps the testing team to set the right expectations about the intended test activities. It makes for sound business sense to have the review and sign-off happen before starting the test campaign. This will help in avoiding any mis-understanding around the testing effort as well as protect the testing team from potential complaints about the testing performed later in the project.
The Test Plan is a product of the test planning process. The Test Plan may be referred to both as a map and a blueprint for testing. It is a comprehensive document that offers clarity to all the project's stakeholders about testing. It addresses the how, what, when, who, where and why of testing and helps the testing group to focus.
3. What constitutes a Test Plan ?
The IEEE Std 829 provides an outline of the Master Test Plan (MTP). MTP involves selecting the constituent parts of the project's test effort, setting the objectives for each part, setting the division of labor in terms of time and resources, and the interrelationships between the parts, identifying the risks, assumptions, and standards of workmanship to be considered and accounted for by the parts, defining the test effort's controls and confirming the applicable objectives set by quality assurance planning. It also identifies the number of levels of test, the overall tasks to be performed and the documentation requirements.
Master Test Plan outlines1. Introduction
This section identifies the document and describes the entire test effort, including the test organization, test schedule and the system characteristics (such as complexity, risk, safety level, security level, desired performance, reliability, and/or cost) selected as important to stakeholders, and arranged into discrete levels of performance or compliance, to help define the level of quality control to be applied in developing and/or delivering the software. A summary of required resources, responsibilities, tools and techniques may also be included here.Identifier: This uniquely identifies the version of the document by including information such as the date of issue, author(s), approver signatures, status (e.g. draft, reviewed, corrected or final), reviewers and managers. This information may be placed either at the beginning or end of the document. Scope: This describes the purpose, goals and scope of the test effort. Identify the project(s) for which this plan is written and the specific processes and products covered by the test effort. Describe the inclusions, exclusions, assumptions and limitations. Test tasks will reflect the overall test approach and the development methodology. For example, if the development is based on the waterfall methodology, then each level of testing will be executed once. However, if the development is based on an iterative methodology, then there will be multiple iterations of each level of testing. The test approach identifies what will be tested and in what order for the gamut of testing levels such as component, component integration, system and acceptance. The test approach identifies the rationale for testing or not testing as also the selected order of testing. The test approach may also identify the type of testing performed at the different levels listed earlier.
References: This lists all applicable reference documents, both internal and external.
System overview and key features: This describes the purpose and key features of the system or software product under test or reference where the information can be found.Test overview: This describes the test organization, test schedule, system characteristics (such as complexity, risk, safety level, security level, desired performance, reliability, and/or cost) selected as important to stakeholders, test resources, responsibilities, tools, techniques and methods necessary to perform testing.
While describing the test organization, an organization chart may be included to clarify the reporting structure. Include information on the authority for resolving issues raised by testing, and the authority for approving test products and processes.
The test schedule describes the test activities within the project lifecycle and milestones. Summarize the overall schedule of testing tasks, identifying where the task results feed back to the development and related process such as quality assurance and configuration management. Also, describe the task iteration policy for re-execution of test tasks and any dependencies.
The test resources requirement should be summarized to include, staffing, facilities, tools and any special procedural requirements such as security, access rights, etc.
Include information on the responsibilities for testing tasks. Responsibilities may be primary (task owner / leader) or secondary (providing support) in relation to specific test related responsibilities.
Describe the hardware, software, test tools, techniques, methods and test environment to be used in testing. Include information pertaining to acquisition, training and support for each tool, technology and method. Include the metrics to be used by the testing effort.
2. Details
This section describes the test processes, test documentation and test reporting requirements.
Test processes and definition of test levels: Describe the test activities and tasks for all development lifecycle processes. List the number and sequence of levels of test. Levels of test may include component, integration, system, acceptance, security, usability, performance, stress, interoperability, regression, etc. Not all projects will have all the levels of test. Some projects may have fewer levels of test and could combine multiple levels. The test processes may either be described here or reference to already defined standards may be provided.
In addition, the IEEE Std 829 recommends that for each test activity, the following topics be addressed.
Test tasks: Identify the test tasks
Methods: Describe the methods and procedures for each test task, including tools. Also, define the criteria for evaluating the test task results
Inputs: Identify the required inputs for the test task. Specify the source of each input. Inputs may be derived from preceding tasks or activities
Outputs: Identify the required outputs from the test task
Schedule: Describe the schedule for the test tasks. Establish specific milestones for initiating and completing each task, for obtaining input and for delivery of output
Resources: Identify the resources for the performance of the test tasks. Example of resources include people, tools, equipment, facilities, budgets, etc.
Risks and Assumptions: Identify any risks and assumptions associated with the test tasks. Include recommendations to eliminate, reduce or mitigate risks identified
Roles and responsibilities: Identify for each task, who has the primary and secondary responsibilities for task execution and the nature of the roles they will play
Test documentation requirements: Here, define the purpose, format and content of all other testing documents that are to be used (in addition to those that are defined in the "Test reporting requirements" section)Test administration requirements: These are needed to administer tests during execution and involve describing the following.
Anomaly resolution and reporting process: Describe the method of reporting and resolving anomalies. This would include information about the anomaly criticality levels, authority and time line for resolution.
Task iteration policy: Describe the criteria for repeating testing tasks when its input is changed or task procedure is changed. Example, re-executing tests after anomalies have been fixed.
Deviation policy: Describe the procedures and criteria for deviation from the MTP and test documentation. The information for deviations includes task identification, rationale and effect on product quality. Also, identify the authorities responsible for approving deviations.
Control procedures: Identify control procedures for test activities. These procedures describe how the system, software products and test results will be configured, protected and stored. They may also describe quality assurance, configuration management, data management, compliance with existing security provisions and how test results are to be protected from unauthorized alterations.
Standards, practices and conventions: Identify the standards, practices and conventions that govern the performance of testing tasks.
Test reporting requirements: This specifies the purpose, content, format, recipients and timing of all test reports. Test reporting includes test logs, anomaly reports, level interim test status reports, level test reports, master test report and any optional reports defined during preparation of the test plan.
3. General
This section includes the glossary of terms, acronyms, description of the frequency and process by which the master test plan is changed and base-lined and may also contain a change page mentioning the history of changes (date, reason for change and who initiated the change).
A result of the test planning process must be a clear and agreed upon definition of the product's quality and reliability goals in absolute terms. There must be no room for subjective interpretation. Everyone on the project team must know what the testing team intends to do and the quality goals. Test planning is not about filling in a template or writing a document. Test planning is an important activity that must involve testers and representatives from all functions part of the project team. Getting everyone on the same page and on agreement regarding what is to be tested, why it is to be tested and how it is to be tested is key to testing success.
3. General
This section includes the glossary of terms, acronyms, description of the frequency and process by which the master test plan is changed and base-lined and may also contain a change page mentioning the history of changes (date, reason for change and who initiated the change).
A result of the test planning process must be a clear and agreed upon definition of the product's quality and reliability goals in absolute terms. There must be no room for subjective interpretation. Everyone on the project team must know what the testing team intends to do and the quality goals. Test planning is not about filling in a template or writing a document. Test planning is an important activity that must involve testers and representatives from all functions part of the project team. Getting everyone on the same page and on agreement regarding what is to be tested, why it is to be tested and how it is to be tested is key to testing success.
A popular template for Test Plan preparation is the format specified by the IEEE 829 standard for Software Test Documentation.
Before we look at the contents of the template, we should bear in mind that templates are broad guidelines which should not lead to users of the template to stop thinking and focus on just filling up the blanks in the template document. While using the template, one should understand the organization's requirements and evaluate if the template fits your specific requirements or needs any modifications. Sticking to the stock template may result in some information which needs to be captured being left out.
With that short note, lets look at the template itself. The IEE 829 Test plan template includes the following sections.
Before we look at the contents of the template, we should bear in mind that templates are broad guidelines which should not lead to users of the template to stop thinking and focus on just filling up the blanks in the template document. While using the template, one should understand the organization's requirements and evaluate if the template fits your specific requirements or needs any modifications. Sticking to the stock template may result in some information which needs to be captured being left out.
With that short note, lets look at the template itself. The IEE 829 Test plan template includes the following sections.
Test plan identifier : A unique name by which the test plan may be identified and may include version information
Introduction : Summary of the test plan, including type of testing, level of testing (master test plan, component test plan, unit test plan ...), any references to other documents, scope of testing and so on
Test items : The artifacts that will be tested
Features to be tested : The features or items of the specification that will be tested
Features not to be tested : The features or items part of the specification that will not be tested
Approach : Addresses “how” the testing will be performed
Item pass/fail criteria : This could be viewed as the criteria for completion of testing per this plan.
Suspension criteria and resumption requirements : List the criterion for pausing or resumption of testing
Test deliverable : The artifacts created by the testing team that will be delivered as per this plan. Examples include - test cases, test design specifications, output from tools, test reports, etc.
Testing tasks : The testing tasks involved, their dependencies if any, time they will take and resource requirements
Environmental needs : List needs such as hardware, software and other environmental requirements for testing
Responsibilities : List the people responsible for the various parts of the plan
Staffing and training needs : The people & skill sets needed to carry out the test activities
Schedule : List the schedule dates when testing will take place. A safe bet is to tie the schedule to the development schedule in a relative manner without listing hard dates since slippages upstream in development will mean that testing slips correspondingly. Hard dates would result in any development slippages causing compression of testing time.
Risks and contingencies : Identify the risks, likelihood and impact as well as possible mitigation steps
Approvals : Sign-off by the stakeholders denoting agreement
QA is not Testing (QA vs QC)
QA, QC, Testing – more often used inter-changeably and generally meant to imply "Testing".
Lets get our facts straight - QA is not Testing; QC is not only about Testing; Testing is QC.
More explanation follow.
Quality Control (QC) - is oriented towards detecting defects and correction of these defects. QC works on the product rather than the process of producing the product. QC involves a set of tasks carried out to evaluate the product that has been developed. QC is normally the responsibility of the testing team and is considered to be a line function. Although testing is a QC activity, it is not the only type of QC activity. QC includes any activity that examines products to determine if they meet their requirements. Examples of QC activities apart from testing – inspections, reviews, walk-throughs of work products like requirements, designs, code and documentation.
Quality Assurance (QA) - is oriented towards defect prevention and focuses on the process by which the product or application is built. QA involves a set of tasks to ensure that the development process is adequate to produce a system that meets its requirements. QA activities include reviewing design activities, setting standards to be followed in coding and such other process requirements aimed at ensuring that a quality product is built. QA ensures that the process is well defined and looks at methodology and standards development. QA is performed through the life cycle of the product and applies to all involved in developing the product. QA is normally considered to be a staff function. QA looks at items such as identifying areas for improvements in current methods and processes being followed, making processes effective, ensuring consistency in the way these are followed and so on. While QC evaluates the product, QA evaluates the activities involved in creating the product.
Lets get our facts straight - QA is not Testing; QC is not only about Testing; Testing is QC.
More explanation follow.
Quality Control (QC) - is oriented towards detecting defects and correction of these defects. QC works on the product rather than the process of producing the product. QC involves a set of tasks carried out to evaluate the product that has been developed. QC is normally the responsibility of the testing team and is considered to be a line function. Although testing is a QC activity, it is not the only type of QC activity. QC includes any activity that examines products to determine if they meet their requirements. Examples of QC activities apart from testing – inspections, reviews, walk-throughs of work products like requirements, designs, code and documentation.
Quality Assurance (QA) - is oriented towards defect prevention and focuses on the process by which the product or application is built. QA involves a set of tasks to ensure that the development process is adequate to produce a system that meets its requirements. QA activities include reviewing design activities, setting standards to be followed in coding and such other process requirements aimed at ensuring that a quality product is built. QA ensures that the process is well defined and looks at methodology and standards development. QA is performed through the life cycle of the product and applies to all involved in developing the product. QA is normally considered to be a staff function. QA looks at items such as identifying areas for improvements in current methods and processes being followed, making processes effective, ensuring consistency in the way these are followed and so on. While QC evaluates the product, QA evaluates the activities involved in creating the product.
Defect classification
Here's a pictorial representation of a defect management lifecycle, that's mapped to the anomaly classification process proposed by the IEEE Standard Classification for Software Anomalies (IEEE Std. 1044-1993).
The IEEE 1044-1993 definition of anomaly: "Any condition that deviates from expectations based on requirements speciļ¬cations, design documents, user documents, standards, etc. or from someone’s perceptions or experiences. Anomalies may be found during, but not limited to, the review, test, analysis, compilation, or use of software products or applicable documentation."
The process is divided into the following four sequential steps.
The IEEE 1044-1993 definition of anomaly: "Any condition that deviates from expectations based on requirements speciļ¬cations, design documents, user documents, standards, etc. or from someone’s perceptions or experiences. Anomalies may be found during, but not limited to, the review, test, analysis, compilation, or use of software products or applicable documentation."
The process is divided into the following four sequential steps.
Step 1: Recognition
Step 2: Investigation
Step 3: Action
Step 4: Disposition
Example Defect Lifecycle
Test Metrics
Introduction
Test metrics accomplish in analyzing the current level of maturity in testing and give a projection on how to go about testing activities by allowing us to set goals and predict future trends.
No comments:
Post a Comment