NATIONAL SOFTWARE TESTING GUIDELINE 2019

 
NATIONAL SOFTWARE TESTING GUIDELINE 2019
NATIONAL SOFTWARE TESTING GUIDELINE
               2019
National Software Testing Guideline

Table of Contents
TERMS AND DEFINITIONS ...................................................................................................................... 4
EXECUTIVE SUMMARY ......................................................................................................................... 19
1       INTRODUCTION ............................................................................................................................ 20
    1.1        Authority............................................................................................................................... 20
    1.2        Scope ..................................................................................................................................... 20
    1.3        Benefits of a National Software Testing Guideline ............................................................. 20
    1.4        Objectives of The National Software Testing Guideline ..................................................... 21
2       THE SOFTWARE DEVELOPMENT LIFECYCLE (SDLC) ..................................................................... 21
    2.1        The Software Testing Life Cycle ........................................................................................... 23
    2.2        Software Testing Principles .................................................................................................. 24
    2.3        Project Organisation Structure ............................................................................................ 24
    2.4        Minimum Requirements to become a Software Test Professional .................................... 26
3       SOFTWARE TESTING GUIDELINE APPROACH............................................................................... 26
    3.1        Organisational Test Process ................................................................................................. 27
    3.1.1      Organisational Test Specifications ....................................................................................... 27
    3.1.1.2 Purpose................................................................................................................................. 28
    3.1.2      Practical Guideline................................................................................................................ 29
    3.2        Test Management Processes ............................................................................................... 30
    3.2.2      Practical Guidelines .............................................................................................................. 31
    3.3        Test Planning Process ........................................................................................................... 31
    3.3.1      Test Planning Input Requirements ...................................................................................... 31
    3.3.2      Practical Guideline................................................................................................................ 36
    3.3.3      Information Items ................................................................................................................ 40
     .......................................................................................................................................................... 41
    3.4        Test Monitoring and Control Process .................................................................................. 42
    3.4.1      Overview ............................................................................................................................... 42
    3.4.2      Purpose ................................................................................................................................. 43
    3.4.3      Outcomes.............................................................................................................................. 43
    3.4.4      Practical Guideline................................................................................................................ 43
    3.4.5      Information Items ................................................................................................................ 44
    3.5        Test Completion Process ...................................................................................................... 45
    3.5.1      Overview ............................................................................................................................... 45
    3.5.2      Purpose ................................................................................................................................. 45
    3.5.3 Outcome ................................................................................................................................ 45
    3.5.4 Activities and Tasks ............................................................................................................... 45
                                                                                                                                 1|Page
National Software Testing Guideline

    3.5.5       Practical Guideline............................................................................................................... 45
    3.5.6       Information Items................................................................................................................ 46
    3.6        Dynamic Testing Process ...................................................................................................... 46
    3.6.2      Test Design and Implementation Process ........................................................................... 47
    3.6.3      Purpose ................................................................................................................................. 47
    3.6.4      Practical Guidelines .............................................................................................................. 48
    3.6.5      Information Items ................................................................................................................ 49
    3.7        Test Environment Set-Up and Maintenance ....................................................................... 49
    3.7.1 Overview................................................................................................................................ 49
    3.7.2 Purpose .................................................................................................................................. 50
    3.7.3 Outcomes............................................................................................................................... 50
    3.7.4 Activities and tasks................................................................................................................ 50
    3.7.5 Practical Guideline ................................................................................................................ 50
    3.7.6 Information Items ................................................................................................................. 50
    3.8        Test Execution Process ......................................................................................................... 51
    3.8.1      Overview ............................................................................................................................... 51
    3.8.2      Purpose ................................................................................................................................. 51
    3.8.3 Outcomes............................................................................................................................... 51
    3.8.4 Activities and tasks................................................................................................................ 51
    3.8.5 Practical Guideline ................................................................................................................ 51
    3.8.6 Information Items ................................................................................................................. 52
    3.9        Test Incident Report Process ............................................................................................... 52
    3.9.1 Overview................................................................................................................................ 52
    3.9.2 Purpose .................................................................................................................................. 52
    3.9.3 Outcomes............................................................................................................................... 52
    3.9.4 Activities and tasks................................................................................................................ 52
    3.9.5 Practical Guideline ................................................................................................................ 53
    3.10       Software Testing Practical Steps .......................................................................................... 53
4       LEVELS OF TESTING ....................................................................................................................... 58
    4.1        Unit Testing........................................................................................................................... 59
    4.1.1      Purpose ................................................................................................................................. 59
    4.2        Integration Testing ............................................................................................................... 59
    4.2.1      Purpose ................................................................................................................................. 59
    4.3        System Testing...................................................................................................................... 59
    4.3.1      Purpose ................................................................................................................................. 60
    4.4        Acceptance Testing............................................................................................................... 60

                                                                                                                                          2|Page
National Software Testing Guideline

    4.4.1      Purpose ................................................................................................................................. 60
    4.5        Software Test Automation Tools ......................................................................................... 61
5      TEST TECHNIQUES ........................................................................................................................ 62
    5.1        Specification-Based Testing Techniques (Black-Box Testing Techniques) ......................... 62
    5.2        Structure-Based Testing Techniques (White Box Testing Techniques) .............................. 62
    5.2.1      White Box Testing Techniques ............................................................................................. 62
    5.3        Experience-Based Testing Technique .................................................................................. 63
6      TEST DOCUMENTATION ............................................................................................................... 63
    6.1        Overview ............................................................................................................................... 63
    6.2        Organisational Test Policy Documentation ......................................................................... 64
    6.2.1      Test Policy Document Template .......................................................................................... 64
    6.3        Organisational Test Strategy Documentation ..................................................................... 65
    6.3.1      Organisational Test Strategy Template ............................................................................... 65
    6.4        Test Plan Documentation ..................................................................................................... 67
    6.4.1      Test Plan Template ............................................................................................................... 67
    6.5        Test Status Report Documentation ..................................................................................... 70
    6.5.1      Test Status Report Template ............................................................................................... 70
    6.6        Test Completion Report Documentation ............................................................................ 71
    6.6.1      Test Completion Report Template....................................................................................... 71
    6.7        Test Design Specification Documentation ........................................................................... 73
    6.7.1      Test Design Specification Template ..................................................................................... 73
    6.8        Test Case Specification Documentation .............................................................................. 74
    6.8.1      Test Case Specification Templates ....................................................................................... 74
    6.9        Test Procedure Specification Documentation ..................................................................... 75
    6.9.1      Test Procedure Specification Template ............................................................................... 75
    6.10       Test Data Requirements Documentation ............................................................................ 76
    6.10.1 Test Data Requirements Template ...................................................................................... 76
    6.11       Test Environment Requirements Documentation .............................................................. 76
    6.11.1 Test Environment Requirements Template ........................................................................ 77
    6.12       Test Execution Log Documentation ..................................................................................... 77
    6.12.1 Test Execution Log Template ............................................................................................... 77
    6.13       Incident Report Documentation .......................................................................................... 77
    6.13.1 Incident Report Template .................................................................................................... 77
APPEDIX A ............................................................................................................................................. 79
    6.14       Software Testing Certifications ............................................................................................ 79
REFERENCES .......................................................................................................................................... 80

                                                                                                                                          3|Page
National Software Testing Guideline

TERMS AND DEFINITIONS

                                      4|Page
National Software Testing Guideline

Actual Results                  Set of behaviours or conditions of a test item, or set of
                                conditions of associated data or the test environment,
                                observed as a result of test execution
                                EXAMPLE: outputs to hardware, changes to data, reports,
                                and communication messages sent
BSI                             British Standards Institution, the national standards body
                                of the United Kingdom
CEO                             Chief Executive Officer, the most senior executive with
                                responsibility for running an organization
CFO                             Chief Financial Officer, a senior executive with
                                responsibility for the financial affairs of an organization
Compliance                      A general concept of conforming to a rule, standard, law,
                                or requirement such that the assessment of compliance
                                results in a binomial result stated as "compliant" or
                                "noncompliant"
                                (A Guide to the Project Management Body of Knowledge
                                (PMBOK® Guide) -- Fifth Edition)
                                EXAMPLE: comply with a regulation.
Conformance                     Fulfilment by a product, process or service of specified
                                requirements
Defect Management               A test practice involving the investigation and
                                documentation of test incident reports and the retesting
                                of defects when required
Dynamic Testing                 (1) Testing that requires the execution of the test item
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering-- Software testing--Part 1: Concepts and
                                definitions, 4.9)
                                (2) Testing that requires the execution of program code
                                (ISO/IEC/IEEE 29119-2:2013 Software and systems
                                engineering-- Software testing--Part 2: Test processes, 4.4)
Exhaustive Testing              A testing approach where every possible combination of

                                                                                        5|Page
National Software Testing Guideline

                                input value is executed (infeasible for all practical test
                                items)
Expected Results                Observable predicted behaviour of the test item under
                                specified conditions based on its specification or another
                                source
                                (ISO/IEC/IEEE 29119- 1:2013 Software and systems
                                engineering-- Software testing--Part 1: Concepts and
                                definitions, 4.15)
Experience-Based Testing        A testing practice, in which testing is based on the tester's
                                previous experience, such as their domain knowledge and
                                knowledge of particular software and systems, along with
                                metrics from previous projects
Feature Set                     (1) Collection of items which contain the test conditions of
                                the test item to be tested which can be collected from
                                risks, requirements, functions, models, etc.
                                (ISO/IEC/IEEE 29119- 1:2013 Software and systems
                                engineering-- Software testing--Part 1: Concepts and
                                definitions)
                                (2) Logical subset of the test item(s) could be treated
                                independently of other feature sets in the subsequent test
                                design activities
                                (ISO/IEC/IEEE 29119-2:2013 Software and systems
                                engineering-- Software testing --Part 2: Test processes,
                                4.10)
ICT                             Information and Communication Technology
                                (ISO/IEC/IEEE 23026:2015 Systems and software
                                engineering--Engineering and management of websites for
                                systems, software, and services information, 5)
IEC                             The International Electrotechnical Commission is a global
                                organization for the preparation and publication of
                                International Standards for all electrical electronic and

                                                                                        6|Page
National Software Testing Guideline

                                related technologies.

IEEE                            The Institute of Electrical and Electronics Engineers in the
                                world's largest technical professional organization
                                dedicated to educational and technical advancement of
                                electrical and electronic engineering, telecommunications,
                                computer engineering and allied disciplines.
Incident Report                 Documentation of the occurrence, nature, and status of
                                an incident
                                (ISO/IEC/IEEE 29119- 1:2013 Software and systems
                                engineering--Software testing--Part 1: Concepts and
                                definitions, 4.18)
Information Item                Separate identifiable body of information that is
                                produced, stored, and delivered for human use
                                (ISO/IEC/IEEE 15289:2015 Systems and software
                                engineering--Content of life-cycle information products
                                (documentation). 5.11)
ISO                             The International Organization for Standardization (ISO) is
                                an international standard-setting body composed of
                                representatives from various national standards
                                organizations.
Lessons Learned                 The knowledge gained during a project which shows how
                                project events were addressed or should be addressed in
                                the future with the purpose of improving future
                                performance
                                (A Guide to the Project Management Body of Knowledge
                                (PMBOK ® Guide) -- Fifth Edition)
One-to-One Test Case            An approach to test case design, where one test case is
Design                          derived for each test coverage item
Organizational Test             A document that provides information about testing for
Specification                   an organization. i.e. information that is not project specific
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems

                                                                                       7|Page
National Software Testing Guideline

                                engineering-- Software testing--Part 1: Concepts and
                                definitions, 4.24)

Organizational Test Strategy    Documents that express the generic requirements for the
                                testing to be performed on all the projects run within the
                                organization, providing detail on how the testing is to be
                                performed
                                (ISO/IEC/IEEE 29119- 1:2013 Software and systems
                                engineering-- Software testing--Part 1: Concepts and
                                definitions, 4.25)
Product Risk                    Risk that a product could be defective in some specific
                                aspect of its function, quality, or structure
                                ISO/IEC/IEEE 29119-3:2013 Software and systems
                                engineering--Software testing-- Part 3: Test
                                documentation, 4.8)
Project Risk                    Risk related to the management of a project
                                ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering-- Software testing--Part 1: Concepts and
                                definitions, 4.31)
                                EXAMPLE: lack of staffing, strict deadlines, changing
                                requirements
Regression Testing              Testing following modifications to a test item or to its
                                operational environment, to identify whether regression
                                failures occur
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering-- Software testing--Part 1: Concepts and
                                definitions, 4.32)
Requirements-Based Testing An approach to deriving test cases based on exercising
                                specified user requirements

                                                                                        8|Page
National Software Testing Guideline

Retesting                       Re-execution of test cases that previously returned a "fail"
                                result, to evaluate the effectiveness of interviewing
                                corrective actions
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering--Software testing-- Part 1: Concepts and
                                definitions, 4.34)
                                SYN: confirmation testing
Risk                            Combination of the probability of an abnormal event or
                                failure and the consequence(s) of that event or failure to a
                                system's components, operators, users, or environment
                                (IEEE 1012-2012 IEEE Standard for System and Software
                                Verification and Validation, 3.1)
Risk Exposure                   Potential loss presented to an individual, project, or
                                organization by a risk
                                (ISO/IEC 16085:2006 Systems and software engineering--
                                Life cycle processes--Risk management, 3.10)
Risk Register                   A document in which the results of risk analysis and risk
                                response planning are recorded
                                (A Guide to the Project Management Body of Knowledge
                                (PMBOK® Guide) -- Fifth Edition)
                                NOTE: The risk register details all identified risks, including
                                description, category, cause, probability of occurring,
                                impact(s) on objectives, proposed responses, owners, and
                                current status. It can be kept in a database. See Also: risk
                                management plan
Risk-Based Testing              Testing in which the management, selection, prioritization,
                                and use of testing activities and resources are consciously
                                based on corresponding types and levels of analyzed risk
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering-- Software testing--Part 1: Concepts and
                                definitions,4.35)

                                                                                         9|Page
National Software Testing Guideline

Scripted Testing                (1) Dynamic testing in which the tester's actions are
                                prescribed by written instructions in a test case
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering--Software testing--Part 1: Concepts and
                                definitions, 4.37)
                                (2) Testing performed based on a documented test script
                                (ISO/IEC/IEEE 29119-2:2013 Software and systems
                                engineering--Software testing--Part 2: Test Processes,
                                4.23)
                                NOTE: Normally applies to manually executed testing,
                                rather than the execution of an automated script
Static Testing                  Testing in which a test item is examined against a set of
                                quality or other criteria without code being executed
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering--Software testing--Part 1: Concepts and
                                definitions, 4.42)
                                EXAMPLE: reviews, static analysis
                                SEE ALSO: inspection
Test Automation                 A test practice involving the automation of testing using
                                software that is usually referred to as test tools
                                NOTE: Automated testing is often considered to be mainly
                                concerned with the execution of scripted tests rather than
                                have the testers execute tests manually, however many
                                additional testing activities can be supported by software-
                                based tools.
Test Basis                      Body of knowledge used as the basis for the design of
                                tests and test cases
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering --Software testing--Part 1: Concepts and
                                definitions, 4.47)
                                NOTE: The test basis can take the form of documentation,

                                                                                     10 | P a g e
National Software Testing Guideline

                                such as a requirements specification, design specification,
                                or module specification, but can also be an undocumented
                                understanding of the required behaviour.

Test Case                       Set of test case preconditions, inputs (including actions,
                                where applicable), and expected results, developed to
                                drive the execution of a test item to meet test objectives,
                                including correct implementation, error identification,
                                checking quality, and other valued information
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering -- Software testing--Part 1: Concepts and
                                definitions, 4.48)
                                NOTE: A test case is the lowest level of test input (i.e. test
                                cases are not made up of test cases) for the test sub
                                process for which it is intended.
Test Completion Criteria        A set of criteria to determine whether a specific test sub-
                                process can be considered as completed
                                EXAMPLE: 100% of statements exercised by test cases
                                EXAMPLE: Zero critical defects remain open
Test Completion Report          A report that summarizes the testing that was performed
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering--Software testing--Part 1: Concepts and
                                definitions, 4.51)
                                SYN: test summary report
Test Condition                  Testable aspect of a component or system, such as a
                                function, transaction, feature, quality attribute, or
                                structural element identified as a basis for testing
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering--Software testing--Part 1: Concepts and
                                definitions, 4.52)

                                                                                        11 | P a g e
National Software Testing Guideline

Test Coverage Item              An attribute or combination of attributes that is derived
                                from one or more test conditions by using a test design
                                technique that enables the measurement of the
                                thoroughness of the test execution
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering--Software testing--Part 1: Concepts and
                                definitions, 4.54)
Test Data                       Data created or selected to satisfy the input requirements
                                for executing one or more test cases, which can be
                                defined in the test plan, test case, or test procedure
                                (ISO/IEC/IEEE 29119-2:2013 Software and systems
                                engineering-- Software testing--Part 2: Test processes,
                                4.34)
Test Environment                Facilities, hardware, software, firmware, procedures and
                                documentation intended for or used to perform testing of
                                software
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering--Software testing--Part 1: Concepts and
                                definitions, 4.60)
Test Execution                  A process of running a test on the test item, producing
                                actual results
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering-- Software testing--Part 1: Concepts and
                                definitions, 4.64)
Test Execution Log              A document that records details of the execution of one or
                                more test procedures
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering--Software testing--Part 1: Concepts and
                                definitions, 4.65)

                                                                                     12 | P a g e
National Software Testing Guideline

Test Item                       Work product that is an object of testing
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering--Software testing--Part 1: Concepts and
                                definitions, 4.68)
                                EXAMPLE: A system, a software item, a requirements
                                document, a design specification, a user guide
                                SYN: test object
Test Level                      Specific instantiation of a test sub-process
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering--Software testing--Part 1: Concepts and
                                definitions, 4.69)
                                EXAMPLE: Component, component integration, system,
                                and acceptance testing
                                SYN: test phase
Test Management                 Planning, estimating, monitoring, reporting, control and
                                completion of test activities
                                (ISO/IEC/IEEE 29119-2:2013 Software and systems
                                engineering--Software testing--Part 2: Test processes,
                                4.49)
Test Measures                   Variable to which a value is assigned as the result of
                                measuring an attribute of the testing
                                EXAMPLE: 80% branch coverage
Test Objective                  Purpose of performing a testing activity associated with a
                                feature or feature set
                                EXAMPLE: Provision of information about product
                                qualities
Test Phase                      Specific instantiation of a test sub-process
                                (ISO/IEC/IEEE 29119-2:2013 Software and systems
                                engineering--Software testing--Part 2: Test Processes,
                                4.52)
                                SYN: test phase

                                                                                    13 | P a g e
National Software Testing Guideline

Test Plan                       Detailed description of test objectives to be achieved and
                                the means and schedule for achieving them, organized to
                                coordinate testing activities for some test item or set of
                                items
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering--Software testing--Part 1: Concepts and
                                definitions, 4.75)
                                EXAMPLE: A project test plan (also known as a master test
                                plan) that encompasses all testing activities on the project;
                                further detail of particular test activities can be defined in
                                one or more test sub-process plans (i.e. a system test plan
                                or a performance test plan)
                                NOTE: It identifies test items, the features to be tested,
                                the testing tasks, who will do each task, and any risks
                                requiring contingency planning. Typical contents identify
                                the items to be tested, tasks to be performed,
                                responsibilities, schedules, and required resources for the
                                testing activity.

Test Policy                     Executive-level document that describes the purpose,
                                goals, principles and scope of testing within an
                                organization
                                (ISO/IEC/IEEE 29119-3:2013 Software and systems
                                engineering--Software testing--Part 3: Test
                                documentation, 4.26)
                                SYN: organizational test policy
Test Procedure                  Sequence of test cases in execution order, associated
                                actions to set up the initial preconditions, and wrap-up
                                activities post execution
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering--Software testing--Part 1: Concepts and

                                                                                      14 | P a g e
National Software Testing Guideline

                                definitions, 4.78)

Test Procedure Specification Document specifying one or more test procedures, which
                                are collections of test cases to be executed for a particular
                                objective
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering-- Software testing--Part 1: Concepts and
                                definitions, 4.79)
                                NOTE: A test procedure specification for an automated
                                test run is usually called a test script.
Test Process                    Process that provides information on the quality of a
                                software product
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering-- Software testing--Part 1: Concepts and
                                definitions. 4.80)
                                NOTE: Often comprised of a number of activities, grouped
                                into one or more test sub-processes

Test Result                     Indication of whether or not a specific test case has
                                passed or failed, i.e. if the actual result observed as test
                                item output corresponds to the expected result or if
                                deviations were observed
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering-- Software testing-- Part 1: Concepts and
                                definitions, 4.82)
Test Script                     Test procedure specification for manual or automated
                                testing
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering-- Software testing-- Part 1: Concepts and
                                definitions, 4.83)

                                                                                      15 | P a g e
National Software Testing Guideline

Test Set                        (1) Set of one or more test cases with a common
                                constraint on their execution
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering-- Software testing-- Part 1: Concepts and
                                definition, 4.84)
                                (2) Collection of test cases for the purpose of testing a
                                specific test objective
                                (ISO/IEC/IEEE 29119-2:2013 Software and systems
                                engineering-- Software testing-- Part 2: Test processes,
                                4.62)
                                NOTE: The test sets will typically reflect the feature sets,
                                but they could contain test cases for a number of feature
                                sets. Test cases for a test set could be selected based on
                                the identified risks, test basis, retesting, or regression
                                testing.
Test Status Report              Report that provides information about the status of the
                                testing that is being performed in a specified reporting
                                period
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering-- Software testing-- Part 1: Concepts and
                                definitions, 4.86)
Test Strategy                   Part of the Test Plan that describes the approach to
                                testing for a specific test project or test sub-process or
                                sub-processes
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering-- Software testing-- Part 1: Concepts and
                                definitions, 4.87)
                                NOTE: The test strategy usually describes some or all of
                                the following: the test practices used; the test sub
                                processes to be implemented; the retesting and
                                regression testing to be employed; the test design

                                                                                       16 | P a g e
National Software Testing Guideline

                                techniques and corresponding test completion criteria to
                                be used; test data; test environment and testing tool
                                requirements; and expectations for test deliverables.

Test Sub-process                Test management and dynamic (and static) test processes
                                used to perform a specific test level (e.g. system testing,
                                acceptance testing) or test type (e.g. usability testing,
                                performance testing) normally within the context of an
                                overall test process for a test project
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering-- Software testing-- Part 1: Concepts and
                                definitions, 4.88)
                                NOTE: Depending on the life cycle model used, test sub-
                                processes are also typically called test phases, test levels,
                                test stages or test tasks.
                                SYN: test sub process
Test Type                       Group of testing activities that are focused on specific
                                quality characteristics
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering-- Software testing-- Part 1: Concepts and
                                definitions, 4.91)
                                Example: security testing, functional testing, usability
                                testing, performance testing
Traceability Matrix             Matrix that records the relationship between two or more
                                products of the development process
                                (ISO/IEC/IEEE 24765:2017 Systems and software
                                engineering-Vocabulary)
                                EXAMPLE: a matrix that records the relationship between
                                the requirements and the design of a given software
                                component

                                                                                      17 | P a g e
National Software Testing Guideline

Unscripted Testing              Dynamic testing in which the tester's actions are not
                                prescribed by written instructions in a test case
                                (ISO/IEC/IEEE 29119-1:2013 Software and systems
                                engineering-- Software testing-- Part 1: Concepts and
                                definitions, 4.94)
Validation                      Confirmation, through the provision of objective evidence,
                                that the requirements for a specific intended use or
                                application have been fulfilled
                                (ISO/IEC 25000:2014 Systems and software Engineering--
                                Systems and software product Quality Requirements and
                                Evaluation (SQuaRE)-- Guide to SQuaRE, 4.41
Verification                    Confirmation, through the provision of objective evidence,
                                that specified requirements have been fulfilled
                                (ISO/IEC 25000:2014 Systems and software Engineering--
                                Systems and software product Quality Requirements and
                                Evaluation (SQuaRE)-- Guide to SQuaRE, 4.64

                                                                                    18 | P a g e
National Software Testing Guideline

EXECUTIVE SUMMARY
The importance of software testing cannot be over emphasised in today’s rapidly growing
technological environment; according to Forester publication (2018) the global tech market will grow
from 3.4% to 4% in 2017 and 2018 respectively; reaching a three (3) trillion dollar high in 2018. The
acquisition of technology driven solutions to drive growth in the public and private sectors have
continued to increase in Nigeria. The indigenous software market has not been left out of this growth
trend, but continues to suffer stiff competition from foreign off-the-shelf software used to meet local
needs, where indigenous software could have provided the appropriate solutions. According to
industry analysts, this trend has affected growth of the local software sector, which by now should be
in excess of probably ten (10) billion dollars annually if well harnessed. However, quality assurance
challenges amongst other factors have contributed to low adoption of indigenous software products
in the country.

In view of this, the National Information Technology Development Agency (NITDA) advocates a
structured approach to software quality assurance through a holistic software testing framework. The
National Software Testing Guideline (NSTG) adopts a multi-layer approach to software testing, which
is easily adaptable to any software domain to improve the quality of software and reduce cost of
development. Developed with the public and private sector in mind, the NSTG provides a practical
guide towards software testing as defined by the ISO/IEC/IEEE 29119 series of standards.

The guideline provides a brief introduction of NITDA’s purpose, its scope and objectives. It identifies
the relationship and importance of software testing in the Software Development Life Cycle (SDLC)
and introduces the multi-layer approach to testing, principles of software testing and the project
organisation structure.

The third section of the guideline focuses on the test approach, detailing the tree processes defined
by ISO/IEC/IEEE/29119; which are the Organisational Test Process, Test Management Processes and
Dynamic Test Process. It provides activities and tasks required during the test processes and practical
steps to implement software testing from planning to test execution.

Section four provides the important concept of testing levels, where every unit or component of
software is tested for defects and conformance with the intent of the system.

The fifth section provides details on testing techniques and its role in software testing. The final
section of the guideline provides details on test documentation and how it fits into the multi-layer
testing approach.

                                                                                           19 | P a g e
National Software Testing Guideline

1   INTRODUCTION
In a bid to create the necessary enabling environment for Information Technology (IT) for national
growth and development, the National Information Technology Development Agency (NITDA) is
focused on developing and regulating the IT sector in Nigeria, through the development of
appropriate guidelines and frameworks, to promote trust, efficiency, accuracy, reliability of
information systems in delivering information technology services. The National Software Testing
Guideline (NSTG) is developed on the premise that mitigating software vulnerability risks through the
promotion of structured software testing practice for safety and quality of software development will
create the enabling environment for growth of the indigenous testing sector in Nigeria.

The NSTG is developed based on international software testing standards which provides support for
a wide range of application areas in any software development life cycle; thus, being generic and can
be applied to different software domains.

1.1 Authority
In exercise of the mandate conferred on NITDA, specifically by section 6(b) and 6(e) of the National
Information Technology Agency Act of 2007; the National Software Testing Guideline is hereby issued.

1.2 Scope
This document shall play the specific role of ensuing a National Software Testing Guideline, developed
in line with international standard practices for software testing teams in initiating, planning,
managing, implementing and maintaining testing for software by public and private organizations;
and applicable to the testing of software developed, purchased or modified in-house.

1.3 Benefits of a National Software Testing Guideline
Every guide or standard brings about its own specific values and inherent benefits. This is obviously
the case with the NSTG, which stands to attain the following benefits among others:
           Assist NITDA to achieve its slated mandate as required by Law, through effective
            regulation of software practices in Nigeria;
           It provides a standard guideline in identifying and fixing errors before the software
            becomes operational, by reducing the risk of software failures considerably, which would
            improve quality of software products developed in the country; thereby increasing user
            satisfaction and demand for local software products;
           Reduces cost of mitigation using structured risk-based techniques for testing and
            maintaining software development and products;

                                                                                          20 | P a g e
National Software Testing Guideline

           Improved reliability of software product by following set software testing technical
            guidelines;
           Improves integration of legacy and varying software platforms by providing assurance
            that integration would work optimally with no performance retardation through
            structured testing processes;
           Helps to reduce production time spent on software integration for public and private
            sectors; and,
           Helps attain professional discipline in software testing practices in the country.

1.4 Objectives of The National Software Testing Guideline
The objectives of this document are:

           To provide the software industry with a structured software testing guideline, which
            offers best practices in software testing to promote safety and quality of software
            development in Nigeria; and,
           To provide an enabling environment for indigenous software testing administration,
            management and sustainability, through structured testing guidelines to improve quality,
            availability, reliability and reduce cost of software developed in Nigeria.

2 THE SOFTWARE DEVELOPMENT LIFECYCLE (SDLC)
The Software Development Life Cycle (SDLC) are phases in software development with an initiation
phase through to system maintenance. Each phase produces outputs required by the next phase in
the lifecycle. A typical SDLC is initiated by gathering business requirements, which are then analysed
and translated into a design; codes are then produced based on the design specification
(development phase). Software testing typically starts after coding is complete and before software is
deployed for use (deployment phase). During the deployment phase, the software is maintained
(maintenance/operation phase) until it is retired. There are many SDLC models used during software
development process, which include:

           Waterfall model
           Incremental model
           V-model
           Iterative model
           RAD model
           Agile model

                                                                                            21 | P a g e
National Software Testing Guideline

           Spiral model
           Prototype model

The NSTG is not developed upon any specific SDLC model but built in line with the ISO/IEC/IEEE 29119
standard which defines test terminology, processes, documentation and techniques that can be
adopted within the testing phase of any SDLC model. The following phases are predominately found
in all SDLC models:

           Requirement gathering and analysis phase
           Design phase
           Implementation phase
           Testing phase
           Deployment phase
           Maintenance Phase

                      SDLC                                                STLC
    REQUIREMENT & ANALYSIS PHASE                              REQUIREMENT SPECIFICATION

               DESIGN PHASE                                          TEST PLANNING

              CODING PHASE                                      TEST CASE DEVELOPMENT

              TESTING PHASE                                    TEST ENVIRONMENT SET-UP

            DEPLOYMENT PHASE                                         TEST EXECUTION

            MAINTENNCE PHASE                                       TEST CYCLE CLOSURE

Figure 1: SDLC and STLC Relationship Diagram

                                                                                       22 | P a g e
National Software Testing Guideline

The software testing phase starts after the codes have been developed, which are in turn tested
against the system requirement specification (test basis) gathered during the SDLC requirement
phase to make sure the software conforms with the intent of the development. During the testing
phase various tests are conducted to test for conformity, such tests are:

       Functional testing;
       Non-functional testing;
       Unit testing;
       Integration testing;
       System testing; and,
       Acceptance testing.

2.1 The Software Testing Life Cycle
Software developers follow the Software Development Life Cycle (SDLC), while software testers adopt
the Software Testing Life Cycle (STLC). The STLC refers to a testing process with distinct steps to be
executed in sequence to ensure that the quality goals of the software have been achieved. Each
phase of the STLC has different goals and deliverables, which provides a quality management process
for testing software. The STLC is initiated from the testing phase of the SDLC (See Figure 1: SDLC and
STLC Relationship Diagram) after coding is completed and again after deployment, where the
software is operational and being maintained until it’s retired.

The NSTG follows the STLC multi-layer testing process defined by ISO/IEC/IEEE 29119 standard, which
includes the Organisational Test Process, Test Management Process and the Dynamic Test Process.

                                  ORGANISATION TEST PROCESS

                                  TEST MANAGEMENT PROCESS

                                    DYNAMIC TEST PROCESS

          Figure 2: Multi-layer Approach Software Testing Process

                                                                                         23 | P a g e
National Software Testing Guideline

2.2 Software Testing Principles
While conducting software testing, the expectation is to achieve 100% test coverage within the
threshold of the testing process. To achieve this, the following testing principles are common
practice, which should be maintained:

        1. Testing should be conducted through risk-based assessment and not on exhaustive
            testing methods;
        2. Testing should start early at the development phase of the SDLC, so that any defects in
            the requirements and design phase are detected early. The earlier a defect is detected,
            the cheaper it is to repair;
        3. External test teams should be used for software testing implementation different from
            those that developed the software;
        4. All software testing processes must be comprehensive to test for invalid and unforeseen,
            as well as valid and anticipated input conditions;
        5. Any software testing specification must be detailed, complete and clear;
        6. Test cases need to be regularly reviewed and revised, with new and different test cases to
            find more defects;
        7. If the software product does not meet the user’s requirements, there is no point in
            testing to find defects and repairing it; and,
        8. Do not assume in your test planning that no errors would be found in the system.

2.3 Project Organisation Structure
In IT projects, it is important to create a project organisation structure for the management of the
day-to-day activities of the project. In software testing projects, several organisational structures are
possible, each having its own inherent requirements depending on the use. For example, in safety
systems where strict adherence to safety is required, a Software Quality Assurance team would be
required in the project organisational structure as a subject specialist on quality of the system.
However, for this guideline, the Test Project Team is streamlined and composed of the Test Team
(Test Manager/Test Lead, Testers, Developer in Test and Test Administrator), Development Team
(Development Manager and Programmers) and Software Quality Assurance Members; and they all
report to the Project Manager.

The Project Manager in turn reports to the organisation’s management (Project Board). The test team
are external to the organisation and different from those who developed the software. This scenario
can also be applied to off-the-shelf purchased software and to external software development

                                                                                            24 | P a g e
National Software Testing Guideline

contractors. Depending on the nature of the software and the size of the organisation, the
organisation structure below can be adapted to fit any testing scenario as required.

                                              PROJECT MANAGER

             DEVELOPMENT MANAGER                    SQA                     TEST MANAGER

    PROGRAMMERS                                           TESTERS        DEVELOPER IN TESTING   TEST ADMINISTRATOR

Figure 3: Software Testing Project Organisation

        Position                                                    Responsibilities
 Project Manager (PM)              Represents the client and manages the day to day activities on the
                                   project level.
  Test Manager (TM)                TM manages the whole testing project processes:
                                           Test Planning Process;
                                           Test Management Process;
                                           Test Execution Process; and,
                                           defines the test project direction.
         Tester                           Tester develop the Test Cases;
                                          Generate the Test Procedures;
                                          Executes the test;
                                          Logs the results of the tests; and
                                          Report the defect.
   Developer in Test                      Creates the program to test the codes created by the developers
                                           (unit and integration testing); and,
                                          Creates the test automation scripts.
   Test Administrator                     Builds the Test Environment;

                                                                                                     25 | P a g e
National Software Testing Guideline

                                         Manages and maintains the test environment assets;
                                         Supports the test team to use the test environment for test
                                          execution; and,
                                         Decommissions and archives the test environment when testing
                                          is completed.
Software Quality Assurance               Provides quality assurance support on the test project.

 2.4 Minimum Requirements to become a Software Test Professional
 It is expected that a Software tester shall possess the following:
     a. A bachelor’s degree in Computer Science, Information Systems, Software Engineering or a
           related technical field;
     b. Should be able to create and document automated and manual test plans and procedures,
           execute tests, analyse results and report on test problems and anomalies (document bugs);
     c. Perform software testing in all level of the design-develop-test-release-maintain software life
           cycle;
     d. Understand various development methodologies;
     e. Possess thorough knowledge of several testing tools;
     f.    Be fluent in UNIX, Linux and/or Windows, as well as scripting and command-line tools;
     g. Be an excellent communicator (written and verbal) with development, operations, product
           management and customers;
     h. Have working knowledge of various programming languages, such as Java, JavaScript, C# or
           C++, SQL, Python, PHP and Ruby on Rails;
     i.    Have Professional Certification such as:
              I.    ISTQB Certified Tester: The American Software Testing Qualifications Board (ASTQB);
             II.    Certified Software Tester (CSTE): The International Software Certification Board
                    (ISCB);
     j.    3 to 6 years of cognate experience depending on the certification level attained.
 See the full list of Software testing certifications in Appendix A.

 3        SOFTWARE TESTING GUIDELINE APPROACH
 The NSTG adopts the multi-layer approach to software testing process (See Figure 4: Multi-Layer
 Testing Process of ISO/IEC/IEEE 29119). This approach combines various standards that have been
 sequentially integrated into a single structured standard for software testing. Noting that

                                                                                               26 | P a g e
National Software Testing Guideline

conformance to the standard is flexible and simple to adopt in any software testing domain. The
ISO/IEC/IEEE 29119 structure consist of the following standards:

            BS 7925-1
            ISO/IEC 33063
            IEEE 829
            ISO/IEC 20246
            BS 7925-2

         Organsational                      •Creation and Maintenance of Organisational Test
                                             Policy, Strategy, Processes, Procedures and Other
          Test Process                       Assets.

    Test Management                         •Test Planning Process;
                                            •Test Monitoring and Control Process; and,
        Processes                           •Test Completion Process.

                                            •Test Design and Implementation Process;
         Dynamic Test                       •Test Environment Set-up and Maintenance Process;
          Processes                         •Test Execution Process; and,
                                            •Test Incident Reporting Process.

        Figure 4: Multi-Layer Testing Process

3.1 Organisational Test Process
This process is adopted for the development and management of the Organisational Test
Specifications (OTS). The Organisation Test Process is instantiated twice to develop the Organisational
Test Policy and Organisational Test Strategy. The OTS generally applies to testing throughout the
organisation.

3.1.1    Organisational Test Specifications
The following documents are components of the Organisational Test Specifications:

         a. Organisational Test Policy: Executive level document that lays out the purpose, goals and
             scope of testing in the organisation. It also provides the mechanism for establishment,
             review and updates of the Organisation’s Test Policy, Test Strategy and approach to
             project test management (See Test Management Process in section 3.3).

                                                                                           27 | P a g e
National Software Testing Guideline

          b. Organisational Test Strategy: A detailed technical document, defining how testing is
              carried out in the organisation.     It’s a generic document (programme level), which
              provides guidelines for software testing projects in the organisation.

  NOTES:

  The OTS is generic (programme oriented) and not specific to a test project. E.g. testing for
  different test levels and test types may require different approaches which may not be defined in
  the Organisational Test Strategy but defined in the project Test Strategy during test planning by
  the Test Manager.

      ORGANISATION TEST                                       ORGANISATION TEST PROCESS
          PROCESS                                                   (TEST POLICY)

                                                              ORGANISATION TEST PROCESS
                                                                   (TEST STRATEGY)
  Organisation Test Process initiated twice, once
  each to create and maintain the Organisational
   Test Policy and Organisational Test Strategy

          Figure 5: Organisation Test Specification Process

3.1.1.1    Purpose
The purpose of the Organisation Test Process is to develop, monitor conformance and maintain
Organisational Test Specifications.

3.1.1.2    Outcomes
           a. Requirements for the Organisational Test Specifications are identified;
           b. Organisation Test Specifications document is developed;

                                                                                           28 | P a g e
National Software Testing Guideline

           c. Stakeholder buy-in is solicited for the Organisational Test Specifications;
           d. The Organisational Test Specifications is made accessible across the organisation;
           e. Conformance to the OTS is monitored;
           f.    Stakeholders agree to updates to the OTS; and,
           g. Updates are affected to the OTS document.

3.1.1.3    Activities and tasks
The following activities and tasks are to be implemented in line with the applicable organisational
policies and procedures by the department (e.g. ICT department) or person (Director/Head of ICT)
responsible for the Organisational Test Specifications:

          a. Develop the Organisational Test Specifications through gathering requirements from
                existing; testing practices, stakeholders, documents, workshops, interviews and other
                relevant resources;
          b. The developed requirements are used to create the Organisational Test Specifications
                document;
          c. Stakeholders approval is required on the content of the Organisational Test
                Specifications; and,
          d. The Organisational Test Specifications availability is communicated to relevant
                stakeholders.

3.1.2     Practical Guideline
The following steps should be followed while maintaining the OTS.

3.1.2.1    Monitor and Control Use of Organisational Test Specifications
           a. Monitor usage of OTS to determine if conformance is maintained; and,
           b. Keep stakeholders aligned with the Organisational Test Specifications document
                 updates.

3.1.2.2    Update Organisational Test Specifications
           a. Review all practice feedback of the Organisational Test Specifications;
           b. Effectiveness of use and management of the Organisational Test Specifications should
                 be considered and feedback on changes to improve its effectiveness should be
                 determined and approved by stakeholders;
           c. Implement all improvement changes that have been approved; and,
           d. All changes to the document should be communicated to all stakeholders.

                                                                                            29 | P a g e
National Software Testing Guideline

  NOTES:

  Stakeholders in this process refers to the organisation’s management (Directors, CEO, CTO,
  Heads of ICT, etc) and internal Project Managers’ involved in testing projects.

  Find Organisational Test Policy and Organisational Test Strategy document template in Test
  Documentation section.

3.2 Test Management Processes
This process covers the management of testing for a test project (project test management), and test
management for test levels (unit test, integration test, system test, acceptance test) and test types
(usability, accessibility, performance, configuration, security, availability tests etc) within a test
project.

                                                         Test Management Process initiated
                                                              once at the project level
                                                               (i.e. small test projects)

     TEST MANAGEMENT                                TEST MANAGEMENT PROCESS
          PROCESS                                         (PROJECT LEVEL)

                                                    TEST MANAGEMENT PROCESS
                                                         TEST MANAGEMENT PROCESS
                                                              TEST MANAGEMENT PROCESS
                                                               (TEST LEVELS & TEST TYPES )

                                                      Test Management Process several instances
                                                       initiated at the test levels & for test types
                                                                 (i.e. large test projects)

           Figure 6: Test Management Process
The Test Management Process includes:

           a. Test Planning;
           b. Test Monitoring and control; and,
                                                                                          30 | P a g e
You can also read
NEXT SLIDES ... Cancel