Test Strategy Approaches

Testing is an integral part of a project regardless of the type of development model which is being applied.   Whether the development model is:

•       Sequential such as waterfall, v-model & W-model; 
•        Iterative or incremental models, such as rapid application development, and  Rational Unified Process;
•        Agile models including SCRUM and extreme programming; or
•        DevOps involving practices such as continuous build, integration, test, and deployment processes, creating environments on demand.

The test manager and test engineers must be properly aligned with the development model being used for the project.  Irrespective of the project’s scale, complexity, risk and development model, managing the testing process for any development project comprises of developing test artifacts, processes and systems to maximize the probability that the solution is fit for purpose within the project context.  There is not a one size fits all approach to a test strategy and management.

Quality Management

Over the past several decades, a well-defined set of methods were used during the design, production and delivery of products, which collectively became known as quality engineering.  A subsequent body of knowledge evolved which was described by various terms such as quality control or quality assurance.  Out of this quality movement a new management philosophy to encapsulate the quality system became known as the body of knowledge called “total quality management.”

According the ISO 9000-2005 standards, quality was defined as the “degree to which a set of inherent characteristics fulfills requirements.”  A well-known definition of quality by Gavin (1984), which included numerous dimensions was:

1. Performance- the product’s ability to do the work it is supposed to do.
2. Features- things that add to the convenience and comfort.
3. Reliability- the ability to perform without failure over time.
4. Conformance- the degree to which the product meets the codes of a state or community. 
5. Durability- the length of time the product will last until it is discarded.
6. Serviceability- the ability for making repairs easily, quickly, and at a reasonable cost.
7. Aesthetics- sense of appeal, such as color, sound, feel, and comfort
8. Perceived quality- the impression the product creates in the customer’s mind.  

This definition is characterized by many dimensions, and would certainly have different meanings for different types of products. 

Quality management in manufacturing and other industrial settings set a foundation of knowledge that became part of the lexicon for quality management in software engineering. 

The ISO 9000 Standard 

A quality management system comprises of the process guidelines, rules and accountability that a company works towards, including the documentation of these process descriptions, procedures, work instructions, and best practices and a mechanism to monitor and continually improve the quality management system.   The QA system documentation would usually include outlining all the business processes which are necessary to meet the company’s quality objectives. This would include development/engineering, procurement/purchasing, production and maintenance/service, and regulating all the QA protocols within each of processes within each of these areas of the business.  Most large firms seek compliance with the quality management guidelines outlined in the ISO 9000 family of standards.  

The ISO 9001: 2000 version was a more process-oriented version of the standard and made a distinction between management processes (legal roles & decision-making) and value creation (development, production, services and maintenance) and internal processes such as supply chain management, accounting, procurement/purchasing. 

The test manager should be aware that software testing forms part of the quality management set of activities necessary to produce and deliver software products which are fit for use by customers and users and comprising of the desired features. Quality management may extend to the various deliverables including requirements documents, development deliverables such as unit test results, code review information, and test deliverables.  The study and practice of the broad set of quality management activities is sometimes called software quality assurance.  Comprehensive quality management stipulates that software quality assurance occur throughout the entire software lifecycle.  This could include key competencies such as reviews, verification and validation, process standardization and software process maturity.  

Quality Management in Agile

Since the agile movement came on the scene and has become prevalent in many organisations, quality management has needed to adapt and be incorporated into the agile project methodology.

“Conventional quality management is a top-down process. A central QM team is responsible for documenting and, where it makes sense, standardising the processes involved. The team draft the process manuals and rolls out standardised processes in all relevant departments. In contrast, agile methodology is a bottom up process. Agile teams devise their own working methods, so QM staff have to learn to adapt.” (p. 148, Linz, Testing in Scrum) 

Retrospectives and Process Improvement

One of the central tenets of the ISO 9000 standard is the requirement for continual improvement, and the Sprint Retrospective that forms part of the Scrum methodology is one method of achieving this.

 Test Strategy Overview                                                          

The test strategy should describe the test approach and methodology for a project or an organisation’s general test approach and methodology.  The test strategy should be informed by the project methodology, system architecture and development lifecycle. Also the test strategy should not only be informed, whether by an organisational policy on quality assurance, but the overall project management framework (such as agile or Prince2). Importantly it should take into consideration the situational context of the project environment- the technical context, the business context and the social context.  Situational awareness means having awareness of the classic project management trade-offs in time, quality and resources for a project or programme of work.  The test strategy and the processes described within it, should be consistent with these considerations,  the business goals of the organisation and the project management framework. 

The test strategy should include the levels of testing. Depending upon the scope, technology, complexity and risk of the project these levels will vary from project to project. Typically test levels would begin with component testing, then integration and system testing and usually user and operational acceptance testing. In highly agile projects these levels are executed in small increments in short timeframes.

 The following are some example of various test strategies:

1. Methodical Strategy which could be based upon software quality characteristics, involving predetermined test conditions, based upon international quality standards such as ISO25000 (which replaced ISO9126).  This could include a collection of logical test conditions which are associated with a particular application, functional and/or non-functional areas. 
2. Process or standard-compliant strategies, such as biomedical devices subject to compliance standards (for example the FDA), requires the test team to follow a set of processes defined by the standards committee or other panel of experts.  This would necessitate developing procedures that address compliance specifications and the relevant standards documentation.
3. Consultative strategies may be appropriate where the test team relies on the knowledge and skill of one or more key stakeholders such as architects, developers, business analysts and product specialists.  
4. Model-based strategies and model-based testing, involves the generation of test input data, based upon a model which is the information about the domains of the input values and the test generation involves clever selection and combination of a subset of those values to produce test input data.  For example, in model-based performance testing of mobile device application, models could be developed of incoming and outgoing network traffic, active and inactive users, and resulting processing load, based upon current usage and trends over time.  Also models might be developed to considering the current production environment’s  hardware, software, data capacity, network and infrastructure. Models may be developed for expected throughput rates, response times, and resource allocation. 
5. Analytical strategies, including risk-based testing, where the test team analyses the test conditions to cover and conducts a risk assessment which informs the level of test coverage. 
6. Agile strategies – are used when the development project(s) adopt agile development practices.  An agile testing strategy is based upon a whole team approach .  This could include agile programmers using test-driven development (TDD).  The following diagram illustrates a well-recognized framework for an agile testing strategy in the book, Agile Testing: A practical guide for testers and agile teams:  

Eclectic Test Strategy

Depending on the context, in order to address various technical and business drivers, such as security compliance, agile methodology and the level of design documentation or its absence, the test strategy may need to be eclectic in its approach. This means that rather than a monolithic approach which narrowly focuses on a particular method, such as a methodical strategy, an eclectic strategy encompasses various approaches. In my own experience, it is often valuable to combine a methodical approach which is complimented with a risk-based assessment and prioritisation of functions and with non-planned exploratory testing techniques. Exploratory testing techniques relies upon the level of skill and experience of the tester(s) involved in test execution. In summary, an eclectic test strategy may well yield greater breadth of testing than a single focused testing framework.

 Trade-off Decisions  

Most projects require a trade-off between quality, schedule, budget and features. These trade-offs may be known at the beginning of the project or may become apparent during later phases of the project. The test manager should be aware of the inter-dependencies between these project components (time, resources and quality) and be prepared to participate in trade-off decisions. While the test manager should always defend quality, there are times when quality is not the most important aspect of the project. The test manager must be able to objectively analyse what changes in the schedule, budget or feature set may affect the overall quality of the project, for better or worse. For example adding unplanned features into a project may reduce the overall quality as the testing resources are spread more thinly but the new feature may be critical to the acceptability of the project.

 While the test manager may not agree with a project trade-offs particularly those that may lower the overall quality of the delivered product, the manager must be able to provide the information to the stakeholders so that everyone can clearly understand the anticipated impact of the change. For example, if the net result of a proposed changes to reduce the testing time for all components by one third, what does this mean for the overall quality and in terms of the solution being fit for use by the customer? The test manager should be able to explain the impact of the reduction in testing in terms of the increased risk to the product quality and should clearly explain areas that will receive less or no testing a result of the added features.

 Change Management

Changes to project scope can influence the overall schedule budget features and quality goals of the project. Changes must also be managed in order to be tracked, scheduled, and assist regarding impact. A good change management system is a requirement for a well-managed project. The test manager, in particular, needs to track changes that affect the original estimates and schedules. These changes are not necessarily limited to additions or deletions of features. Any change that affects the testing aspects of the project must be taken into consideration. For example, if the development team does not receive a server that is required for compatibility testing, the testing schedule will be affected.