Test Strategy Approaches

Testing is an integral part of a project regardless of the type of development model which is being applied.   Whether the development model is:

•       Sequential such as waterfall, v-model & W-model; 
•        Iterative or incremental models, such as rapid application development, and  Rational Unified Process;
•        Agile models including SCRUM and extreme programming; or
•        DevOps involving practices such as continuous build, integration, test, and deployment processes, creating environments on demand.

The test manager and test engineers must be properly aligned with the development model being used for the project.  Irrespective of the project’s scale, complexity, risk and development model, managing the testing process for any development project comprises of developing test artifacts, processes and systems to maximize the probability that the solution is fit for purpose within the project context.  There is not a one size fits all approach to a test strategy and management.

Quality Management

Over the past several decades, a well-defined set of methods were used during the design, production and delivery of products, which collectively became known as quality engineering.  A subsequent body of knowledge evolved which was described by various terms such as quality control or quality assurance.  Out of this quality movement a new management philosophy to encapsulate the quality system became known as the body of knowledge called “total quality management.”

According the ISO 9000-2005 standards, quality was defined as the “degree to which a set of inherent characteristics fulfills requirements.”  A well-known definition of quality by Gavin (1984), which included numerous dimensions was:

1. Performance- the product’s ability to do the work it is supposed to do.
2. Features- things that add to the convenience and comfort.
3. Reliability- the ability to perform without failure over time.
4. Conformance- the degree to which the product meets the codes of a state or community. 
5. Durability- the length of time the product will last until it is discarded.
6. Serviceability- the ability for making repairs easily, quickly, and at a reasonable cost.
7. Aesthetics- sense of appeal, such as color, sound, feel, and comfort
8. Perceived quality- the impression the product creates in the customer’s mind.  

This definition is characterized by many dimensions, and would certainly have different meanings for different types of products. 

Quality management in manufacturing and other industrial settings set a foundation of knowledge that became part of the lexicon for quality management in software engineering. 

The ISO 9000 Standard 

A quality management system comprises of the process guidelines, rules and accountability that a company works towards, including the documentation of these process descriptions, procedures, work instructions, and best practices and a mechanism to monitor and continually improve the quality management system.   The QA system documentation would usually include outlining all the business processes which are necessary to meet the company’s quality objectives. This would include development/engineering, procurement/purchasing, production and maintenance/service, and regulating all the QA protocols within each of processes within each of these areas of the business.  Most large firms seek compliance with the quality management guidelines outlined in the ISO 9000 family of standards.  

The ISO 9001: 2000 version was a more process-oriented version of the standard and made a distinction between management processes (legal roles & decision-making) and value creation (development, production, services and maintenance) and internal processes such as supply chain management, accounting, procurement/purchasing. 

The test manager should be aware that software testing forms part of the quality management set of activities necessary to produce and deliver software products which are fit for use by customers and users and comprising of the desired features. Quality management may extend to the various deliverables including requirements documents, development deliverables such as unit test results, code review information, and test deliverables.  The study and practice of the broad set of quality management activities is sometimes called software quality assurance.  Comprehensive quality management stipulates that software quality assurance occur throughout the entire software lifecycle.  This could include key competencies such as reviews, verification and validation, process standardization and software process maturity.  

Quality Management in Agile

Since the agile movement came on the scene and has become prevalent in many organisations, quality management has needed to adapt and be incorporated into the agile project methodology.

“Conventional quality management is a top-down process. A central QM team is responsible for documenting and, where it makes sense, standardising the processes involved. The team draft the process manuals and rolls out standardised processes in all relevant departments. In contrast, agile methodology is a bottom up process. Agile teams devise their own working methods, so QM staff have to learn to adapt.” (p. 148, Linz, Testing in Scrum) 

Retrospectives and Process Improvement

One of the central tenets of the ISO 9000 standard is the requirement for continual improvement, and the Sprint Retrospective that forms part of the Scrum methodology is one method of achieving this.

 Test Strategy Overview                                                          

The test strategy should describe the test approach and methodology for a project or an organisation’s general test approach and methodology.  The test strategy should be informed by the project methodology, system architecture and development lifecycle. Also the test strategy should not only be informed, whether by an organisational policy on quality assurance, but the overall project management framework (such as agile or Prince2). Importantly it should take into consideration the situational context of the project environment- the technical context, the business context and the social context.  Situational awareness means having awareness of the classic project management trade-offs in time, quality and resources for a project or programme of work.  The test strategy and the processes described within it, should be consistent with these considerations,  the business goals of the organisation and the project management framework. 

The test strategy should include the levels of testing. Depending upon the scope, technology, complexity and risk of the project these levels will vary from project to project. Typically test levels would begin with component testing, then integration and system testing and usually user and operational acceptance testing. In highly agile projects these levels are executed in small increments in short timeframes.

 The following are some example of various test strategies:

1. Methodical Strategy which could be based upon software quality characteristics, involving predetermined test conditions, based upon international quality standards such as ISO25000 (which replaced ISO9126).  This could include a collection of logical test conditions which are associated with a particular application, functional and/or non-functional areas. 
2. Process or standard-compliant strategies, such as biomedical devices subject to compliance standards (for example the FDA), requires the test team to follow a set of processes defined by the standards committee or other panel of experts.  This would necessitate developing procedures that address compliance specifications and the relevant standards documentation.
3. Consultative strategies may be appropriate where the test team relies on the knowledge and skill of one or more key stakeholders such as architects, developers, business analysts and product specialists.  
4. Model-based strategies and model-based testing, involves the generation of test input data, based upon a model which is the information about the domains of the input values and the test generation involves clever selection and combination of a subset of those values to produce test input data.  For example, in model-based performance testing of mobile device application, models could be developed of incoming and outgoing network traffic, active and inactive users, and resulting processing load, based upon current usage and trends over time.  Also models might be developed to considering the current production environment’s  hardware, software, data capacity, network and infrastructure. Models may be developed for expected throughput rates, response times, and resource allocation. 
5. Analytical strategies, including risk-based testing, where the test team analyses the test conditions to cover and conducts a risk assessment which informs the level of test coverage. 
6. Agile strategies – are used when the development project(s) adopt agile development practices.  An agile testing strategy is based upon a whole team approach .  This could include agile programmers using test-driven development (TDD).  The following diagram illustrates a well-recognized framework for an agile testing strategy in the book, Agile Testing: A practical guide for testers and agile teams:  

Eclectic Test Strategy

Depending on the context, in order to address various technical and business drivers, such as security compliance, agile methodology and the level of design documentation or its absence, the test strategy may need to be eclectic in its approach. This means that rather than a monolithic approach which narrowly focuses on a particular method, such as a methodical strategy, an eclectic strategy encompasses various approaches. In my own experience, it is often valuable to combine a methodical approach which is complimented with a risk-based assessment and prioritisation of functions and with non-planned exploratory testing techniques. Exploratory testing techniques relies upon the level of skill and experience of the tester(s) involved in test execution. In summary, an eclectic test strategy may well yield greater breadth of testing than a single focused testing framework.

 Trade-off Decisions  

Most projects require a trade-off between quality, schedule, budget and features. These trade-offs may be known at the beginning of the project or may become apparent during later phases of the project. The test manager should be aware of the inter-dependencies between these project components (time, resources and quality) and be prepared to participate in trade-off decisions. While the test manager should always defend quality, there are times when quality is not the most important aspect of the project. The test manager must be able to objectively analyse what changes in the schedule, budget or feature set may affect the overall quality of the project, for better or worse. For example adding unplanned features into a project may reduce the overall quality as the testing resources are spread more thinly but the new feature may be critical to the acceptability of the project.

 While the test manager may not agree with a project trade-offs particularly those that may lower the overall quality of the delivered product, the manager must be able to provide the information to the stakeholders so that everyone can clearly understand the anticipated impact of the change. For example, if the net result of a proposed changes to reduce the testing time for all components by one third, what does this mean for the overall quality and in terms of the solution being fit for use by the customer? The test manager should be able to explain the impact of the reduction in testing in terms of the increased risk to the product quality and should clearly explain areas that will receive less or no testing a result of the added features.

 Change Management

Changes to project scope can influence the overall schedule budget features and quality goals of the project. Changes must also be managed in order to be tracked, scheduled, and assist regarding impact. A good change management system is a requirement for a well-managed project. The test manager, in particular, needs to track changes that affect the original estimates and schedules. These changes are not necessarily limited to additions or deletions of features. Any change that affects the testing aspects of the project must be taken into consideration. For example, if the development team does not receive a server that is required for compatibility testing, the testing schedule will be affected.   

Can software engineering contribute to solving the climate crisis?

Who feels a sense of urgency to be a part of the solution to the climate crisis? How can software engineers be involved in projects that contribute, even in a small way, to preventing global warming to less than 1.5 Celsius by the end of the century?  The short answer is yes there are numerous use cases in which software engineering can positively impact the planet and climate change.   

Glasgow Climate Summit

Views of those involved vary widely in relation to the degree of success of the recent Glasgow Climate Summit. British host of the event, Boris Johnson, told leaders that it was ‘one minute to midnight’ on the doomsday clock.  But at the 11th hour of the Summit during which India and China, due to their reliance on coal, changed the wording of the agreement to ‘phase down’ rather than ‘phase out’ coal, Johnson observed the result was ‘tinged with disappointment’ and giving the result just over ‘6 out of 10’. 

The climate summit at the least appeared to facilitate a sense of urgency among nations of moving more quickly towards the goal of mitigating the impact of climate change.

Through the COP26 negotiations in Glasgow, 140 countries strengthened their 2030 emissions targets, which was an improvement on the previous Paris Agreement.

“Glasgow has not closed the emissions gap in 2030, which is vital to keeping 1.5C alive, but it has set up an urgent process to address this gap, inviting countries to submit more ambitious national climate targets by the end of 2022,” Bill Hare, senior scientist and chief executive at Climate Analytics, said.

The real test of COP26 was a success is whether the promises made are turned into reality.

Each leader involved in policy decisions might consider the diverse range of solutions which are systematically described and ranked in Paul Hawken’s book, Drawdown: The Most Comprehensive Plan Ever Proposed to Reverse Global Warming.  Hawken ranked the solutions based upon the amount of greenhouse gases they can potentially avoid or remove from the atmosphere. Energy efficiencies in refrigeration is at the top of the list.  

Software engineering and climate solutions

How can software engineers get involved and contribute to the climate emergency?

There are numerous technologies and use cases which contribute to more sustainable climate change solutions at different scales.  For example, systems dynamics modelling and predictive analytics, energy markets and carbon pricing mechanisms systems, energy optimisation and smart metering, internet of things, smart grids to name a few.  

Easterbrook’s paper Climate Change: A Grand Software Challenge, summarizes the types of projects that software engineers could use their skills to contribute to climate change solutions.  Easterbrook reminds us that software engineering is part of the problem as well as the solution:

“Software plays a major role, both as part of the problem and as part of the solution. A large part of the massive growth of energy consumption in the past few decades is due to the manufacture and use of computing and communication technologies, and the technological advances they make possible.”

Easterbrook points out the benefits to climate science:  

“… software also provides the critical infrastructure that supports the scientific study of climate change, and the use of that science by society. Software allows us to process vast amounts of geoscientific data, to simulate earth system processes, to assess the implications, and to explore possible policy responses. Software models allow scientists, activists and policymakers to share data, explore scenarios, and validate assumptions.”  

Easterbrook categorizes the ways in which software engineering can contribute to solutions associated with mitigating climate change as follows:

• Computer-supported science;
• Software for decision-making;
• Green IT.

Computers-supported science 

This category includes software development and optimisation of systems dynamic modelling for the earth’s systems, data-intensive science, and electronic notebook and workflow tools.

Software for decision-making

Easterbrook details various applications for software that is used for information sharing to improve public understanding of science and to improve decision-making at various levels.

Green IT

Green IT is concerned with improving power consumption of software and its infrastructure.

On-Premises versus Cloud Infrastructure

One of the significant determinants of energy efficiency in organisations is whether an organisation’s datacentre is on premises or outsourced to a cloud provider.  The Natural Resources Defense Council conducting an independent analysis detailing some of the factors to consider which are attributable to how effective the cloud solution is in mitigating climate change including:  server utilisation, the carbon footprint of the electricity used to power data centers, power usage effectiveness, and hardware efficiency.   

Source: Natural Resources Defense Council (Oct, 2012) Is Cloud Computing Always Greener?

Small, medium and large businesses and NGOs could consider the relative carbon footprint of each option when it comes to deciding where their datacentre will reside and when considering alternative cloud service providers, one criteria could be which provider is the most environmentally friendly in the long term.

Software engineers can get involved in projects concerning climate science, decision-making, systems dynamic modelling and predictive analytics, education, and green IT projects.  Parts of the climate solution include the development of smart cities, electric transportation, smart grids, energy efficiency at all scales and carbon pricing mechanisms.  Software engineers can play a part in one of the defining challenges of the 21^st century.

References

Easterbrook, Steve  Climate Change: A Grand Software Challenge

Hawken, Paul (2018). Drawdown: The Most Comprehensive Plan Ever Proposed to Reverse Global Warming

Mims, Christopher, (Aug, 2010) How coders can help fight climate change,    https://www.technologyreview.com/2010/08/31/200618/how-coders-can-help-fight-climate-change/

Natural Resources Defense Council (Oct, 2012) Is Cloud Computing Always Greener?

Slezak, Michael (14 Nov 2021) COP26 has wrapped up in Glasgow. So was the climate summit a success?  https://www.abc.net.au/news/2021-11-14/was-cop26-a-success-glasgow-climate-pact/100619070

Failure is an option… as an ongoing process in design and learning

Testing software can be thought of as a highly organized experimental method.  Those of us who work in software development, are generally interested in maximizing the possibility that a software product or service is fit for purpose and meets the expectations of the customer.

We use a range of testing techniques that are appropriate to the technology and business context: from risk assessment to formal reviews, static and dynamic testing, manual and automated, progression and regression, functional and non-functional, requirements-based as well as exploratory testing techniques.  Testing should provide valuable feedback about whether the design after a particular iteration meets the various stakeholders’ expectations.

But what about the psychology of managing teams when testing reveals a quantity of problems and incidents that were not anticipated? How can you cultivate an atmosphere, which encourages looking for errors, mistakes and defects as a means to learning and improving the whole development process?

The problem is that frequently people do not want to confront errors or feel threatened by them. They may feel affronted or offended.   However this is missing an understanding of the fundamental reason why errors and mistakes are necessary to learn from, not only  in software development, but indeed any process of design and innovation.

I like to get inspiration from a range of sources and different modes of design.  For example, one of the most famous architects in the last 50 years, Frank Gehry says this about mistakes in the design process of large commercial buildings:

“So as you are working in real life and in real time you are constantly having small victories and small mistakes….the important thing is to keep moving ahead and learning from the mistakes and building on it, building positive momentum from that. Because it is a very complex endeavor.”

Large-scale enterprise software project transformations can be of comparable in terms of scale and complexity of commercial building projects so I think there is a parallel.

In the book Fail Fast, Fail Often, Michael Bloomberg, the billionaire who developed the financial software tools including analytics and an equity trading platform, is quoted on the process of developing software for his financial software:

“We made mistakes, of course. Most of them were omissions we didn’t think of when we initially wrote the software. We fixed them by doing them over and over, again and again. We do the same today. While our competitors are still sucking their thumbs trying to make the design perfect, we’re already on prototype version #5….It gets back to planning versus acting: We act from day one; others plan how to plan – for months.”

Bias Towards Action And Agile Iterations

Bloomberg’s quote reveals a keen insight about the bias towards action versus being paralyzed by excessive planning. The agile movement is a response to limitations of traditional project management in part where according to Mike Cohn in his book Agile Estimating and Planning:

“At the start of each new iteration, an agile team incorporates all new knowledge gained in the preceding iteration and adapts accordingly.” (Cohn, 2006, 26)

The process of testing software is one of the main methods in which new knowledge is acquired and unexpected behavior of the design is observed during each iteration. Therefore finding errors is a necessary way to obtain information which assists the development team to adapt to this new knowledge.   As Frank Gehry suggested “the important thing is to keep moving ahead and learning from the mistakes and building on it …”

So to inspire and uplift a team from uncomfortable feelings of perfectionism or threatened by mistakes, errors or omissions or even misjudgments in the software design and development process, a manager can do a number of things to cultivate a positive environment for creativity and productivity:

1. Build an atmosphere of safety, where it is OK to make mistakes, ask crazy questions, find errors and this is, in fact, a necessary part of the evolutionary nature of the development process;
2. Facilitate discussions around shared risks which promote trusting cooperation;
3. Tell a narrative which creates a sense of shared goals, values and purpose and in doing so builds a coherent working environment.

Sprints as a structure for prototyping to aid design and product development

“If you already know what you are doing, you wont do it.” Frank Gehry

Part of the benefit of establishing a short timeframe for a sprint in software development, is that the time sets in motion a constraint for the team to create a boundary for focused effort. This creates a space for prioritizing development activity in a very focused way. It is well-known that sprints also assists in the relative measurement of velocity of the team between sprints.

Regular Feedback and Double Loop Learning

But there are other benefits. Short duration sprints ensure that there is regular feedback. As one of the original creators of Scrum, Jeff Sutherland describes, in his book of the same name, a more evolutionary, adaptive and self-correcting system of software development than its predecessor, waterfall was needed. He writes:

“And so my team embarked on what we called “sprints”. We called them that because the name evoked a quality of intensity. We were going to work all out for a short period of time and then stop to see where we were.” (Sutherland, 2014)

Teams can become more efficient by examining how they can improve their performance during retrospectives. Retrospectives mean double-loop learning, or generative learning, which may involve updating mental models, modification of goals or decision-making in the light of experience. This is particularly relevant when there is a high level of requirements uncertainty or technological complexity.

Encourage Problem Solving Through Prototyping

But depending on the level of requirements uncertainty and innovation, short iterations or sprints can provide a disciplined structure to encourage prototyping and problem solving. The advantage of a multi-disciplinary team providing regular feedback on a rough prototype of software or product cannot be overstated. The sort of inquiry that should result include such questions as:

1. Are we building the right product?
2. Is this what the client wants? Is it suitable?
3. Is it usable?
4. Is it functional?
5. What about its non-functional quality characteristics such as performance?

Prototyping- Build In Order To Think

But sprints, or short iterations leading to early prototyping in product development (not just in software) can also provide a framework for a problem-solving method. Prototyping is an approach to doing in order to think rather than thinking in order to do. This can make visible thought processes and initial intuitions through building multiple concrete prototypes or models of the final product.

Famous Architect Frank Gehry’s Working Method

Frank Gehry is perhaps best known for his iconic architecture including his idiosyncratic design of the Guggenheim Museum in Bilbao, Spain and the Walt Disney Concert Hall in Los Angeles. Building an extensive number prototypes was his working method which formed the backbone of his design process.

“Disney Hall did not appear to Gehry as a fully developed image, or a single idea. It emerged through a process of little bets through which Gehry and his team worked within constraints to frame and identify thousands of problems….Gehry and his team would, in fact create eighty-two prototypes models, working closely with the planning committee, until they arrived at the final form of the hall.” (Sims, P, 2011)

Disney Concert Hall

 

This philosophy reflects the Design Program at Stanford where the maxim, ‘building is thinking’ is an organizing principle of learning about the design process. Architecture, design and software development are activities that thrive on rapid prototyping to facilitate thinking, feedback and problem solving.  Sprints are a way to structure time and create constraints to focus effort and provide a continuous feedback loop which creates an adaptive, self-correcting system.  Design prototypes in order to think and solve problems.

References

Sims, Peter, (2011) Little Bets: How breakthrough ideas emerge from small discoveries: 79

Sutherland, Dr. Jeff, (2014) Scrum: A revolutionary approach to building teams, beating deadlines and boosting productivity: 73

Image By Carol M. Highsmith https://commons.wikimedia.org/w/index.php?curid=4544231https://en.wikipedia.org/wiki/Frank_Gehry#/media/File:Disney_Concert_Hall_by_Carol_Highsmith_edit2.jpg

Dealing with uncertainty in product and software development projects

The world is full of uncertainty, and when it comes to software engineering projects or new product development, uncertainty is a familiar conundrum to be negotiated in the process of development.  In our projects, how do we best deal with uncertainty?

The degree of uncertainty varies of course according to the level of complexity and unknowns in a product development or project.  On the one end of the spectrum are serious innovations which require the inventor to heroically dive into unchartered waters and move beyond conventional wisdom.  James Dyson explained that it took 5,127 prototypes over 15 years to develop the cyclone technology vacuum. Starting with cardboard and duct tape, then moving on to ABS polycarbonate, his phenomenal vision, and feat of blistering persistence lead to a blockbuster consumer product.

Dyson’s iconic cyclone technology vacuum.

“By the time I made my 15th prototype, my third child was born. By 2,627, my wife and I were really counting our pennies. By 3,727, my wife was giving art lessons for some extra cash. These were tough times, but each failure brought me closer to solving the problem. It wasn’t the final prototype that made the struggle worth it. The process bore the fruit. I just kept at it.” (James Dyson, 2011)

An exploratory method, trial and error are necessary for true innovation like the design process of the cyclone vacuum. It is interesting to note that Dyson, prolific inventor that he is, clearly a very wealthy man, attributes his persistence to excelling at long distance running as a young man.

There is an important lesson here: persistent systematic testing of a product and prototyping inherent in a design process encompasses learning and overcoming uncertainty. Dyson did not know upfront all the requirements and specifications of his final product.   However he understood that his inventing the best vacuum cleaner in the world involved learning through a design process and in the process overcoming uncertainty in relation to optimizing the invention under various constraints.

Uncertainty in Software Development Projects

Similarly in software development, there is a difference between plan-driven and agile projects in their approach to uncertainty.   A plan-driven project attempts to minimize uncertainty about the product by making all the decisions about the requirements upfront. However that is not realistic in many situations. By making decisions too far in advance on complex projects, there is the risk of making erroneous assumptions based on incomplete knowledge.

Requirements Uncertainty Principle

Someone said that the only certainty in life is death and taxes.   In software engineering, I would add another certainty is requirements uncertainty! It was best articulated by W. S. Humphrey, whose well-known Requirements Uncertainty Principle, which states that:

“For a new software system, the requirements will not be completely known until after the users have used it. The true role of design is thus to create a workable solution to an ill-defined problem. While there is no procedural way to address this task, it is important to establish a rigorous and explicit design process that can identify and help to resolve the requirements uncertainties as early in the developmental process as possible.”

Since the publication of Humphrey’s book: “A Discipline for Software Engineering”, the agile movement has emerged and created a different approach to managing uncertainty to traditional project management. Agile planning gradually reduces uncertainty as the project proceed. This is illustrated in the following diagram.

source: Charles Cobb,  2014-18

According to Kenneth Rubin in his popular text, Essential Scrum: A Practical guide to the Most Popular Agile Process:

“Plan-driven, sequential processes focus on using (or exploiting) what is currently known and predicting what isn’t known. Scrum favors a more adaptive, trial-and-error based upon the appropriate use of exploration.”

An agile project is built around recognizing, managing, and reducing uncertainty as the project progresses.

I have a strong appreciation that using testing and an experimental method deals directly with the problem of uncertainty by systematic exploration of functionality and risk under various conditions in moving towards a system to becoming fit for purpose.

Disciplined Learning

Alistair Cockburn, an early advocate of the agile methodology, refers to the systematic exploration as disciplined learning in very small doses (for illustration see Cockburn’s knowledge-acquisition curve diagram below), where each step provides information that can be used to adjust the four categories of learning:

1. What a project team should be building (rather than what they thought they should be building at the outset);
2. Whether they have the right people on the team;
3. Where the technical ideas are flawed;
4. How much it will cost to develop. (Cockburn, 2014, 15)

I have personally experienced, as Cockburn describes, the phenomenon that in many software development projects, not necessary just waterfall projects, the major integration occurs towards the end of development. This leads to much learning about the interoperability and related risks associated with the integration after most of the development costs have been accrued.   Whereas if uncertainty is mitigated earlier and learning takes place incrementally from the outset, this can lead to reduced risks in the final product and provide valuable feedback to the project sponsor to direct and redirect the project throughout the development effort towards a better result.

source: Alistair Cockburn, 2014

Risk and uncertainty is particularly high for new products, products that are new to the market, and applications that haven’t been implemented before. And new technologies are forming the foundation of many new applications that have never been developed before. That is where the interaction of technology uncertainty and requirements uncertainty is at its highest level.

How to think about uncertainty?  One way in which a project team can think about uncertainty is in relation to the perceived level of complexity of a technology undertaking.

The Stacey complexity model can be used to illustrate the level of complexity in relation to two dimensions:

• REQUIREMENTS: How well-understood and well-defined are the project’s goals and requirements?
• TECHNOLOGY: How well-understood is the technology and solution for solving the problem?

Source: Charles Cobb,  2014-18

Agile projects may help address what Daniel Kahneman calls the planning fallacy. People and groups, susceptible to the planning fallacy, overestimate the benefits and underestimate risks and costs of delivering the product. This can be seen as a form of wishful thinking. People want to be successful. Project leaders can be prone to optimism bias. Teams may also fail to take into consideration outside influences and the level of complexity of all the system of systems within an organisation.

The 12^th annual state of agile survey, conducted in the second half of 2017, provided a snapshot on the state of play of agile practices in software organisations around the world. It concluded that while agile adoption is growing within organisations, only 12% of respondents said that their organisations have a high level of competency in agile practices.

An agile methodology, such as scrum, involve a more adaptive approachwith an appropriate use of exploration.  An agile project can be organised to reduce uncertainty incrementally as the project progresses. However it is critical to have very skilled engineers and team members who understand when it is better to use a traditional, tradition-agile hybrid or agile model. A scrum project still will produce requirements and plans, however with the assumption that details of those plans and requirements will be articulated as the project teams and product owner learns more about the design of the product during incremental process of building in structured sprints and learning periods (retrospectives). Plan-driven approaches to software projects use methods to handle large products and teams, and highly critical products. These type of projects create highly detailed plans upfront that are suitable for highly stable environments. The approaches can be combined which could be the topic of another discussion. Agile projects have an incremental approach to mitigating uncertainty as the project progresses and facilitates disciplined learning that Cockburn advocates. When there is a high level of requirements uncertainty and changes in the early stages of a project, an agile, adaptive approach is what may be needed so long as there is a mature level of competency in fundamental software engineering practices.

References:

Boehm, Barry, & Turner Richard, Using Risk to Balance Agile and Plan-driven Methods, IEEE computer society (June 2003)

Cobb, Charles Understanding Agile at a deeper level: management of uncertainty, High Impact Project management (2014-2018)

Cockburn, Alistair, Disciplined learning: the successor to risk management, Cross Talk Jul/Aug 2014

https://en.wikipedia.org/wiki/James_Dyson

Kahneman, Daniel Thinking Fast and Slow, 2011

Kenneth Rubin, Essential Scrum: A Practical guide to the Most Popular Agile Process

S. Humphrey A Discipline for Software Engineering, Addison-Wesley, 1995

www.wired.com/2011/04/in-praise-of-failure/

Leonardo to produce VW Dieselgate

Volkswagen has admitted to fitting 11 million vehicles on the planet with fraudulent emission-detection software. Meanwhile Leonardo Di Caprio has announced that he will be coproducing a movie about the scandal at a time when the VW corporation is facing enormous financial repercussions and brand depreciation for their deception around the world. This scandal is putting in the spotlight how corporate fraud can derail an entire quality management system in a company that prides itself on its quality and have dire consequences.

On 18 September 2015 the U.S. EPA served a Notice of Violation (NOV) on Volkswagen Group alleging that approximately 480,000 VWs and Audis, equipped with 2-litre TDI engines, and sold in the U.S. between 2009 and 2015, had non-compliant emissions software installed. But the consequences have been felt in many parts of the world. There are recalls, inquiries, government interventions and numerous class actions taking place in Canada, China, India, the European Union, and the United States.

Australian Owners

Volkswagen owners in Australia are launching a class action over the emissions-rigging scandal with possibly more than 91000 Australian owners being affected.  Mr Scattini from Maurice Blackburn lawyers argues that Australian owners of cars with fraudulent software were emitting many times above the accepted level of poison into the atmosphere. Mr Scattini suggests to drivers that cars fitted with the fraudent emissions-reducing software chip might be noncompliant with Australian emissions laws.  Models affected in Australia include Golfs, Polos and Skoda Octavias built within particular years.

Shifting blame?

 “This was a couple of software engineers who put this in for whatever reason,” Michael Horn, VW’s U.S. chief executive, explained to the subcommittee hearing. “To my understanding, this was not a corporate decision. This was something individuals did.” Horn, revealed that three VW employees had been suspended in connection with software that detects emissions in the company’s diesel vehicles. Reuters is reporting that VW will dismiss the heads of R & D of Audi and Porsche, as well as US chief executive Michael Horn.  Only a couple of engineers were to blame?

Challenge to Toyota on quality and innovation

Back in 2007 at VW’s headquarters in Wolfsburg, Germany, former Audi chief Winterkorn was optimistic about transforming the VW automaker into a world-leading car manufacturer by delivering strong profits and challenging the market leader Toyota on quality and innovation. “We will bring the Volkswagen group to a new and higher level,” said Winterkorn, the incoming CEO in January, 2007.   VW were attempting to gain greater market share on quality so they would have had a strategic direction in quality management. 

Quality Management

 From the 1920s when Bell Labs developed statistical control charts to the 1990s during which ISO 9000 standards gained acceptance in the United States, quality management has been evolving for more than a century in manufacturing.   VW would have a quality management system to ensure that all the attributes of an automobile’s quality- performance, features, reliability, comformance, durability, serviceability, aesthetics and perceived quality are taken into consideration.   The quality of a VW Golf, for example, would comprise of measurable characteristics and their limits of variability.  The same applies to the software components. 

Any quality management system necessarily needs to be a strategic management decision which pervades the entire organization. See the chart below for a model of how various dimensions (eg. people, processes, products) of a quality management system might be visualised within an organisation. 

 (source: Manfred Seika) 

When the outgoing US executive stated that there were just 3 rogue engineers to blame who devised the fraudulent software, my sense is that it is not credible. One can only speculate that it is more probable that managers and engineers from different departments (product development, mechanical, electrical and software engineering) collaborated on the solution in designing the emissions-detection algorithms. The trade-off between fuel efficiency, power and emissions would have been discussed and built into the hardware/software solution. The result was that it created an attractive product for customers, appealing to their environmental conscience, not compromising on power, while appearing to conform to the EPA and other regulations under various conditions.  

It is impossible to know at this time the full extent of the relationships and decisions that went on within the organization that lead to this crisis. Were the managers and engineers trapped in their role to conform to the dominant logic of the US subsidiary, while having poor sleep, with elevated levels of stress hormones?

We must learn that passively to accept an unjust system is to cooperate with that system, and thereby to become a participant in its evil.   M. L. King Jr.

Tesla driven by the environmental Zeitgeist

 In contrast Tesla has been scaling up in U.S. their electric car battery charging station infrastructure. Earlier in March this year, the innovative automaker, Tesla, announced that it had reached a milestone of having 2,000 battery rechargers worldwide, located at almost 400 Supercharger Stations in North America, Europe, Asia, and Australia for their electric vehicles.

 On being questioned about the effects of consumer’s perception of green technologies: Tesla Motors CEO Elon Musk: “What Volkswagen is really showing is that we’ve reached the limit of what’s possible with diesel and gasoline. The time has come to move to a new generation of technology.”

Whether we have reached the limit of what is possible with diesel is another discussion.  But as far as the quality management system at VW is concerned, it appears to have been ineffectual in the face of dishonest and fraudulent behavior at various levels of the organization, not least in the U.S subsidiary in a culture of complicity.  Ethical principles must prevail in corporate governance.  VW Golfs, rogue engineers, clandestine software development, car scandal of the 21st century….  maybe it will make a good movie Leonardo. 

How I discovered how to motivate cross-border virtual teams

“As inner work life rises and falls, so does performance.”

How can we motivate people to do activities at work that they may prefer not to do? For example doing the work of testing software infrastructure when there is little time?   In a manager’ toolkit what can systematically assist members of teams to be productive?  I wanted to test the basic proposition of a book published in 2011,  The Progress Principle, that a manager or team leader can improve a team’s performance by creating an environment to improve their inner work life.   As a team leader, I found that the guidelines to be highly practical heuristics.  People are more likely to act and get work done when they are provided with a supportive environment and a sense of making progress with small wins.

Why?

After I read The Progress Principle several years ago by Terese Amabile and Steven Kramer, I decided to experiment with applying the principles to see whether they are effective in working with virtual cross-border teams.   I needed to find a way to coordinate 20 teams in the Pacific, S. E. Asia, United States and the Middle East in working to tight deadlines, practicing new processes, identifying incidents and handling setbacks on a software transformation project.

The principle of progress?

The Progress Principle describes a method for a manager to create a psychological environment where people are more creative and productive at work.  The authors are husband and wife team, Steven Kramer and Terese Amabile.   Terese Amabile is a professor and director of research at Harvard Business School, with research interests in creativity, productivity, innovation and inner work life. Her team conducted research involving 26 projects teams in 7 companies in three industries. The researchers analyzed 12000 individual diary entries of knowledge workers to gain insight into their inner work life.   This data provided information about the study participants’ perceptions, emotions, and motivations during the day and how the dynamic interplay of these factors influenced performance.

Some conclusions

Some of the conclusions from this research were:

“inner work life influences people’s performance on four dimensions:

creativity, productivity, work commitment, and collegiality”

Also “inner work life matters for companies because, no matter how brilliant a company’s strategy might be, the strategy’s execution depends upon great performance by people inside the organization.”

(Source: The Progress Principle: Using small wins to ignite joy, engagement, and creativity at work. 2011 Harvard Business Review Press. Teresea Amabile and Steven Kramer p.6)

The results of their extensive research was that positive emotions unequivocally have the effect of improving an employees performance at work.

The components of inner work life

The authors describe the inner work life of an individual are a dynamic interplay of:

• Perceptions and thoughts (sense-making of events at work);
• Emotions and feelings (positive and negative emotions about events at work);
• Motivation (the desire to do the work).

My own experience

I have applied this methodology to leading 20 international teams in a multinational corporation as a part of an enterprise-scale software and infrastructure transformation project. I found that people positively respond to work when they know where their work fits into the overall purpose of the organization, when they feel supported and emotionally encouraged in the face of difficulties, and when I regularly emphasize that small daily wins is good progress. Progress can be slow, incremental and at times frustrating. It often can feel as if things are going backwards rather than forwards when the technology or a process is not producing the immediate results you want to see. Many teams were also resistant to changing their working habits with the implementation of new systems.

I developed a habit of frequently, everyday recognising and affirming that we are making progress each time we find a problem or an incident occurs!  This may seem counterintuitive but it works.  People need to know that risks, incidents and faults identified are crucial steps in their remediation in order to make a system robust and fit for purpose.

 An alternative viewpoint

There have traditionally been different views of what the key variables are that affect performance in the workplace. Some observers adopt the stance that short bursts of negative emotion can enhance creativity. According to this view, work performance can be catalyzed by stress, external pressure, discomfort, or any combination of the above. Whereas most of us have experienced stress or a negative event which has disrupted the performance of our work.  Other traditional views have centred on extrinsic rewards such as financial incentives.

An alternative view is that performance improves when positive emotion is elicited in the work environment when a sense of progress is being made.

The authors of The Progress Principle claim to have conducted more comprehensive research than previous studies. Their overall conclusion: positive inner work life promotes good performance.   This is called the inner work life effect.

“People do better work when they are happy, have positive views of the   organization and its people, and are motivated primarily by the work itself.”

Take home message

In my experience of managing cross-border teams on a global transformation program, I found that emphasising the importance of small wins when a problem is found, plays an important part in creating the subjective sense of making progress. Providing the time, resources, assistance, emotional support and encouragement are also critical.   Providing this environment affects someone at a positive, visceral level.   I found that The Progress Principle works. Productivity improves.  A sense of making progress is a key contributor to the inner life of members of a team.   If you want a comprehensive set of heuristics to increase the productivity of your team, then you might find this book practical and surprisingly instructive.

Reference:   The Progress Principle: Using small wins to ignite joy, engagement, and creativity at work. 2011 Harvard Business Review Press  Authors: Teresea Amabile and Steven Kramer

Eastern versus Western ways of knowing in international teams

The opposite of a great truth is also true.

Zen Buddhist Dictum

If you are a Westerner interested in doing business or managing a team with the Chinese and not familiar with cultural differences, it is enormously useful to learn about differences in reasoning when presenting a business proposal or organizational change. As a Westerner being persuasive can be greatly enhanced by understanding some of the fundamental differences in Western and Eastern thought.

Eastern Dialecticism versus Western Formal Logic

The Chinese developed a system of thought, dialectical reasoning, which is opposed to formal logic from the Western tradition in many ways. Eastern dialecticism is at the foundation of Eastern thought and has the following characteristics:

1. The principle of relationships. The whole is more than the sum of the parts. Parts are meaningful only in relation to the whole.
2. The principle of change. Reality is a process of change (what is currently true will shortly be false;
3. The principle of contradiction. Contradiction is the dynamic underlying change;

Whereas at the foundation of the Western tradition of thought is formal logic.

Aristotle exemplified the foundation of formal logic such that:

1. A=A A is itself and not some other thing;
2. A and not A cannot be both the case. A proposition and its opposite cannot be both true.
3. Exclude everything in between. Everything must either be or not be.

Based upon various research of cultural difference, such as by the social psychologist, Richard Nisbett, people who grow up under the influence of Chinese traditions do not accept these propositions as being necessarily true for all kinds of problems. (Source: Richard E. Nisbett (2015) Mindware: Tools For Smart Thinking.)

While formal logic is broadly accepted in Western quarters, people influenced by Eastern thought may find the contradictions in certain contexts since, in Eastern thought, the principle of contradiction is the dynamic underlying change.

Dynamic of change in geographic context

I worked within a team within a global corporation where a senior manager displayed annoyance and complained about a deadline not being likely to be met by the end of the week by a Taiwanese team.   Little did he know that a typhoon had hit the capital and had all but taken the business district to a grinding halt. The Taiwanese team I was working with had understandably stayed at home for a couple of days in the interests of their own personal safety. This story highlights a cognitive blindness caused by concentrating on a specific problem (the need to meet a deadline for a project) without seeking to understand the broader situational context (an extreme weather event). An extreme weather event had not only disrupted the usual business momentum, but more importantly team members’ personal safety at was at stake. This is an example of not seeking to understand in a more holistic manner while failing to take into the geographical context.  Focussing on the brutal logic of a deadline makes no sense.  Relationships are important.  Parts are meaningful only in relation to the whole.  And where is the  humanity?

What to do?

If you are a Western educated manager leading a team of people from countries influenced by Eastern thought, you will have greater influence and illicit greater support if you explain things in terms of the big picture and a broad context. Rather than give directives to individual team members in isolation, you would be better if you provided an overview of how all the pieces fit together in the system, provide a macro-view, and provide insight into what other members of the team are doing.

Nisbett makes an insightful recommendation, which may be applicable to a Western educated manager attempting to influence a team from an Eastern educated audience:

“Sometimes it is helpful to try to dissolve contradictions, but sometimes it’s more productive to acknowledge them and see whether the truth might lie between the contradictory ideas, or whether it’s possible to transcend the contradictions and find some respect in which both are true.” 2

Effective cross-cultural collaboration with our Eastern educated friends and colleagues can be dramatically enhanced with a sensitivity to the cultural influence of Eastern thought. Try to embrace seemingly contradictory ideas and paradox and know that the parts are meaningful only in relation to the whole.

1 Richard E. Nisbett (2015) Mindware: Tools for Smart Thinking

What can we learn about quality management from the VW fraud scandal?

As a manager how do you define quality?  This is a question that must troubling the Quality managers at VW.  When we think of some definitions include ‘fitness for use’, conformance to specification’, or ‘fitness for intended use’, would have these got them out of serior trouble . How can an IT manager be confident that a product or service is fit for use? Incidently what is the quality characteristic that detects fraud?

 

Being confident about a critical software product requires an objective assessment of risk.   Developing a strategy to test software depends on the technical and business context. There is not a one size fits all best practice for all different types of software and infrastructure projects.   While I have devised numerous approaches over the years, one of the most versatile and practical is a risk-based test strategy.  A risk-based test strategy is a good approach in the manager’s toolkit.

Risk-Based Approach

Risk-based testing is an approach which provides a high level of confidence that the right features, interfaces and functions are being tested at the right time.

What is a risk-based approach to testing? (Do you know anyone who has a risk-based approach to their life?)

In project work, there are broadly two types of risks:

1. project risks; and
2. product (quality) risks.

Project risks relate to problems that arise that affect a project’s success. Lack of the right resources is an obvious example. Or the risk of not being able to deliver in the required market in a timely manner. Perhaps VW did not assess the impact to their brand and share price if their systemic fraud was discovered.

Product quality risks depend upon the specified quality characteristics and requirements definition.

Risk Identification

It is desirable to use a combination of a top down and bottom up risk assessment approach. A top down approach uses quality standard classifications and checklists.  Whereas a bottom up assessment means using interviews with experts:  solution architects, software developers and business analysts to gain insight into  the relative importance of risks and quality characteristics for the various components of a system.

A top down method of risk assessment can use software quality standards (eg. ISO9126) or other risk templates. For example the following is an example of quality characteristics derived from several sources:

•  Capability. –   Can it perform the required functions?
•  Reliability –   Will it work well and resist failure in all required    situations?
•  Usability  –   How easy is it for a real user to use the product?
•  Performance. –   How speedy and responsive is it?
•  Installability–   How easily can it be installed onto its target platform?
•  Compatibility–   How well does it work with external components & 
configurations?
•  Supportability–   How economical will it be to provide support to users of the 
product?
•  Testability–   How effectively can the product be tested?
•  Maintainability–   How economical will it be to build, fix or enhance the 
product?
•  Portability –   How economical will it be to port or reuse the technology 
elsewhere?
•  Localizability–   How economical will it be to publish the product in another 
language?

Source: Heuristic Risk-Based Testing by James Bach in Software Testing and Quality Engineering Magazine, 11/99

But what about another that if overlooked could reverberate throughout the company in years to come:  Is the software compliant with relevant regulations?

• Compliance –  Does the software comply with regulations in the various jurisdictions that it will be used in?

Risk Assessment

Assessing the risk is about determining the consequences and likelihood of potential risks. It could be useful to ask questions such as:  How serious is this potential risk? What is the likelihood of the risk eventuating?

Probability

It is important to determine the criteria to assess the probably of the risk happening. These criteria could include complexity, degree of change to a function or component, program size, programming skill etc.

Consequences

What are the consequences if this function or component was to fail?

You could assign a criticality to each risk based upon the impact on the business or company if it were to fail.   A numeric scale may be used to rate the relative criticality.

FMEA

There is a formal procedure which is another alternative which can be used if it suits the context of the project. Failure Mode and Effects Analysis (FMEA) is a formal methodology created to identify potential failure modes for a process or product, to assess the risk associated with those failure modes, which allows ranking of issues in terms of importance, and to identify and carry out corrective actions to address the most serious concerns.

Failure Modes, Effects and Criticality Analysis (FMEA / FMECA) involves the identification of the following information:

• Item(s)
• Function(s)
• Failure(s)
• Effect(s) of Failure
• Cause(s) of Failure
• Current Control(s)
• Recommended Action(s)

Most analyses include some method to assess the risk associated with the issues identified during the analysis and to prioritize corrective actions. Two common methods include:

• Risk Prioritization; and
• Criticality Analysis

Risk Mitigation

The level of coverage of software testing is contextualised by the information gained from the previous activity of identifying and assessing risks. Scope and risk assessment combine to inform a test strategy which may include resourcing and scheduling, techniques of static analysis, dynamic testing, progression and regression, manual and automated, functional and non-functional dimensions in the overall approach.

Juggling time, resources and quality

A risk-based approach to testing is designed to mitigate risks and find the right trade-offs in time, resources and quality.   Quality should not be compromised by other project imperatives.  Is your product’s quality and conformance ready for the world to see?  The systemic fraud that was designed into the emissions software of VW’s cars highlights the imperative for compliance as well all the usual product risks to be mitigated.

Hows your fitness for use?

As an IT manager are you prepared to go against the dominant logic of your organisation if there is systemic fraud and non-compliance?  How is your product’s fitness for use?

References:

Heuristic Risk-Based Testing by James Bach in Software Testing and Quality Engineering Magazine, 11/99

A First Course in Quality Engineering:  Integrating Statistical and Management Methods of Quality (2nd Ed.)  K.S. Krishnamoorthi, V.  Ram Krishnamoorthi.

Risk Based Testing and Metrics, article by Stale Amland for 5th International Conference, EuroSTAR ’99, November 8-12, 1999, Barcelona, Spain.

Software  Risk Management:  Principles and Practices by Barry W. Boehm, IEEE Software, Vol. 8, No. 1 January 1991.

How does a manager manage a multitude of mental models ?

Mental models are the way that we interpret the world. The simple representation we form of something is a model. For a manager this is important since these representations of reality are usually incomplete, imprecise and fuzzy.

Our Mental models are based upon the inferences we make of our a priori assumptions and beliefs. But these underlying beliefs are more often than not outside our awareness.  My experience has been that it is important to respect not only your own but other’s unique perspectives and mental models.  This might mean that a leader or manager would be well suited to applying eclecticism to leading people and solving problems – not being fixed on a single paradigm or set of assumptions, but instead draw on multiple theories to gain a more  balanced  perspective on an issue or problem.

Prejudicial assumptions

 If we see someone of a vastly different cultural background, what initial inner dialogue runs through our mind? Do we need to consciously step in to ensure that we do not act on an unconscious prejudice?   Imagine an elderly white man walking down an alley who encounters a group of black teenagers wearing hooded sweaters. What are his initial thought impressions? Does the elderly white man’s unconscious assumptions produce inferences about their likely behavior as being threatening? It is often the case that people are not aware of what unconscious assumptions influences their behaviour and thoughts.

Organisational Learning

The concept of mental models was popularized in the organisational learning literature of the 1990s in Peter Senge’s seminal work, The Fifth Discipline: the art and practice of the learning organisation. “What we carry in our heads are images, assumptions and stories” is one of Senge’s descriptions.

“ ‘In the traditional authoritarian organization, the dogma was managing, organizing, and controlling,’ says Hanover’s CEO Bill O’Brien. ‘In the learning organization, the new ‘dogma’ will be vision, values and mental models. The healthy corporations will be one which can systematize ways to bring people together to develop the best possible mental models for facing any situation at hand.’ ”

Source: Fifth Discipline by Peter Senge, p.181

Mental model in a successful online retailer

Zappos, an online shoe retailer has been enormously successful, which went from $1.6 million in sales in 2000, to over a billion in 2008. The CEO, Tony Hsieh wrote about the culture of Zappos in his book, Delivering Happiness. One of the mental models that has infused the culture at Zappos is of the importance of connection. Zappos executives conducted research which concluded that a feeling of connection among employees is good for productivity. The greater connection that people feel with their colleagues at work, the happier they are. This feeling of connection leads to greater productivity.   The mental model of the importance of creating happiness as an important key to productivity and morale lead the Zappos executives to also generate a culture of learning. Subsequently free classes taught by other employees were created to contribute to the growth of employees within the company. (Source:  Scrum: A revolutionary approach to building teams, beating deadlines and boosting productivity by Dr Jeff Sutherland)

What can a leader do to facilitate creative mental models?

Create meaning through empowering story telling. The stories should create meaning and in doing so highlight core values and basic rules that are the key to the excellence of running your business.  Creating a positive culture should inform strategy and creative mental models are like a toolkit to achieve the business objectives.