Redefining the Testing Pyramid: Adaptation Strategies

The testing pyramid has dominated industry thought regarding how to create and execute effective automation testing strategies. However, the modern demands on the industry have found it to be lacking an individualized automation strategy

Is the testing pyramid obsolete? We outline the pros and cons and describe a solution that experts believe proves that the testing pyramid still works.

What Is the Testing Pyramid?

The test pyramid is best understood as a heuristic model that outlines three different categories of software testing to ensure end-product performance. These categories are: unit testing, integration testing, and UI/exploratory testing. 

The model aims to guide software developers through a process of optimal testing before a product is released.

The testing pyramid was developed by Mike Cohn in 2009 and has since dominated industry thought regarding systems testing and the baseline process for automation. It establishes different layers of testing complexity and illustrates the most efficient and cost-effective processes to automate.

However, in recent years, the increasing demand for automation and production testing has found areas of the testing pyramid lacking. 

As the demands on the industry have become more complex, multiple pyramid variations have surfaced based on Cohn’s original model. Unfortunately, these haven’t delivered the clarity that the industry requires. 

This has led to many in the industry asking if the testing pyramid is now obsolete. We look at exactly what Cohn’s model offers, the challenges the industry faces, and why the testing pyramid still has a place in automation testing strategies today.

The 3 Layers of Mike Cohn’s Testing Pyramid

Cohn’s testing pyramid separates the process of testing automation into three categories. It also outlines reductions to the recommended number of tests and automation processes at each level.

The three testing categories are:

Unit tests

This forms the bottom layer of the testing pyramid. It consists of testing single units of code to ensure optimal functionality. 

In Cohn’s model, these tests are viewed as the most easily automated, as their scope is small. The tests target one variable and do not have external dependencies. 

These tests are best run using virtual devices such as simulators and emulators. This is effective because virtual devices provide the necessary feedback while being faster and more cost-efficient.

Integration tests

This is the second layer in Cohn’s model, and it covers testing across integrations of more than one system. This layer is more complex as it requires more features to be assessed and a variety of different tests to ensure end-product usability.

Testing should take place using both virtual and real devices in order to achieve optimal results. For example, elements such as acceptance testing perform best using virtual devices, while performance, security, and certain accessibility features need to be tested using real devices.

The traditional view is that a large amount of automation testing can take place at this level.

UI/Exploratory Tests

The top of the testing pyramid consists of UI and exploratory testing, which is usually more costly and labor-intensive. 

The tests run at this level need to account for more dependencies and integrations. While UI testing can largely be automated, exploratory tests require human input to succeed.

These tests should solely take place on real devices. This is because it more closely represents the software function on a user’s device and allows for real user simulation.

Is the Testing Pyramid Obsolete?

The industry has faced many challenges trying to meet the rapidly changing testing demands in the modern era. As a result, many industry members have come to doubt the utility of the testing pyramid. 

Let’s unpack some of these challenges and the proposed solutions.

Expected vs Unexpected Outcomes

The industry’s main challenge when relying on the testing pyramid is that it is designed to look for specific outcomes determined by the developer. 

Unfortunately, this limits the testing ability to expected outcomes and doesn’t allow the testing structure to account for unexpected outcomes. As a result, most of the tests designed in this fashion inevitably break during the testing process. 

This is not only inefficient and time-consuming during the later stages of testing, but makes it incredibly difficult to automate the early stages. It also fails to account for the increasing requirement for production testing.

However, experts in the industry have realized that this may come down to how the industry is utilizing Cohn’s model, rather than the fallibility of the model itself.

Approval Testing

One approach suggests reintroducing more human intervention at different levels of testing. Known as approval testing, this method works by programming the automation so that it is only required to detect differences and present them to the developer for approval. 

This allows developers to be aware of both expected and unexpected outcomes and to make situation-based decisions that inform the new baseline of testing for the next automated process.

However, other experts believe that the testing pyramid is just one area where the industry is encountering challenges. A more top-down approach is needed to establish an automation testing structure that works.

Individualized automation strategies

A case has been made to empower development teams with the ability to create individualized automation strategies. 

These strategies should leverage off of: a team’s available skills, the requirements of specific projects, the needs of stakeholders, product knowledge, understanding of the technological variety, and an assessment of the testing infrastructure.

Experts in favor of this approach maintain the utility of certain aspects of the testing pyramid while indicating that many other traditional ways of thinking need to be reworked. Primarily, they see the testing pyramid as a heuristic model and invalidate methods that treat it as a copy-paste concept. 

They advocate that one’s ability to test and one’s ability to automate are separate functions. As such, automation is a tool to improve testing, whereas testability is the industry’s core function. 

This change in thinking requires development teams to expend more energy on the creation of an automation strategy prior to implementation. It takes the view that developers need to ensure that their strategy is individualized to meet context-determined requirements to overcome today’s obstacles.

Creating an Individualized Automation Strategy

Individualized automation strategies aim to address industry challenges by asking development teams to change their thinking in the following six areas:

1. Team skills
2. Understanding stakeholders
3. Product knowledge
4. Project scope
5. Technology assessments
6. Established infrastructure

We outline the recommended changes for each area below:

Team Skills

Developers need to assess their combined team skills for variety. The more available skills, the more opportunities there are for complex problem-solving. 

A team needs to have the ability to work with multiple languages, tools, and devices in order to optimize their potential to execute an automation strategy.

As a result, new recruits should be considered across multiple skill profiles, and strategies need to include plans to up-skill existing team members.

Understanding Stakeholders

Stakeholder requirements need to be understood more fully. It isn’t enough to follow an instruction without understanding why it has been issued.

Stakeholders often don’t speak the technical language; that’s why they hire a development team. However, if a team can understand what problem they are trying to solve, they are more likely to be able to deliver an optimal solution (even if it differs from the original request). They are also better equipped to explain their testing decisions and timelines.

Product Knowledge

A more in-depth understanding of the product in use is essential. It will be impossible for a team to understand the context of their results and assess validity without thorough product knowledge.

This knowledge will also empower developers to make more educated recommendations to stakeholders regarding which parts of the product should be automated and which parts need to be tested manually.

Project Scope

Knowing the project requirements is essential to developing an efficient automation strategy and delivering results. 

If a team has a two-week timeline, they won’t be able to make use of tools that require a longer development pipeline. However, if they have six months to work on a project, it radically changes the scope of the strategy they can execute. It is crucial to understand the time frame vs input to automation in order to streamline efficiency and validate proposed solutions.

Technology Assessments

Having an intimate understanding of the technology being used also makes a huge difference to a team’s automation testing strategy. 

Frameworks are continually changing, and understanding the details of the framework they are operating in will allow them to make better decisions regarding which tools will deliver the best results.

Established Infrastructure 

What infrastructure environment is the team working in, and do they have the time to create additional environments? 

It is really important for a team to adapt their automation strategy to the available infrastructure. This will make it possible to deliver on stakeholder requirements while managing their expectations. 

It is also important to remember that stakeholders rely on a team’s expertise to determine the realities of their development projects. A team needs to be able to provide and motivate this feedback.

Once a development team has established the individual context for its automation strategy, it can then make use of the testing pyramid with renewed confidence.

Optimizing the Testing Pyramid: How to Get the Best Results

The traditional heuristic testing pyramid fits perfectly into an automation strategy once that strategy has been individualized (as described above).

The key lies in understanding that the testing pyramid is a heuristic model and not a required end outcome to a successful automation strategy. 

As a result, you can safely use it as a basic guideline to your automation testing strategy in so far as you accompany it with a more in-depth analysis of your testing aims.

These aims should ensure that your testing strategy is targeted, reliable, informative, maintainable, and efficient.

• Your tests should be targeted to individual layers of the stack (not solely aimed at the UI) and focus on the lowest point required to mitigate risk while automating at that layer.
• They need to be reliable enough that you don’t need to keep changing your strategy.
• The feedback you receive should be informative enough to ensure that you understand what is going on in a broader context.
• Ideally, they will be maintainable because you will have to maintain them as you make changes. Not to mention, regular maintenance allows you to get up and running faster.
• Speed and efficiency are major industry requirements. The better thought out your strategy and test implementation are, the faster you will produce results.

Remember, if an individualized automation strategy doesn’t come out looking like the traditional testing pyramid on every project – you should consider that a success. 

After all, the value of the testing pyramid lies in the fact that it’s a heuristic model, and you need to adapt its insights to support your requirements.

The Bottom Line: Why the Test Pyramid Still Matters 

The testing pyramid is a foundational model in the industry for a reason. It provides key insights that are useful and offers developers a starting point from which to build an automation testing strategy.

Times have changed, but you don’t want to throw the baby out with the bathwater. The testing pyramid still has a lot to offer developers. The value is evident so long as they recognize the necessity of creating individualized automation testing strategies and how best to integrate Cohn’s testing pyramid within them.

What’s more, the industry’s challenges prove that while automation is efficient, developers will always need to play a part in testing processes to achieve optimal results.