Production Testing: Essential Practices and Benefits

While most software development companies are familiar with staged environments for feature testing, continuous delivery developers are perfecting how software can be tested live under the real-world strain of use. 

Catching every bug in the development stage is next to impossible due to an inability to accurately replicate the weight of the multitude of end-users expected at the production stage. This means it’s a valuable skill to have safe and effective production testing methods.

Testing in production in software development has significant benefits over testing in a staged environment but comes with increased risks and can be tricky. But how does it work, and why should you consider it?  

Production Testing

Testing in production is a valuable QA practice that, when implemented well, can save costs, provide accurate results and gather valuable feedback. Simply, it’s using real user data to run code changes in a realistic environment. 

Production testing is particularly useful for websites or applications with multiple features that need to be rolled out. It is commonly used in continuous development (CD) software. 

Testing in staging is all well and good, but if your feature isn’t being tested in production, there’s no real way to be sure it works. Further, staged tests that are laboriously designed to mimic the production environment will take up many staff hours and still miss bugs. 

Production testing is most powerful than a few types of testing in particular:

Stress testing is a way of establishing how the application handles heavy loads. These tests can be in the form of a spike test, where large amounts of traffic are suddenly sent through and designed to illuminate points at which the application needs to be scaled. Running this test in production is the only way to really know what kind of load you’re getting.

Performance testing – Again, there’s no real way to emulate the real-world load for an application that will be as accurate as reality. Production testing finds bottlenecks with performance tests a lot more reliably when the app is in full swing, and again, this points out areas for scalability.  

Integration testing ensures that each feature integrates neatly with the rest of the application. Running this in production provides more accurate test results and is necessary for continuous integration systems. 

Production does not replace other types of testing. However, running the tests mentioned above in staged environments will reduce the chances of them breaking your app in production, so the same set of tests should be run in pre-production. As with any form of testing, there are strengths and weaknesses to be considered with production testing.

Benefits and Drawbacks of Production Testing

Yoni Goldberg, the software architect and consultant, showed in a survey of over 300 developers worldwide that unpredicted inputs were the most common cause for production bugs after logic issues. This demonstrates that the pre-production testing environment will never fully prepare an application for real-world usage. 

Staged environments simply do not have all the data or load to mimic the real world accurately. This means speed and lag times are hard to test accurately.

They’re also expensive to maintain, so optimizing the testing process to focus more on production testing can save a lot of money in the development process.

Perfecting this process can save vast amounts of time and effort in the development stage, so much so that some companies are ditching some of the pre-production testing entirely. 

With production testing, every possible scenario can be tested. But there are some significant risks and drawbacks, too. 

Testing live runs the risk of exposing your users to a bad user experience. It can also result in corrupt data and response times need to be on point. Reacting to bugs in staged environments is not usually as urgent as in production. 

So, making changes to code while users are involved brings plenty of benefits and some important risks. As such, it’s good to know some best practices for production testing.

How to Go About Testing in Production

Attitudes

Testing in production involves the whole team. Everyone should share the responsibility of production testing, and tests should be run early and often. Trust in your team and your product, and make sure everyone is on call to respond quickly to issues. 

Production testing can be risky! It can affect your users, your business, and any other third parties your software might be integrated with. So to do it safely, it’s important to be prepared.

Tools

Make sure all your tools are up to date. Feature-flagging tools give you control over who sees what and the ability to turn features on and off. This means that you can run tests with changes only visible to the dev team without affecting the end-user experience. In addition, assigning a new feature flag for every feature tested allows it to easily be toggled on and off for bug fixing. 

Another necessary tool is an automation framework. This removes the need to test everything manually, which is simply impossible at scale. A job scheduler runs within this framework to run tests on your behalf. Integrating an alerting tool here means that your job scheduler will let you know if a test fails. 

Make sure everyone on the team is on call. This ties into the shared responsibility mentioned at the beginning of this section.

Test Data

Once you have all your necessary tools and your team is prepared, it’s time to create your data. 

Test data in production is a little trickier than in staging. It needs to be manipulated without causing any issues for end-users. One helpful tip is to have a flagging system so you can tell your bots from real users. This allows you to exclude any testing entities from your analytics easily. 

Test entities should only be allowed to interact with one another, so make sure your automation scripts reflect this.

The Tests

There are multiple ways to do this, and everyone has their own methodologies and frameworks, but it can be a good idea to use a combination of approaches. Choosing multiple methods that complement each other creates a much more powerful way of designing tests. 

An example of three complementary methodologies is Test-Driven Development (TDD), Domain-Driven Design (DDD), and Behavior-Driven Development (BDD). Utilizing a hybrid practice of these methodologies allows the strengths of one to fill in for the weaknesses of the others. 

At the setup and teardown stage, make sure the best practices are followed: log in with clear cookies for setup and make sure to perform a clean teardown after the test is completed. If any test sends up a failure alert, stop all testing immediately.

Risk Mitigation 

Employing risk reduction practices will minimize the damage in the case of a failed test. For example, mitigation practices could involve rolling out an update to a small portion of your user base before making it fully public, known as using a test canary. Then, if there’s a catastrophe, only a tiny fraction of your users experience it. The worst-case scenario then becomes having to roll back to a previous version for your subset of users and fix the issue in a controlled environment. 

There will be specific needs for every team and product, and different types of testing in production will have their own pitfalls and quirks that need to be learned. But following basic principles such as these allow a more streamlined process and frees up your dev team to work on what’s important to push the project forward.

Conclusion

For software development, particularly in CD processes, sandbox or staged environments don’t accurately reflect real-world usage. While the consequences of failure in production testing are potentially more severe, learning how to run tests with live user data involved is an invaluable skill. It can even streamline and reduce the cost of the entire development process. 

Using basic precautions and simple best practices, you can reduce your risk and maximize your data’s accuracy, responding rapidly to issues that arrive and making sure every iteration is an efficient improvement on the last. 

The MuukTest Team