What Truly Matters in a Software Test Report?

Your team just uncovered a big issue in your upcoming product 3 weeks before going out to market. You know this will have a negative impact, but it seems none of the decision-makers are aware of this. To avoid these situations, we need to communicate properly and in time the results of our testing. We need to show off our work.

Much has been said about the importance of testing activities to achieve an acceptable level of quality within software development. However, this information might remain hidden if we fail to communicate it to the key players within our organization. Worse, important decisions might not be made at all due to this lack of information.

But how to decide what we need to present? What do my stakeholders want to see? What would help them to make decisions?

It is important first to know our audience. Are we presenting to our team members, to our manager, or to development? This first key step will help us to define what information actually matters in our report. Let’s say you will be presenting the top ten defects to development. You could include some information like the project name or the current progress of testing. On the other hand, a project manager will find this information very useful.

This does not mean you need to create a specialized report for each person that wants to know information about the work you are doing. Instead, we can define some essential content and later tweak our report to present it to a different public.

The essential content can be presented within three basic sections that almost every software test report has:

1. Project Overview
2. Test summary
3. Defects.

Let’s describe each one to have a better understanding of what information we should add to them.

3 Key Software Test Report Sections

1. Project Overview

This section contains general information about the project. We can include the different line items we are working on, a reminder about important upcoming dates, the current version/level of software being tested, whatever information is relevant. It is important that this first section gives a general idea of the overall status of the project. Details will be presented in the following sections, but this first glimpse of the information should be clear enough for the stakeholders to get an idea of how things are going.

Suppose there is a company called “Example”. They are developing the next version of their “Example SW” that will be released in September 2021. By looking at the summary of their software test report, we can see they might run into problems to achieve their releasing dates.

Example line item 2 is only 25% completed. Since this is a July report, they only have 1 month to catch up. We don’t know yet why Example line item 2 is so behind in the test execution.

2. Test Summary

We just had a general glimpse of the status. Here we can explain the details. It is important to present only relevant information. Try not to include every single detail about your execution. Do include the information that gives your audience the reasons behind the previous overview. Following our example, we can quickly mention that line item 1 and line item 3 are almost completed and focus on the reasons behind the red and yellow statuses of line items 2 and 4.

In our example, there is a very disruptive defect found in line item 2. It is important to communicate as much information as possible about this situation to the stakeholders because they are the decision-makers. Also, if you already have alternatives to continue your work, let everybody know about it.

3. Defects

This piece of information could become a trap. If you include all the defects your team finds, it might be overwhelming for your stakeholders.

Instead of presenting an endless list of defects, try to ask your potential audience what they would like to see.

For example:

• Top 10 defects
  • Defects that might be disturbing your execution or are potentially destructive or will impact highly on the quality of your product.
• Defected opened since the last software test report
  • You will present only the defects opened since the last time you presented; you may even preserve in the list the defect with higher priority.
• Open defects only
  • Report only defects that development has not fixed yet so the stakeholders can evaluate where to put development efforts.

There is nothing written in stone here. If your stakeholders require a full list of all defects, you might want to put this information on a separate document and present it alongside your main report. The key here is to learn with your audience what they find valuable so you can gather that information.

These 3 sections can assemble our base report. From it, we can add some other sections or extra information to create customized reports for other stakeholders.

The ISTQB Foundation Level guide mentions other points that should be taken into consideration:

• Deviations from the plan, including deviations in schedule, duration, or effort of test activities.
• Residual risks, risks that might not be fully contained despite test efforts, or risks that can be assumed.
• Reusable test work products produce, such as automated tests, tools to ease or work, even documentation that can be used in future rest cycles.

While these topics are also relevant, it is very important that you ask for feedback from your audience. This way you will eventually build a software test report that fits perfectly in your organization.

How Often Should Test Reports Be Presented?

That depends totally on how your organization works. Are you executing under a scrum framework? Then this report can be part of the Sprint Review where developers can request feedback. Are you working under a waterfall model? Surely, there will be weekly status meetings where you will be asked for this information. No matter the frequency you need to report, keep in mind where to collect your info so this report will not become a burden for your daily activities.

I want to remark how we moved from general to specific information as we progressed through the software test report.

First, we showed an overall view of our progress and we ended with very specific information about the defects discovered. It is important that you consider this flow of information so your stakeholders can follow you through the presentation.

Second, I want you to notice the visual aids in the “Example Company Test Report”. We colored in red the information that we consider important, so the audience quickly notices it.

To finalize, remember to be consistent throughout all your software test report and the future reports you present. Do not color in orange failed tests in August’s report then grey in September’s. Your stakeholders will thank you if you ease their work.

Remember, this will be an input to make important decisions. We want this information to reach decision-makers in the clearest and easiest way.