Tips for Requesting a Test Report

A test report is a document that provides information about the quality of a product or service. It may also include recommendations for improving the product or service. A test report can be written by a team from a third-party organization or by an internal team at the company that developed the product or service. Test reports highlight what was done, what was discovered, and what needs to be done next.

Test reports offer many benefits, but some of the most prominent are

• It helps us understand the quality of our product
• It provides us with an opportunity to improve our products and services
• It helps us gain insights to maintain customer satisfaction

In this article, I will discuss some key recommendations for requesting a test report from your testing team. This article will help you gain valuable insights from your test team.

Some Important Questions

Test reports are often discussed in terms of format and various sections. However, a test report should fundamentally answer the following questions.

How much testing have we done? Were there any significant findings from it?

There are several points covered, such as

• Scope
• Status
  • Done.
  • Pending (To Do).
  • In Progress.
• Risks.
• Blockers.

How thorough has our testing been? 

The depth of testing is discussed here. A quick mnemonic to remember the various testing levels is Q PRA DEEP.

• Q – Quick testing
• PRA – Practical testing or well-balanced testing
• DEEP – Deep testing or overly detailed testing

If you want to learn how different levels of testing work, you can check this article by Pradeep Soundrarajan.

How effective has our testing been?

This aims to determine whether we are focusing on the right areas. Information presented in this category typically includes:

• Product elements considered for testing.
• Quality criteria/attributes checked and assessed as part of the testing.
• Different coverage types that you are considering. For instance, code structure, function, data, interfaces, platform, and timing coverage.
• Types of testing done/planned.

Have we tested enough?

The conclusion of a good test report often includes recommendations for the subsequent rounds of testing before making any decisions. It mentions how much time the team needs before making the:

• Alpha release
• Beta release
• Stable release

Key Components of a Test Report

After evaluating the important questions that should be answered in a test report. Let’s now look at some of the key components that can be included in a test report to help you get valuable insights:

Report Overview: This is ideally the first section of any report that consists of key data points for a quick gist of the entire report. There are points covered, such as:

• Summary – High-level overview of the report
• Mission – Purpose of testing
• Scope – The various areas or aspects of the tested product
• Version of Product
• Key Metrics
• Status
  • Completed
  • In-Progress
  • To-Do
• Test environment details
• Duration of the test cycle
• Any key recommendations

Issue Overview: It covers a summary of key issue items, such as:

• Blockers (for testing) – Problems that require immediate attention or threaten the success of the project
• Recommended to fix (before release) – Major issues that threaten product value
• Can be postponed (for next release) – Not all issues or bugs need to be fixed immediately. There are always things that can be postponed or scheduled for a later time.
• Anything else?
  • Change requests: Findings that are not bugs or critical issues but are usually recommendations for improving a product feature or attribute.
  • Known issues for release notes

Test Notes: This is the main section of any test report, which covers the entire testing story. This is usually for stakeholders who need a clear understanding of the testing process and plan. Details may include:

• Test process
• Product coverage
• Notable test results
• Contains key issues (worth noting)
  • Consists of:
    • Issue title | ALM Link (Ex: Jira link)
    • Issue summary
    • Risk
    • Implications
• Open questions
• Assumptions
• Other relevant details

Pitfall: Surrogate or Proxy Measures

It is common for traditional test reports to include specific test metrics. However, many of the attributes we wish to study do not have generally agreed-on measurement methods.

Some factor that can be measured is used instead to overcome the lack of measure for an attribute. These alternate measures are called surrogate measures. It is a widely used opportunity for disaster! 

Ex: Measuring testing productivity and effectiveness via:

• Bugs
• Test case count
• Test script automation %
• Pass vs. fail %

Pitfall

• Wrong assumptions that these measures would reflect the effectiveness of testing.
• They are mostly used because they are easy to count!

If the team is asked for too many surrogate measures, distortion and dysfunction can arise.

A measurement system yields distortion if it incentivizes a person to make the measurements look better rather than optimize for achieving the organization’s goals.

A system is dysfunctional if optimizing for measurement yields so much distortion that the result is a value reduction. The organization would have been better off with no measurement” than with this measurement.

Other common pitfalls:

• Repeated data in multiple sections – A test report that contains duplicate data lacks intensity and crispness.
• Grammar and typo errors – Most document editors support proofreading; plus, it is always a good idea to ask your team to review the report before sending it.
• Sticking to fixed fields in the reports – Each project is unique, and it’s essential to know the fields that fit your context and need. Fields can always be added or removed as required.

For all our visual readers, I have summarized this entire article in this mind map.

I appreciate you taking the time to read this article. I hope this article has helped you better understand how to request an efficient test report. Our next article will discuss factors to consider when reviewing a testing strategy.