Protecting and Proving Software Strength Through Negative Testing

When it comes to our work as software testers, our goal should always be to ensure that the software we test is of the highest quality, meeting the requirements and expectations of the end users. In order to achieve this, we employ several techniques, such as functional testing, performance testing, security testing, and more. Included among the aforementioned is negative testing. 

Though often overlooked, negative testing is a critical aspect that can be just as important as positive testing. In this article, we’ll explore what it is, as well as why it’s essential and how it differs from positive testing. Stick around to learn about some ready-to-use resources and tools to get you started with systematic negative testing.

What Do We Mean by “Negative” Testing?

Negative testing is the practice of testing software by intentionally providing invalid, unexpected, or invalid inputs, conditions, or behaviors that are not expected from the system. The goal is to identify any bugs or issues that might occur when any of the software’s required conditions are not met or fulfilled. 

Unlike positive testing, which involves providing valid and expected inputs to ensure the software works as intended, negative testing:

• Challenges pre-conditions.
• Challenges usage modes.
• Adds constraints and limitations to the software to see how it responds.

In short, negative testing means intentionally not fulfilling at least one condition the software needs in order to work properly. 

For example: 

• Not giving enough memory: Operating in a low memory (RAM or HDD space) environment.
• Not giving valid inputs: Entering an invalid IP address in an IP address text field.
• Not doing the correct connections: Incorrect connection for any hardware device.
• Blocking certain parts of the system: Disabling JavaScript in the browser.

It’s important to note that negative testing is not the same as:

• Positive testing.
• Edge case testing.
• Corner case testing.
• Unusual operation testing.

While most testers focus solely on positive testing, it’s important to understand that negative testing is essential for providing good test coverage for any feature or functionality. Software can be made more robust and reliable through identifying and addressing potential issues. This ultimately leads to a better user experience.

How is it Different Compared to Positive Testing?

To understand the differences between positive and negative testing, we can start with their definitions: 

• Positive testing involves testing software while affirming every required condition to ensure that it behaves as expected. 
• Negative testing involves negating at least one required condition to see how the software behaves under those circumstances.

Unlike its counterpart, negative testing is usually not based on documented requirements or preconditions. Instead, it focuses on:

• Undocumented requirements.
• Discovering unknown information.
• Ensuring graceful failures or exits.
• Risks from the app’s outer ecosystem.
• Challenging the boundaries and limits.
• Unexpected usage patterns and error-proofing.
• Safeguarding app from potential attacks such as SQL Injections, XSS Attacks, JS Script Attacks, etc.

However, negative testing can also be a great complement to positive testing in terms of increasing coverage. By ensuring that the software can handle unexpected usage patterns and error-proofing it, negative testing can improve the quality of the software and increase its reliability.

Key Activities for Systematic Negative Testing

It is a systematic and effective manner. Systematic negative testing involves a series of activities that can help ensure comprehensive and effective testing of software. The following are some key activities for systematic negative testing:

1. Preparation: To conduct systematic negative testing, it’s important to prepare for the testing phase. This involves:
   1. Identifying product constraints.
   2. Researching about failures. There are many sources to research about failures and adapt them to your testing. Some of the ones I choose to use are:
      1. Social media: Follow hashtags such as #SoftwareFailures, #Failures, #SoftwareCrash, etc.
      2. Community forums.
      3. Create alerts on Google Alerts for software failures to know about the latest software failures and stay up to date with new ideas.
   3. Referring to the written requirements to find:
      1. Boundaries: Mark the territory inside the boundary and the territory outside the boundary. Then focus on the territory outside the boundary.
      2. Class partitions: Identify valid and invalid class partitions. Focus on the invalid class partitions.
2. Consider negative personas: Negative testing should also take into account negative personas and consider how they might try to break the system or violate any of its conditions. Some such personas are:
   1. Hackers.
   2. Attackers.
   3. Script kiddies.
   4. Unintentional Disrupter (i.e., an unintentional mistake by a genuine user).
3. Work towards constraints and conditions: The next step is to work towards constraints and conditions. This involves:
   1. Identifying.
   2. Reversing
      1. One condition at a time.
      2. Multiple conditions at a time.
   3. Removing
      1. One condition at a time.
      2. Multiple conditions at a time.
4. Verify error handling: It’s essential to verify error handling and check for error proofing during negative testing. This includes checking for:
   1. Error Control
   2.  Error Protection | Prevention
   3.  Meaningful Messages
   4.  Error Correction | Guidance
   5.  Error Recovery
   6.  System Recovery

By following these key activities, it’s possible to conduct comprehensive and effective negative testing of software. This can also help with identifying potential issues, deeper vulnerabilities, and product risks.

Tools and Resources

To conduct effective negative testing, testers can take advantage of ready-to-use tools and resources such as:

1. Browser Extensions: Two popular ones are Counterstring and Bug Magnet.
   1. Counterstring helps generate strings of characters of a particular length, which can be used for testing form field boundaries, while Bug Magnet provides test data with common input errors.
2. GitHub – minimaxir/big-list-of-naughty-strings: The Big List of Naughty Strings is a list of strings that can cause issues when used as user-input data. This can be a great resource for negative testing.
3. Common Software Errors – Checklist – Testing Titbits | Rahul Parwal: This is a comprehensive checklist of common software errors that can be used for negative testing. It can be used to identify potential issues and vulnerabilities in software.
4. E-book on Software Bug Stories: Reading about the experiences of others can provide valuable insights into software bugs and how to test for them. Software Bug Stories is a website that provides real-world examples of software bugs and how they were discovered and fixed.

In Summary

Negative testing is a critical part of software testing that focuses on identifying potential issues and vulnerabilities in software by challenging its boundaries and limits. It involves testing the software by negating at least one required condition, thereby ensuring that the software can handle unexpected usage patterns, gracefully exit, and protect itself from potential attacks.

To conduct effective negative testing, testers need to engage in key activities, such as identifying product constraints, researching failures, considering negative personas, and verifying error handling. Additionally, they can leverage various tools and resources, such as browser extensions, checklists of common software errors, and real-world examples of software bugs.

By incorporating negative testing into their overall testing strategy, testers can increase test coverage and ensure that software is reliable, secure, and free of errors. Therefore, negative testing should be an essential component of any comprehensive testing plan.

For all our visual readers, I have summarized this entire article in this mind map.

I hope this article has helped you better understand the art of systematic exploratory testing.