AI in Testing Roadmap: Learning Pathway

AI is the latest buzz in the industry and it will only continue to grow. The software industry is looking for ways to integrate Artificial Intelligence (AI) with their testing process. As an AI in testing evangelist and tech blogger, I’m excited to share a pathway to learn and create a personalized roadmap for your AI in testing needs. 

In this article, let’s take a step-by-step approach to help you understand the power and limitations of Generative AI (Gen AI) in testing. Each step will include links to several resources to help you get started. However, you will need to conduct your own research as you delve deeper into a topic.

Step 1: Understanding Fundamentals and Usage of Gen AI in Testing

Understanding Fundamentals

The first step towards learning Gen AI as a tester is to understand the fundamentals and various use cases of implementing AI in testing. 

Let’s have a look at some practical resources that I recommend on this topic:

• Introduction to AI/ML in Software Testing: AI and Machine Learning Apps in Software Testing (muuktest.com)

Before diving into Gen AI, it’s essential to understand the basics of Artificial Intelligence (AI) and Machine Learning (ML). This blog covers the basic aspects of AI & ML Apps in software testing.

• Introduction to Generative AI: Generative AI – What is it and How Does it Work? (nvidia.com)

Generative AI is a branch of AI that focuses on creating new content. Unlike traditional AI, which analyzes existing data, Gen AI generates new data based on patterns. This article from Nvidia will help you understand the basics of this topic.

• Conceptual Guide to Transformers: A Conceptual Guide to Transformers: Part I (substack.com)

Transformers are the backbone of many Gen AI models. They use attention mechanisms to process input data and generate output. This multiple-part blog series covers everything from the basics to the advanced ideas on how transformers work.

• What is ChatGPT Doing and Why Does It Work?: What Is ChatGPT Doing … and Why Does It Work?—Stephen Wolfram Writings

ChatGPT uses transformers to understand and generate human-like text. It works by predicting the next word in a sentence based on context. This article by Stephen Wolfram talks about how it all works and makes sense. This piece will help you understand why sometimes it works and sometimes it doesn’t.

• The Rise of Generative AI: Judgement Day” with Tariq King, PhD: StarWEST 2023 Keynote

For a high-level overview of the basics, I recommend watching Tariq King’s talk on “The Rise of Generative AI.” as he explains the transformative potential of Gen AI in testing and beyond.

Understanding Gen AI Usage in Testing

Now that we have a solid foundation, let’s explore various ways how Gen AI can revolutionize software testing.

• Test Data Generation: Gen AI can create diverse test data sets, saving a tester, hours of manual work.
• Test Idea Generation: Need fresh test ideas? Gen AI can give innovative scenarios.
• Test Notes Transformation: Transform raw test notes into structured reports effortlessly.
• Changing Data Format: Convert data formats seamlessly with Gen AI.
• Bug Report Drafting: Draft detailed bug reports quickly.
• Email Drafting: Automate email communication with stakeholders.
• Programming Code: Generate code snippets for test automation.

1. Requirement Analysis: Extract key requirements from documentation.
2. Extracting Notes: Extract important information from your notes.
3. Questions: Identify ambiguities in requirements.
4. Specific Data Patterns: Detect specific patterns in data files.

Summarization 

1. Product Documentation: Summarize lengthy product documents for quick learning.
2. Market Research: Condense market research findings to find key risk areas, or opportunities.
3. Test Strategy: Create concise test strategies or easily get a summary of a test strategy on an existing project.
4. Testing Notes: Summarize testing sessions.

1. Domain Concepts: Learn domain-specific knowledge.
2. Error Messages: Understand common error messages and how to debug and fix them..
3. Programming Concepts: Get examples and explanations for programming concepts.
4. AI Concepts: Deepen your understanding of AI concepts and terms.

Disclaimer: Don’t share sensitive data on publicly trained LLMs. Use licensed tools where data is secured or mask sensitive data before feeding it to LLMs.

Step 2: Follow Good Sources and Try Out Gen AI in Testing Tools

Follow Good Sources of Inspiration

AI in Testing is a growing field. There are new updates, resources, tools, tactics, etc. coming out each day. To stay ahead in the game, it’s crucial to learn from experts and follow good sources

• Conferences: Software Testing Conferences

This contains a list of software testing conferences to attend and find inspiration from. You can visit the conferences happening near you or go for virtual conferences. Make sure to filter for AI Tracks or AI focused sessions of your choice when attending any conference.

• Webinars & Events: Testing Daily

This is an app that gives you live feed of all the software testing events and webinars taking place. You can find latest webinars and updates on AI in Testing directly from here.

• Blog: Follow blogs like MuukTest Blog for more posts on AI in Testing. 

MuukTest is evangelizing the AI in Testing space with new tools and learning materials. Bookmark it!

• Video Series: AI in Testing by Rahul's Testing Titbits - YouTube

If you are a video learner, you can check out my video series on YouTube for a structured and continuous learning plan.

• Experts to Follow:

• James Bach | LinkedIn
• Rahul Verma | LinkedIn
• Michael Bolton | LinkedIn
• Tariq King | LinkedIn
• Rahul Parwal | LinkedIn
• Subbarao Kambhampati | LinkedIn

Try Out Gen AI in Testing Tools

Experimenting with Gen AI in Testing tools is one of the best ways to learn and witness the potential those have for your use cases.

I would recommend you to try out any of these popular tools such as:

• Introducing Amikoo (muuktest.com)
• ChatGPT
• Google Gemini
• Gretel AI
• Claude AI, etc.

Step 3: Learning Core Skills as well as the Risks coming from Gen AI in Testing

In this section, we will cover topics that will help you uncover the possibilities as well as challenges in using AI in Testing. 

We will start with covering topics around core skills and then delve deeper into potential risks and other challenges that one needs to be aware of to make effective use of Gen AI in Testing.

This Mukktest article is a precursor to the current article and covers the various skills as well as supported resources for doing AI-assisted testing. It is highly recommended that you check this article out and get a foundational understanding of topics like:

• Basics of AI and ML
• Understanding Large Language Models (LLMs)
• Recognizing LLM Syndromes & Side-Effects
• Using AI Testing Tools
• Navigating AI Ethics
• Prompt Engineering Basics

2. Prompt Engineering: Mastering prompt engineering is key to effectively using Gen AI tools for your needs and context. Here are some curated resources for you to check out:

• Prompting Techniques: Prompting Techniques | Prompt Engineering Guide (promptingguide.ai)

This is a collection of prompting techniques with examples to help you master the fundamentals of prompting.

• Prompting Checklists: Prompting Checklist for Testers - Rahul's Testing Titbits

This is a collection of various hacks and tactics to help you create good prompts for your testing tasks.

• Prompting Hub: AI Prompt Repository for Testers - Rahul's Testing Titbits

This is a collection of ready-to-use prompts for popular testing use cases to help you get good responses from popular AI tools.

3. Critical Thinking on AI Claims: A Super-Quick Guide to Evaluating “AI” Claims – DevelopSense

4. LLM Reversal Curse Limitation: [2309.12288] The Reversal Curse: LLMs trained on "A is B" fail to learn "B is A" (arxiv.org)

Step 4: Starting Gen AI in Testing with High-Value Tasks

Identify high-value tasks where AI can make an impact

• From your work: Refer to Step 1 > “Understanding Gen AI Usage in Testing” section, and find and list out the day-to-day tasks that you can leverage via AI. The tasks could be of varied nature and can be anything such as:

1. Testing related
2. Management related 
3. Presentation & Documentation related 
4. Miscellaneous.

• Focus on easy targets: Once your list is ready, it’s time to prioritize the tasks starting with the lowest-hanging fruits. This will help you make incremental progress.
  
  

What can you learn from my experience?

1. Be consistent throughout the learning process. Block out and invest at least 30 to 45 minutes per day to get consistent results.
2. Follow experts and ask questions. Always be open to possibilities. 
3. Present your work to your project team, organization, or network. Demonstrating value creation unlocks future growth potential.
4. Don’t stop at the first AI response. Ask follow-up questions, think critically, then experiment, and evaluate. AI generates data based on mathematical probability models. It may not give you what you are looking for immediately.  Be Patient and Keep iterating.

Conclusion

The integration of Generative AI in software testing is a pathway to efficient, and innovative testing practices. By understanding the fundamentals, leveraging practical applications, mastering core skills, and addressing risks, testers can unlock new possibilities in their workflows. So what are you waiting for? Dive into the world of Gen AI today and transform your testing journey!