Most teams assume flaky tests are a random nuisance, but they're actually a predictable outcome of five specific design failures. The cause almost always traces back to asynchronous timing issues, shared mutable state between tests, external service dependencies, unstable environments, and resource contention during parallel execution. Fixing flaky tests for good means addressing one of these five root causes directly, not adding retries and hoping the problem goes away.
TL;DR
- Flaky tests are a root-cause problem, not a random annoyance: nearly every flaky test can be traced to timing, state, dependencies, environment, or concurrency issues.
- The cost is real and measurable: flaky tests can cut team output significantly and cost large engineering organizations hundreds of thousands annually in wasted compute and developer time.
- Retries mask the symptom, not the disease: auto-retry logic hides flakiness long enough for it to compound into a trust problem across the whole pipeline.
- Playwright's auto-waiting model reduces a major category of flake compared to manually scripted waits, but it does not eliminate state or environment-based flakiness on its own.
- A flake rate below 1 to 2 percent is the benchmark to aim for; above 5 percent, trust in the suite breaks down, and above 10 percent, flakiness is actively slowing down releases.
About the Author: This article is written from MuukTest's experience building and maintaining automated test suites, including Playwright-based frameworks, for growing engineering teams. MuukTest's QA engineers and Amikoo-powered QA Agents are built specifically to catch and resolve flaky behavior before it reaches a client's CI/CD pipeline.
What Are Flaky Tests?
A flaky test is any automated test that produces inconsistent results against an unchanged codebase: green on one run, red on the next, with nothing in the application code to explain the difference. This is distinct from a test that fails consistently because of a real bug. Most engineering teams encounter flaky tests regularly, making flakiness less of an edge case and more of a standard operating condition for test automation frameworks, which is exactly why treating it as "normal" is the first mistake teams make.
Think of a flaky test like a smoke detector that sometimes beeps when there's no fire and sometimes stays silent during an actual fire. The alarm's signal only has value if it's trustworthy; once people start ignoring it, the one time it's right goes unnoticed too. A flaky test suite does the same thing to a release pipeline: engineers stop trusting red builds, start re-running failures out of habit, and eventually stop looking closely at results at all. That's the real cost, separate from the wasted compute time.
Why Do Flaky Tests Happen?
Flaky tests happen because automated tests make assumptions about timing, state, and environment that don't hold true on every run. Documented root causes fall into five categories: asynchronous timing issues, shared mutable test state, external service dependencies, environment instability, and resource contention during parallel execution [harness.io][trunk.io][datadoghq.com]. Each has a different fix, which is why a single blanket strategy (like "just add retries") rarely works for long.
Race Conditions and Timing Issues
A test clicks a button before the page has finished rendering, or asserts on data before an API call resolves. This is the single most common cause of flakiness in UI automation frameworks like Selenium and Playwright, particularly when tests rely on fixed sleep timers instead of waiting for actual application state [pie.inc][buildpulse.io].
Shared or Leaked State
One test modifies a database record, user session, or global variable, and a later test depends on that state being untouched. Run the tests in a different order, or in parallel, and the second test fails unpredictably. Lack of test isolation is consistently cited as a core cause of flaky behavior [trunk.io].
External Dependencies
Tests that call third-party APIs, payment gateways, or shared staging environments inherit the reliability of those systems. A flaky payment sandbox or a rate-limited API will produce flaky test results even when the code under test is perfectly stable [semaphore.io][ranorex.com].
Environment Instability
Differences between local, CI, and staging environments, think container startup times, network latency, or mismatched dependency versions, introduce variability that has nothing to do with the code being tested [harness.io][ranorex.com].
Resource Contention in Parallel Execution
Running tests in parallel speeds up a suite, but it also means multiple tests may compete for the same database connections, ports, or file locks. Resource contention during parallel execution is a common source of flaky failures [pie.inc], which means a large share of flakiness has nothing to do with the application logic engineers assume is at fault.
How Do You Identify Which Tests Are Flaky?
Identifying flaky tests starts with measuring a single metric: the Flaky Test Rate, defined as the percentage of test failures that pass on a re-run without any code changes. Healthy test suites keep flaky rates below 1 to 2 percent; by the time a suite reaches 5 percent, engineers stop trusting the test results and green builds lose their credibility, and anything above 10 percent is actively blocking release velocity. Tracking this number over time, rather than reacting to individual failures, is what turns flaky test management from firefighting into a process.
- Re-run and compare: automatically re-run failed tests in isolation and flag any that pass without code changes.
- Track failure patterns over time: a test that fails intermittently across many builds, regardless of what changed, is a strong flaky candidate.
- Quarantine, don't delete: move suspected flaky tests out of the blocking pipeline into a monitored queue so they don't stall releases while still being tracked [blog.testery.io].
- Correlate failures with infrastructure events: CI runner restarts, network blips, or deployment windows often line up with spikes in flaky failures.
Does Selenium vs Playwright Matter for Flaky Tests?
Building on the root causes above, framework choice matters because it determines how much timing-related flakiness gets built into a suite by default. Selenium requires engineers to write explicit waits for most dynamic UI conditions; skip that step, and race conditions creep in quickly. Playwright instead auto-waits for elements to be actionable before interacting with them, and recommends auto-retrying assertions, such as expect(locator).toHaveText(), in place of manual timeouts. Playwright also advises isolating tests by giving each one its own backend data using unique identifiers, directly addressing the shared-state category of flakiness.
That doesn't make Playwright flake-proof. Playwright test coverage still degrades if a team ignores isolation guidance or runs tests against shared staging data. The practical takeaway: a modern test automation framework reduces one category of flakiness by design, but the other four causes (shared state, external dependencies, environment instability, resource contention) require deliberate engineering regardless of which tool sits underneath.
What Does Flakiness Actually Cost a Team?
A related but distinct question from "why do flaky tests happen" is what they cost once they do. Flaky tests can significantly impact team productivity and engineering spending. A suite with a high flake rate isn't just unreliable, it also erodes trust in the pipeline and forces teams to spend time investigating false failures instead of shipping code.
How Do You Eliminate Flaky Tests for Good?
Eliminating flaky tests permanently requires fixing root causes rather than papering over symptoms. Here's the sequence that holds up across most test automation frameworks:
- Replace fixed waits with condition-based waits: wait for a specific element state or API response, not a fixed number of seconds.
- Isolate every test's data: generate unique identifiers per test run so tests never read or write shared records.
- Mock or stub unreliable external dependencies: only hit live third-party systems in a small set of true integration tests.
- Stabilize the test environment: pin dependency versions and standardize container startup across local, CI, and staging.
- Design for parallel execution from the start: assign isolated resources (ports, databases, sessions) per test worker instead of retrofitting isolation later.
- Build in self-healing test automation: smart locators that adapt to minor UI changes reduce the selector fragility that causes a large share of long-term flake.
This is where the difference between a tools-only approach and a managed one shows up. Generic automation tools can run a self-healing locator, but deciding which of the five root causes is actually at play, and fixing it correctly, takes testing experience. At MuukTest, Forward Deployed QAs diagnose which root cause is at play and decide how to fix it, while Amikoo, MuukLabs' AI testing tool, repairs broken tests and separates real product defects from test issues, so engineers stop chasing failures that were never bugs. When evaluating managed QA services, ask specifically how flaky tests are diagnosed, not just how often tests are re-run.
Frequently Asked Questions
What are flaky tests, in one sentence?
A flaky test passes and fails on the same code without any changes in between, usually due to timing, shared state, external dependencies, environment instability, or resource contention.
Are flaky tests a sign of bad code?
Not usually. Most flaky tests stem from the test's own design (timing assumptions, shared state) rather than a defect in the application under test.
Does adding retries fix flaky tests?
No. Retries hide the symptom and let the underlying race condition, state leak, or dependency issue persist, often making the problem harder to detect later.
Is Playwright less flaky than Selenium?
Playwright's auto-waiting and auto-retrying assertions reduce timing-related flakiness compared to Selenium's more manual wait model, but both remain susceptible to state, dependency, and environment-based flakiness if those aren't addressed separately.
What flake rate should a team aim for?
Below 1 to 2 percent is considered healthy. By the time a suite reaches 5 percent, teams stop trusting the test results, and above 10 percent is actively blocking release velocity.
Can self-healing test automation eliminate flakiness entirely?
Self-healing locators address selector fragility, one specific cause of flakiness, but they don't resolve shared state, external dependency, or environment instability issues on their own.
How does a managed QA service help with flaky tests compared to handling it in-house?
A managed QA service brings pattern recognition from working across many codebases and frameworks, which speeds up root-cause diagnosis. It extends what an in-house team can do without requiring additional hires solely focused on test maintenance.
About MuukTest
MuukTest is a managed QA service that builds and maintains automated test suites for lean engineering teams. The company builds frameworks on industry-standard tools like Playwright, Selenium, and Appium, and clients fully own the resulting automation framework. This approach lets teams reach broad test coverage in weeks while avoiding the exit costs that come with proprietary testing platforms. MuukTest works as a managed QA partnership that extends lean engineering teams, covering web, mobile, API, regression, and end-to-end testing without requiring additional hires. Behind the service are expert Forward Deployed QAs, who decide what to cover, which workflows matter, and where the real risk is, working with Amikoo, MuukLabs' AI tool for test generation, repair, maintenance, and defect detection.
Flaky tests rarely resolve themselves, and the longer they sit in a pipeline, the more trust and velocity they quietly erode. If your team is ready to see what a flake-free, fully managed test automation framework looks like, get in touch with MuukTest to talk through your current coverage and where the gaps are.
References
- Flaky Tests: How to Find, Fix, and Prevent Them (harness.io)
- The Ultimate Guide to Flaky Tests (trunk.io)
- What is a Flaky Test? Causes, Identification & Remediation | Datadog (datadoghq.com)
- Flaky Tests: Why They Happen and How to Fix Them | Pie (pie.inc)
- Best Practices for Identifying and Mitigating Flaky Tests - Semaphore (semaphore.io)
- Ultimate Guide to Fixing Flaky Tests | BuildPulse (buildpulse.io)
- Flaky Tests in Automation Strategies for Automated Testing (ranorex.com)
- Eliminate Flaky Tests (blog.testery.io)
