How Engineering Teams Using “Shape Up” Test Their Software

“Shape Up” is a new product development process that departs from popular processes like Scrum. The book, describing the philosophy and methodology, was written by Ryan Singer from his experience building the Basecamp product and promises a saner way to build software products.

Thousands of engineering teams at big and small companies are switching to Shape Up after frustration with their current product development methodology: Too much busywork, too much planning, too frequently stressed out and overworked building features that nobody cares about. The dominant methodologies treat software development as a factory assembly-line process; that might be fine in an actual factory, but it’s not the way to build software that customers love.

Shape Up, and Ryan’s next venture Shaping in Real Life, describe the development theory and process. QA, our domain and expertise area, isn’t discussed in Shape Up, though. Because we’re fans of Ryan, Basecamp, and Shape Up – and because many of our customers use Shape Up – we’ve spent the past few weeks exploring one question: “How does QA fit into Shape Up?”

In doing this, we’ve talked to small and large engineering teams - from our customer base and beyond. We hope you find this helpful if you are using, or considering, the Shape Up methodology.

Shape Up TL; DR

If you’re not familiar with Shape Up, we recommend reading:

1. The Shape Up book, which is free online (as well as anything else from Ryan Singer)
2. This quick primer on Shape Up from AgileFirst
3. Klaus Breyer’s writing for a deeper dive

Those are the formal definitions. Here’s how we talk about Shape Up internally and to founder friends:

It’s a different way of thinking about product development. Instead of trying to scope tons of tickets into sprints, Shape Up is designed around a 6-week development cycle where an engineer/small team can get a solid project done. There are nuances like “fixed time, variable scope” as well as specific “betting” and “shaping” approaches for prioritizing and designing solutions, which are also helpful.

Here’s how we’ve seen teams implementing Shape Up think about delivery and QA:

Shape Up & Delivery

First, how do teams use Shape Up when releasing software? You might think that, because developers are on 6-week cycles, they just release every 6 weeks.

You might also think that these teams’ DORA metrics are irrelevant - that they’re not prioritizing delivery performance because if they’re releasing once every six weeks, who cares if it takes a few days to get from code to production?

This was our first hunch - but we were very wrong.

More often developers are constantly pushing code throughout their development cycle. They push smaller amounts of code, daily or even multiple times per day. They are either doing true continuous delivery or close to it.

This means testing is key and has to be both fast and comprehensive.

We’ve broken out testing approaches in four different categories:

1. Unit testing
2. Manual testing
3. End-to-end testing
4. Bug-handling

1. How Teams Handle Unit Testing

Not surprising: Every company we talked to, developers are responsible for unit testing.

Earlier-stage startups with small development teams admitted their teams didn’t always cover their code with unit tests. Some larger teams admitted to this too.

But everyone agreed, in theory: Unit tests are the developers’ responsibility.

2. How Teams Handle Manual Testing

When the code is done, in theory, who’s responsible for playing with the new feature to make sure it works as intended?

This, too, is usually handled by development teams.

One CTO - Juani Villarejo from Nulinga - described a big and important feature that one of his teams was working on. Because of the feature’s importance, he budgeted a few weeks of testing into the feature’s appetite. 

However, we have seen examples of a QA sitting outside the product teams, whose sole responsibility is to manually test every new PR before production to try to find bugs. This makes the QA a single point of failure – and the more people who are frequently deploying, the harder this QA’s job is. The QA can’t automate end-to-end tests (see next) – and the second they go on a much-needed vacation, the team is in bad shape. When teams have manual QAs who exist outside the product teams, they typically view it as an intermediary step before they invest in or complete end-to-end test automation.

3. How Teams Handle End-to-end Testing

Every team using Shape Up we spoke to that had end-to-end test automation structured a separate team (or used a service like MuukTest). This was not part of their core product teams.

The goal of an end-to-end test automation team was to create and maintain a suite of automated tests that cover the majority of their software’s “happy paths” and core functionality. In our conversations, this test suite was integrated into their development lifecycle in one of three different ways:

1. Running daily in the background against a testing environment, but not a blocker to production. Teams receive bug reports daily. (See “bug-handling” section below)
2. Running in their test environment prior to deploying to give a “green light”, after developers manually test features, to make sure new features don’t break old features and core functionality.
3. Running in their CI/CD against every PR, often using an ephemeral environments solution like Shipyard

Most teams philosophically preferred to have test automation engineers in-house, but many had outsourced this role to a specialized / AI service like MuukTest after seeing how much work it was to automate and maintain a test automation suite. (Without this, teams needed to invest 15-20%+ of their engineering budgets into QA automation to get to full coverage; with MuukTest, they get a fully maintained test suite for half the cost of a QA engineer)

4. How Teams Handle Bugs

Juani (CTO of Nulinga) explained this best, reflecting how most teams approached bugs that emerged despite testing’s best efforts after the 6-week cycle was completed.

At Nulinga, one of the Shape Up teams is focused on “reactive” work during a 6-week cycle, which includes bugs and technical debt. In different cycles, different teams get this role. This is an elegant solution, but not ideal: 1/N of their engineering teams is always fixing bugs and technical debt, not building new things that add differentiated value. But hey - this is way better than *every* team scrambling half the time!

Summary

As we continue working with teams that have implemented Shape Up on their end-to-end QA automation, I’m sure we’ll learn more and share more. (We’re also moving towards Shape Up ourselves - more to come!)

If you have a different opinion or a story to share or just want to learn more about how MuukTest can help your team *avoid* spending a ton of time worrying about building and maintaining end-to-end tests… reach out!