What Agile Testing Is (Not)

Testing faster

One thing is to be as time-effective as possible when testing a software product. The completely different thing is to be forced to rush testing something driven by unreasonable deadlines.

Continuous delivery turns this pressure into a constant. When a pipeline can ship to production several times a day, teams shrink testing to whatever fits inside the build window. A pipeline of fast checks confirms that the code compiles and that the main path still works. It says nothing about the behavior nobody thought to specify, and that is usually the behavior the customer finds first.

Even worse if what is expected from the tester(s) is just to confirm (as soon as possible, of course) that everything is ok.

So, to me Agile Testing has nothing to do with fast and shallow validations or with (quickly/dangerously) confirming (potentially wrong) assumptions. This would rather be reckless and meaningless testing.

So, as I see it, good testing takes (reasonable) time and efficient testers. On the contrary, even though bad/inefficient testing may apparently take less time, you should be warned that you'll eventually realize that it was actually bad/inefficient and basically meaningless sooner or later.

Key Takeaway: Agile testing takes reasonable time and skilled testers, because rushing tests to meet an unreasonable deadline produces shallow confirmation of assumptions rather than real information about the product.

Skipping testing

Well, you should be able to foresee the risk of doing that. If you are not, chances are you are living under the illusion that:

• Your team is so good that they are producing perfect software(3) or
• Your customers are so easy-going that they are not going to notice that your software is actually pretty far from perfect.

Let me stress that I've never happened to test any software product which was so spotless it didn't actually need to be tested at all. Basically because such a product does not exist. And I'm pretty sure you too are aware of that.

After all, living in denial doesn't solve anything.

Key Takeaway: Skipping testing does not remove risk, because no software product is perfect and customers will find the defects the team chose not to look for.

Testing too late

According to my experience, starting to test a product at the very late stage of the software development process is unfortunately still very common, especially when organizations believe that developing/delivering something is much more important than challenging their assumptions about it.

So, at some point, someone might have an idea which implies building a software product (hopefully, in order to solve a problem rather than to create another one) and they might start either doing that themselves or hiring a team able to do that.

Meanwhile, nobody is challenging the idea itself, the software architecture or the code that is being produced to implement it.

So, after a minimum viable product (which, by the way, too often is not viable at all) has taken shape, someone realizes that some glitches may be hindering the process.

So, one tester (usually misnamed as a QA(4)) is urgently brought into the team to validate the software product and to check (and assure the stakeholders(5)) that everything is ok.

I wonder, though, would you need a tester if something was ok?

The thing is, challenging assumptions at this point is usually not well accepted: it's actually something that only particularly experienced and brave testers would dare to do.

After all, what most managers want to hear now, without further ado, is just that everything is ok.

Which means the organization in question may end up:

• Delivering a low-quality product without knowing that it is a low-quality product (their customers definitely will, though),
• Delivering a low-quality product in spite of knowing that it is a low-quality product (while keeping their fingers crossed and hoping that their customer will not realize that),
• Not delivering at all what they initially thought was a pretty good product but turned out to be a dangerous and embarrassing minefield(6).

Wouldn't it have been better to start testing earlier? Ideally at the same time other activities (coding, business analysis, etc.) were being performed(7).

The industry now has a name for moving that work forward: shift-left testing. The label describes pulling test design, acceptance criteria, and the awkward questions about a requirement into the same period as the coding, instead of queueing them behind it. The name matters less than the practice. A tester who reads a user story before anyone writes code can find a contradiction in the requirement, and that costs a conversation. The same tester reading it after release finds a defect, and that costs a release.

Key Takeaway: Starting testing only after a minimum viable product is built means nobody challenged the idea, the architecture, or the code in time, so the organization ships a low-quality product or abandons it late.

Making unskilled people test

This is probably the most controversial point to cover, due to the unfortunately pretty common trend/misunderstanding that testing should be everybody's responsibility(8).

After more than fifteen years working with software, and more than ten as a tester, I still struggle to understand what makes some people believe that software testing is so different from all the other things that might be needed to implement a software product (business analysis, software architecture and design, coding, etc.), that it can be performed by anybody.

I wonder, though, if under no circumstances would you like to make unskilled people code, why should you want unskilled people to test?

You may think lack of awareness about software testing could be one of the reasons behind that.

Nevertheless, lack of knowledge about coding doesn't usually make people believe that anybody can code: actually, quite the opposite.

So, why underestimating software testing is so well-spread seems to me a really challenging conundrum to solve.

Throw the usual misconceptions about Agile into the mix and you'll have a recipe for disaster: people saying that, in order to be more efficient at testing, all you need to do is to automate (the execution of) your test cases, will deliver the finishing blow to your software development process.

The same argument now arrives in a newer form. Teams point at AI coding assistants and generated test suites and conclude that testing no longer needs a skilled person. Those tools write test code quickly and they are useful for scaffolding, but they derive their assertions from the same specification the developer already read, so they repeat the original assumptions instead of challenging them. Deciding what is worth testing, what a correct result looks like, and which failures would hurt the business still needs a tester.

What's more, strange as it may sound, fallacies about such an unfortunate discipline are often spread not only outside the software testing community (where we can assume there might be a somehow reasonable lack of knowledge) but also inside the community itself, which seems to be plagued by a dangerous mixture of Dunning-Kruger effect and a lot of mythological Trojan horses.

The thing is, making unskilled people test usually means settling for confirmatory testing, that is to say, demonstrating that a software product does what it is expected to do, without caring at all about exploring a system to learn how it actually works or investigating under which circumstances it might fail. And yes, this is what software testing really is after all, isn't it?

On the other hand, it has to be noticed that even a usually skilled person might be unskilled at some point: if they built the software product themselves, for example, they will most likely lack the critical distance required to deeply and unbiasedly test the product itself, which is what you need when you really want to uncover unanticipated problems, don't you?

Key Takeaway: Software testing needs a skilled tester, because unskilled testers and AI-generated test suites both repeat the assumptions already written into the specification instead of challenging them.

The agile testing quadrants

The four points above say what to avoid. The quadrants say what to cover. Brian Marick drew the original testing matrix, and Lisa Crispin and Janet Gregory adapted it into the quadrants that anchor their book Agile Testing. The model sorts testing along two axes: business-facing against technology-facing, and tests that guide development against tests that critique the product.

Quadrant 1 holds technology-facing work that supports programming, such as unit tests and test-driven development. Quadrant 2 holds business-facing work that supports programming, such as the examples and acceptance testing that become a specification before anyone writes code. Quadrant 3 holds business-facing work that critiques the product, and exploratory testing sits here. Quadrant 4 holds technology-facing work that critiques the product, including performance, load and security testing. Crispin points out that Quadrant 3 and Quadrant 4 testing require code that is written and deployable, which is why that work cannot be pulled forward the way Quadrant 1 and Quadrant 2 work can.

The map is useful here because each misconception deletes a quadrant. Rushing to fit a build window keeps the left half and drops the right. Skipping testing drops everything past Quadrant 1. Bringing a tester in after the minimum viable product removes Quadrant 2, because the examples that should have shaped the specification now arrive after the code. Handing testing to someone unskilled leaves the confirmatory checks in place and quietly removes Quadrant 3, since exploratory work depends on the judgment of the person doing it. A team that can name the quadrant it is missing has a far more specific problem to fix than a team that only knows its testing is not working.

Key Takeaway: The agile testing quadrants sort testing into technology-facing and business-facing work that either supports programming or critiques the product, so a team that can name the quadrant it is missing has a specific problem to fix.

Where AI actually helps in agile testing

AI now sits inside the daily work of most agile teams, so it is worth being precise about which part of testing it touches. Google's DORA research surveyed close to 5,000 technology professionals and found that 90 percent of respondents use AI at work while 30 percent report little or no trust in the code it generates. Those two figures describe the position well. The tools are everywhere, and their output still has to be checked by someone.

What AI handles well is the mechanical part of the work. GitHub Copilot and comparable assistants scaffold test classes, fill in fixtures, and draft selectors faster than a person can type them. Playwright ships a codegen recorder that turns a browsing session into a runnable script. Tools that advertise self-healing locators re-point a selector when the markup around it changes, which lowers the maintenance cost of a large suite.

What AI does not do is decide what is worth testing. A generated test takes its assertions from the specification or from the code itself, so it encodes the assumption that the current behavior is correct. If the requirement was wrong, the generated test confirms the wrong requirement. In quadrant terms that is Quadrant 1 work and part of Quadrant 2: useful, but it supports programming rather than critiques the product. Quadrant 3 exploratory testing depends on a person forming a hypothesis about how a system might fail, and no current tool forms that hypothesis for you.

So AI amplifies whatever testing practice a team already has. A team with a weak practice that adds AI gets shallow tests faster, which is the first misconception in this article wearing new clothes.

Key Takeaway: AI speeds up writing and maintaining tests, but it derives its assertions from the specification it was given, so it cannot decide what is worth testing or form the failure hypotheses that exploratory testing depends on.

Wrapping Up

Wrapping up, through this piece of writing, I made it clear that, to me, Agile Testing has nothing to do with unreasonably fast, meaningless, reckless, late, shallow or ineffective testing (or, maybe even worse, with no testing at all). So, if you think Agile Testing doesn't work for you or it is not providing you with the results you were hoping for, well, chances are you might just be getting it wrong.

Unsatisfied/Disappointed with this article? Bear with me: after talking/ranting about what Agile Testing is not, I'll go through what it really means to me, and how you can make it work, in my next blog.

Footnotes

1- I promise I'll talk about what I believe Agile Testing actually is in my next piece of writing. 2- It might be worth mentioning that, more in general, Agile is not about coding faster either. 3- I was lucky enough to work for almost three years with whom I still believe were the best programmers I've ever known. Even them made mistakes every now and then. The thing is, they were humble enough to acknowledge that and to appreciate the job of the testers within the team. 4- I may end up talking about this topic on another occasion. 5- How can someone possibly do that? 6- This might actually happen only if someone dares to spell out the truth and someone else is willing to listen to it and to accept it. 7- Yes, this is possible. I'll talk about it in my next blog. 8- I do believe that the quality of one's own work should be everybody's responsibility, but this is actually a pretty different thing.