Agile Development Best Practices for Determining What to Automate

What can we automate in an agile environment?

Most types of testing benefit from automation, starting from basic unit tests through system tests. But, as you know, unit tests don't supply enough coverage to catch system regression failures. Running a set of manual tests before every check-in, which could be dozens of times a day, isn't practical. When developers can't run tests by pushing a button, they will not be motivated.

Let's talk about the kinds of tests that are most suitable for automation. Automation starts with an automated framework that allows developers to check their code often and receive quick feedback about its impact. So let's start with this first.

Continous Integration (CI) system

This is where we create the most important, albeit logical and straightforward, ground-rule for automation. Due to the nature of agile software development, which is faster than any other approach, agile teams should focus on automating any repetitive or tedious work involved in developing software.

And no candidate is better for this than automating build creation as part of an agile development process. Due to the fast nature of agile development, the team should create numerous builds per day, especially to test newly added code.

CI systems are crucial in an agile environment. Continuous integration and build processes are the two systems that give the most significant ROI of any automation effort:

• CI allows for immediate feedback at the unit-test level (if you have the relevant unit tests to support it).
• It reduces many of the risks involved in adding new code without testing it.
• It allows the team to create and deploy numerous (stable) builds and provides for multiple check-ins per day.
• It improves communication because team members can receive a notification once the build is ready without checking the status.
• CI systems speed up testing time by reducing the number of errors at the unit level before these errors become apparent in advanced phases of the testing process.

Based on the above, agile teams must implement continuous integration and build the framework as soon as possible. Although it requires continual maintenance, it's the only option for agile teams to succeed and reduce technical debt in large complex projects.

Based on these simple facts, you may see that agile teams must implement continuous integration and build the framework as soon as possible. Although it requires continual maintenance, it's the only option for Agile teams to succeed and reduce technical debt in large complex projects.

Decide what to automate, then decide what to run. CI suites have grown large enough that executing every automated test on every commit becomes its own bottleneck. Teams handle this with test impact analysis, which maps each test to the code it exercises and runs only the tests a given change can affect. The full suite still runs on a schedule or before a release. This widens the set of tests worth automating, because a slow but narrowly scoped test now runs only when the code it covers changes.

Development and Test Environments

Agile teams need to test and develop in a fast-changing environment; as a result, there is less room for the creation and maintenance of work environments. Agile teams can use the automated deployment of their environments without multiple hours of manual work. In addition, the team can use automation to handle many other areas related to their work environment:

• Creation and cleaning of the testing data and configuration.
• Setup of specific topologies and architectures.
• Simulating a particular scenario for reproducing a bug.

Testing of the User Interface (UI)

The agile development process embraces the approach that the team must deliver an incremental working functionality at the end of each iteration; as a result, the team will usually execute basic automated regression tests at the GUI level.

As I mentioned earlier, I'm a great believer in automated testing. Still, in some cases, we need to consider whether we want to use it, especially when we want to test the user interface of an application whose GUI changes.

To overcome the challenges of GUI testing, there is of great importance in selecting the most suitable tool for the job, one that's easy to maintain and flexible enough to absorb changes. This is probably the most critical key to successful GUI automation.

Testing all layers of the application

I'm a great believer in automated solutions that can reduce manual testing efforts to the bare minimum necessary. It starts at the first layer of the application by running unit tests that we all agree are crucial in reducing problems that won't become more significant problems later when found in that layer of testing.

Next, we have the second layer of component tests. Programmers test components as a whole module by using a set of inputs and outputs that the module produces. The third and, for me, the most crucial part of the testing strategy is integration tests, where modules are tested together as one suite. And if that is not enough, why not test the whole system by running the fourth layer of system tests, which test the entire application as an entire system.

Two more categories belong in the automated set whichever layer they sit in. Data-driven tests run one flow against a large table of inputs, and configuration tests run one flow against several browser and operating system combinations. Both multiply a single scripted path across many runs, both are impractical to repeat by hand, and both return their setup cost quickly.

Performance, Load, and Stress tests

Suppose you've ever been involved in the testing process that included one of the testing mentioned above types. In that case, you probably know that it's almost impossible and undoubtedly ineffective to use manual testing methods as the preferred way to run them. Furthermore, there is a wide range of tests that you cannot run without automation tools. In addition, using manual tests will not provide the accurate test results we can achieve by using dedicated automation tools that can simulate the exact scenario without any human interference that may affect the testing process and, therefore, the results.

What shouldn't we automate..?

If you are familiar with how I approach testing, you are probably already aware of my strong support for automation testing frameworks that enable the team to develop a more effective, efficient, and dependable coding and testing process. However, some parts of the testing procedure still require the intelligence, common sense, and eyesight of humans.

Localization testing is one of those parts. Checking a translated interface means reading the wording in context and judging whether it fits the region, the layout, and the reading direction, which an assertion on a string value cannot do. Automate the mechanical half, meaning that every string resolves and nothing overflows its container, and leave the language judgment to a person who speaks it.

Usability Testing

Usability testing is very different from other test types that determine the quality of the software. It cannot be automated because it requires someone to work with the software to determine how it was experienced and where the gaps are in the user experience.

GUI Testing (Is it worth the ROI?)

GUI testing is one of the most challenging areas to automate. I've seen just too many organizations that invested thousands of human hours in automating the GUI of their products but, in the end, found it was a waste of time that didn't provide the expected ROI. Some GUI tests can be used to ensure there are no unexpected changes in the GUI. Still, you should ask yourself whether it's worth the costs and investment instead of improving other quality issues that will provide a better ROI and reduce the risks in that area.

Self-healing locators have changed part of this calculation. Many automation tools now match an element on several attributes and repair the selector when the primary attribute stops matching, which removes a share of the routine maintenance that made GUI suites expensive. The cost that remains is the one that decides the ROI. A reworked workflow, a new checkout step, or a redesigned form still needs a person to decide what the test should assert now. Self-healing lowers upkeep. It is not a reason to automate a screen that changes every sprint.

Tests that are not worth the investment

I used to be involved in an automation project where the test team automated a thousand provided tests (at least on paper). So what is wrong here? They automated almost all the available test scenarios without really thinking about the ROI.

The team invested weeks in automating tests marked as low risk, tests that would never fail, and tests whose failure had a meager chance of impacting the software. The entire automation process was based on the spirit of "let's automate everything" instead of asking the simple question of the ROI that this automation project provides. Running the numbers first with this test automation ROI calculator would have flagged those low value tests before a single hour was spent.

In some cases, some tests are written without real thought as to whether they are essential or not. Once the automation project starts, the team will automate these tests just because they never want to rerun them (because they know there is a 0% chance that it will make a difference).

Tests that need to be executed only once

The main goal of automated testing is to allow the team to focus on essential things in the software development lifecycle. Automating test scenarios that will run only once are not worth the team's time to invest in the design, creation, and execution of these tests.

Exploratory Testing

In my opinion, exploratory testing is the best and most efficient method that agile teams use in any testing process. Exploratory testing can be used for learning purposes (you learn more about the software when testing it) or to provide a fast way to evaluate the overall quality of the software. However, when it comes to real testing effort, a skilled tester must design and execute tests.

Exploratory testing should be done by humans and not by automated scripts because automated scripts will not let the tester take in new information he generated from the exploratory session and use it to improve future testing and development processes.

In addition, although exploratory testing is a great testing approach, there is a real need for automated tests that will allow the team to focus on their exploratory sessions without worrying about any regressions that automated tests should cover.

How do you decide which tests to automate first?

Score every candidate test on five factors before you write it: run frequency, feature stability, business criticality, reusability, and the manual effort it currently costs. A common form of this is a test case selection matrix that rates each factor from 0 to 1 and treats a combined score of 3.5 or higher as the point at which a test earns automation. The threshold itself matters less than the habit of producing one. A score makes the team state why a test belongs in the suite, which is the question the thousand-test project described earlier never asked.

One factor works as a veto rather than a score. Ranorex lists tests with unpredictable results, meaning tests with no clear pass or fail criteria, among the cases you should not automate at all. A test that fails at random costs more than it saves, because someone has to triage every red run to work out whether the build broke or the test did. If you cannot state the expected result as a fixed value before the run, keep that check manual until you can.

Weight the score differently in 2026, because authoring is no longer the expensive half of the job. Playwright now ships three test agents: a planner that explores the application and produces a Markdown test plan, a generator that turns that plan into executable test files, and a healer that runs the suite and repairs failing tests. Once a tool drafts the first version, low authoring effort stops being a reason to automate anything. Stability, determinism and maintenance cost carry the decision instead, because those are the parts an agent does not remove. A generated test against a screen that is redesigned every sprint is still the wrong test to own.

Key Takeaway: Rank automation candidates by run frequency, feature stability, business criticality, reusability, and manual effort, and weight stability and maintenance cost highest now that test-generating agents can draft the first version of a test.

Can AI agents decide what to automate for you?

No. AI agents draft and repair tests, but they do not decide which tests are worth owning. That decision still rests on run frequency, stability, and maintenance cost. The agents shorten the authoring step, and they leave the selection step exactly where it was. Read the documented limits of each tool before you let one widen your suite.

Playwright states in its own agent documentation that the healer may skip a test when it believes the underlying functionality is broken, and that the generator can produce initial errors the healer then repairs. Both behaviors are reasonable inside a repair loop and both are a problem when nobody reads the result, because a skipped test is a coverage gap that reports as a clean run. GitHub sets the same boundary for unit tests, stating that the tests Copilot generates may not cover all scenarios and that you should always review the generated code and add any tests that are missing.

So the five selection factors hold, with one shift in weight. Authoring effort used to be a real cost and is now a small one, which removes the argument for skipping a valuable test because writing it would take a week. Determinism and maintenance cost carry more of the decision instead. An agent that keeps repairing a test on a screen redesigned every sprint is maintaining a check that should never have entered the suite. Point the agents at the stable, high-frequency, business-critical paths you already chose to automate, and put a human review gate in front of any generated test before it joins the suite.

Key Takeaway: AI agents cut the cost of writing and repairing tests but not the cost of choosing them, so review every generated test and keep determinism and maintenance cost at the top of the selection criteria.