Testing at the Speed of AI

teams generate and ship code faster than ever, and testing needs to keep up

Most folks know that I started my tech career a few decades ago in software testing - and probably also know my opinions on test automation (some documented in The A Word - summary here - full book here).

The short version is simple. Our industry has had a long and unhealthy infatuation with UI-based automated testing.

A lot of web applications are easiest to test through the user interface. The customer experience is almost always through that UI. Despite that, it’s extremely difficult to get a strong return on investment from large UI automation suites - especially when development moves rapidly.

Maintenance costs are high. The tests are brittle. The signal is often weaker than people expect.

None of this is new.

The Struggle

The pressure on testing has been building for years.

When Brent Jensen and I started talking about the Modern Testing Principles, it was because delivery was already accelerating. Agile practices shortened release cycles. Teams shipped more frequently. Test cycles shrank while expectations kept growing.

For years we heard about “shift left.” In practice, many teams didn’t actually change their testing strategy. They compressed the timeline instead. Testing still happened late. It just happened faster. The tooling improved. The mental model didn’t always follow.

Then AI arrived and stepped on the gas again.

Teams that were barely keeping up with Agile delivery started producing far more code, far faster. We can now generate more code in a week than many teams used to write in an entire quarter.

That changes the pressure on testing in a fundamental way. Not because testers are slow, but because the volume curve bent sharply upward.

Layout changes and new functionality that used to happen weekly now happen daily. Teams that were deploying daily a year ago are now deploying dozens of times a day.

Testing didn’t suddenly get harder. The system around it changed.

Testing in the Age of AI

AI-generated code increases output, but it doesn’t automatically increase confidence. We still need to verify customer paths. We still need to find regressions, edge cases, and unexpected behavior. In many ways, the need for testing has only increased.

But the traditional model of test automation wasn’t built for this world. It assumes stable selectors. Predictable flows. Carefully scripted scenarios. It assumes teams have time to maintain large suites of brittle tests.

That model has been cracking for years.

Now it’s breaking.

Some teams are still operating with a mindset from the late 90s. Write scripts. Maintain scripts, and hope they don’t break when the UI changes.

A Different Model

What's needed to meet this moment goes beyond traditional quality assurance. We need application integrity - continuous, measurable assurance that your software just works as intended - with governance to operate at AI speed and scale.

What’s emerging now is fully autonomous testing. I’ve been looking closely at how SmartBear is approaching this space. They’re using agentic AI and prompts to test web applications in a way that adapts, explores, and learns.

The important point is not that it’s “AI.” The important point is that it changes the economics of testing. Good tools don’t replace testers. Good tools amplify them. Instead of spending hours maintaining scripts, testers can focus on intent. On risk. On scenarios that matter. Instead of fighting the tools, they direct them.

BearQ

Over the past several weeks, I’ve had the opportunity to experiment with SmartBear’s new product, BearQ.

Plenty of testing tools just bolt on GenAI. This is different. It's an AI-native agentic QA system.

What I like about BearQ is how it shifts the workflow. Instead of building fragile scripts, you guide the testing using prompts and intent. The tool explores the application, adapts to layout changes, and generates useful testing activity without requiring constant maintenance.

With BearQ, I just generate ideas and let the agents do the work. For example, for a test site I created, I just opened a chat with the “QA Lead”, and gave it the following.

The QA Lead parses the instructions, and hands the task off to an Exploration Agent that runs while I go make breakfast and coffee. The QA Lead handed off these instructions to the Exploration Agent.

Before I finished my pour over coffee, the test was done with this summary.

Perhaps my site is too buggy, but for a quick test, I was pretty happy with the quick results.

Next, I wanted to extend the exploratory nature of these autonomous agents. I gave the QA Lead this prompt.

Note - if you don’t know about the list of Big Naughty Strings (and yes, I see the typo in the instructions above, but AI is smart enough to deal with typos), it’s a great resource for input testing.

I had just finished my first sips of delicious Guatemalan coffee when this test finished with the following report.

This report is accurate, but not as actionable as I’d like, so I created a follow up task requesting specific errors and screenshots. As a manager of an autonomous test team, I’m happy to say that my team rocks. Ten seconds later I had this report.

As you’d expect, BearQ can generate reports, schedule tests, and even find API endpoints while testing for further examination or exploration. Overall, it’s a pretty damn good test team, and I’ve only scratched the surface of what it can do.

As with all AI tools, BearQ isn’t replacing testers, but it’s certainly enhancing their ability to examine applications autonomously while we do other testing (or drink coffee).

What’s Next

Software delivery isn’t slowing down. AI is making that super-clear. Teams are generating more code, making more changes, and shipping more frequently than at any point in our industry’s history. Testing has to evolve along with it.

For a long time, automation meant writing scripts and maintaining them forever. That model made sense when change was slower.

But today the problem isn’t just coverage, it’s adaptability. Tools that can explore applications, understand intent, and help testers focus on risk instead of maintenance will fundamentally change what testing looks like.

That’s why I’m paying attention to tools like BearQ. It isn’t just another AI tool. It’s a tool that helps testers to keep up with the accelerated system of software delivery.

AI is changing the economics of software delivery. When the economics change, practices and tools will inevitably follow.

---

* Note. I’m writing this article in partnership with the awesome folks at SmartBear.