Friday, Brent and I met (virtually) to record an episode of AB Testing. As usual, we didn't start with a topic, but I had an idea, and it was going pretty well - a better-than-average podcast. About 40 minutes in, Brent was ranting, and I interrupted him...
Stop.
STOP!
Given how much of a bad time I usually give Brent, he looked at me like I was about to tell him to stop fidgeting or playing with his microphone or ....
But that wasn't it - Brent was on a roll and telling a good story. A few seconds earlier I discovered that I forgot to press Record. After 201 episodes (plus a series of one off interviews), I finally really blew it. Rather than start over, we decided to scrap it and try again another day.
How It Started
Saturday (yesterday), I started writing a post about metrics. My very first conference presentation as about metrics, so I thought I’d dig up a few embarrassing slides to kick things off. I dug into my archived docs and found the presentation - but also at least thirty other papers and presentations that reminded me how much time I spent working on helping teams use metrics in my career. I’ve spent most of my career in tool development, but until I went down memory lane, I didn’t realize how much of a metrics story is buried in my history.
In fact, it was too much. That story needs to percolate for a while before I’ll know how to tell it, so it’s saved in the drafts folder for another day.
How It’s Going
The rest of Saturday was a wash with a rare social obligation - but this morning, I woke up still feeling bad about screwing up the podcast recording, and was wondering if we’d be able to reproduce the conversation the next time we get a chance to record. After about a second and a half of pondering, I came to the conclusion that neither Brent nor I can remember anything for more than a few minutes, and that the topic would either be lost forever - or that I could write about it a bit here this week.
First, it’s probably relevant to talk about the (lack of) rigor and planning that goes into an AB Testing Podcast recording session. When we first started, we’d have a “board" (literally a whiteboard in a room at Microsoft) where we’d list the topics for the day, order them, and discuss details. These days, we start the remote session, I ask Brent what he wants to talk about, he tells me he doesn’t have anything, and I tell him that I have an idea, and that we’ll figure it out as we go.
Remember - the podcast and this Substack are free. You get what you pay for.
The Pitch
The story starts with a question I get almost every time I write about developers doing testing (stemming from last week’s post).
That post started as a post about developers against the you-build-it-you-run-it DevOps philosophy, but while writing it, I realized that there was a parallel with developers who don’t wanna do testing.
The question I frequently get when I talk about developer testing is along the lines of, “A lot of testers write about testing. If developers are so good at testing, why don’t they write about it or share their test code”.
A Fair Question
It’s a good question, but the answer is filled with nuance. First (and I’ll come back to this later), testing is a part of software development. When I’m developing code and writing tests for that code, the tests help me identify design and functional issues almost immediately. They help me write better code. If I were to write an article or give a talk that used this product or code, I think it would be a better talk to talk about the outcome or the problem I solved rather than dive into the process that got me there. An exception could be if I was talking about how my product reached high levels of quality, I may talk about how I approached all aspects of development, including testing.
There are exceptions for sure. When I was asked to write a chapter for Dorothy Grahams Experiences in Test Automation, I wrote about Peter Shier - who wrote a device simulation framework for Windows that enabled (and accelerated) device driver development and testing across the Windows ecosystem. There’s also books like Developer Testing (and similar), and Martin Fowler has written a lot about good developer testing. Bryan Finster often writes about developer testing or quality. I would argue that books like Continuous Delivery, or The DevOps Handbook - while not directly about testing have enough anecdotes around testing and quality to help any developer learn more about testing.
Should there be more? Probably - but the stories are there if you look for them.
But, but, but
Some folks will say that developer testing isn’t testing, that it’s something else (those people are wrong). Or some that developers only do unit tests and those don’t cover much, or that there’s a lot of testing that developers can’t do (those people are wrong too). There’s a lot of research (as well as my anecdotal experience working on high-volume, high-availability software) that shows that developer owned automation is strongly correlated with product quality. Not just unit tests - all test automation.
It’s fair to note that the research - particularly Forsgren’s research in Accelerate goes on to say that testers still provide value beyond developer testing. While I don’t disagree with the research here, I also believe that today’s testers are better suited to helping developers do better testing (both automated and non-automated) - rather than hunker into a silo, hoarding testing knowledge and attempting to increase job security.
The Rub
Like design, architecture, or deployment, testing always has been part of what is done to build software. There was a time when it made sense to separate testing. In those times, software had to be right the first time.
I worked as a dedicated tester / test leader on a few different version of Windows. We had large beta groups, but inevitably after we shipped (and we ran into a whole lot of people using the software in ways we couldn’t anticipate), customers would report bugs. Some were so bad that we’d make a download available asap (or in the old-old days, ship floppies or CDs to customers). Eventually, we’d package up a few hundred of these and release a service pack. We moved slow, and having a separate test team wasn’t even close to being a bottleneck - it was an effective way to ship with the quality levels we wanted.
When I worked on Microsoft Teams (bad example, because Teams is horrible - but bear with me) - we made a big announcement about the beta, did a live webcast thing, and IT admins were excited. Welp - except their downloads didn’t work. It’s a long story about an edge case that we should have fixed ahead of time, but it was broken. Our benefit-of-the-doubt customers were online saying, “it must not be available in my region yet”. In the meantime, our developers fixed, tested, and deployed a change within an hour that unblocked everyone, with nobody (almost) knowing any better.
The point is that most software today doesn’t have to be right the first time - as long as you have a way to discover issues quickly - and then fix them quickly, nobody is going to know. At larger scale (just before I bailed from Microsoft), we would deploy to fractional audiences and roll back if issues were found. The absolute worst case was that 1% of our customers would see an issue that would go away with a refresh (most people ran Teams in a browser back then).
For the record, I use teams daily now, and I hate myself for bringing that product into the marketplace.
The Root
Some folks have said that my viewpoints are “harming” testing (note that I’m not innovating here at all - I’m describing what’s happening already across large parts of the software industry). The truth is, that despite my roots in testing, today I am 100% focused on product quality (and as a manager, by building teams that can build high quality products), but I don’t really care that much about testing.
Testing is one of those things that you do along the way to building a quality product. It’s weird, because I feel like there are folks out there so invested in testing that they value good testing of a bad product over building a good product.
It’s like a surgeon who cares more about how good their cuts and stitches look than the fact that they removed the wrong organ.
It’s weird.
The End
This may be the end of my articles primarily about (or adjacent to) testing for a while. As far as testing goes, I’m pretty adamant about 4 things.
Developer should write the vast majority of all test automation.
Developers should do as much non-automated testing as needed.
Dedicated testers, when on the team, exist to help developers at 1 and 2.
The goal is quality - not “good” testing.
And then, just to keep on top of the completely lost point - share your success stories.
-A 12:0
I think people conflate testing code with testing products. Sure, there's some overlap there, but the code really isn't a product until it reaches a benchmark of quality that customers will accept. Now figuring out who the customer is to understand what benchmarks to reach is a completely different problem, which feedback loops can help with. (All stuff you've talked about before.)