Almost exactly fifteen years ago, I wrapped up the final words on How We Test Software at Microsoft. The final chapter I wrote was the chapter on Model Based Testing (more on that later), but it was a long path to get there.
Freakin’ Ken
For a few years of my career, I worked in a central group at Microsoft that created and delivered training and did a chunk of work and consulting with leadership teams across the company. On some forgotten morning in 2007(ish), my manager, Ken (who was one of the better managers I had at Microsoft) poked his head into our team room and said, “You should write a book about testing at Microsoft”. The notion was that I, along with Ken and my colleagues had a pretty good idea of how testing worked across the company, and that the book would cover a lot of things that we were teaching already. Furthermore, Ken said that I could write the book as part of my job (i.e. he’d lower my obligations in order to give me time to write), and that he and BJ could also write some of it. Easy.
My first clue to the next year came when I wrote the entire proposal (for Microsoft Press) by myself. But - I turned it in, and a few weeks and a few conversations later, we had a book deal. There was an option in the contract to specify whether we were writing the book independently (and getting paid), or writing it on “Microsoft time”, and not getting paid. Because Ken was having me write the book on company time, we chose the second option.
Of course, before I could finish a single chapter, Ken left for another job, my new manager (nicknamed Nimrod) didn’t hold up the deal, and I spent nights and weekends for the next year writing.
Freakin’ Ken
Before we start…
The good news is that Ken and BJ did participate and help out. Not that I’m counting, but I wrote about 270 pages of the book, while Ken and BJ wrote about 70 each. At least we weren’t splitting royalties. I should say that I’m not bitter about the book revenue, the book opened some doors for me, but especially opened the doors for thousands of potential msft testers who were sent copies of the book in order to prepare for their interviews.
For the most part, the book is a snapshot in time. Some of it did not age well, while some bits did slightly better. I’ve been asked by several people to talk more about why, and given we’re pretty much on top of the fifteen year anniversary of finishing the book, it seems worth of a post here.
The Guide
The book is broken into four parts:
About Microsoft
About Testing
Tools and Systems
About the Future
About Microsoft
The first three chapters (two written by Ken) are about how we approached software engineering and software testing in 2007-2008. None of it is true today as far as I can tell, but if you want a bit of history, you can glance through those first three chapters.
Readers Guide suggestion: Skim
About Testing
This part is tough. The section is accurate, but I don’t think is a very good guide on how to do testing in 2023. But - I also know that there are companies who today, test just like Microsoft did in 2008, and there may be some value for those companies.
A few things worth calling out.
The chapter on Test Design (mine) is a pretty shallow look at test design. The information is there, but in hindsight, needs more meat. I think that’s good for an overview, but not so great for those who want to dig in more deeply. A prominent industry figure has mocked me a few times for only writing a page or so on exploratory testing at Microsoft, but I think I was shallow on all of test design.
On the other hand, BJ wrote the next two in-depth chapters on Functional and Structural testing. BJ read a lot of the early books on testing, and designed most of our course for new testers at Microsoft where we covered these topics. In hindsight, these chapters may go too deep. There’s a bit of chapter 5 that sums up one of my main complaints about the book. I just did not have time to edit and give feedback that could have given the book a more consistent voice. This example of “BJ as a pseudo professor” stands out (italics mine)
…The ECP tests are then derived from creating unions of valid class subsets until all valid class subsets have been used in a test, and then evaluating each invalid data subset individually.
This seems like a simple enough explanation…
No, BJ, that’s not simple.
Readers Guide Suggestion (chapters 4-6): Skim
The next two chapters (mine) are about Code Complexity and Model Based Testing. Today, I think there is some value in understanding code complexity, but it doesn’t need a chapter in a book.
Readers Guide Suggestion (chapter 7): Skip
And that gets us to the last chapter I delivered to the publisher. I re-read the chapter this morning, and I still think it’s a nice intro to Model Based Testing. One thing I would push if I re-wrote it would be to triple down on the fact that MBT is super cool and awesome when you’re testing stateful applications. For a while, some people wanted to use MBT everywhere and were mad when it didn’t work. I can tell you that a few years later, Model-Based testing was crucial for testing protocols as part of the protocol documentation that msft was required to do.
Readers Guide Suggestion (chapter 8): Read (if you are curious at all about MBT)
Tools and Systems
Someone once told me that the section on bug tracking taught them how to write good bug reports. Today, I’d prefer not to use a bug tracking system, so I’m not sure how I feel about that. This section is again, a snapshot of tools used in 2007-2008, and is largely skimmable at best.
One chapter to call out is Ken’s chapter on testing software services (chapter 14). Microsoft was fairly new to testing services back then, and even Ken would admit that this chapter did not hold up. Don’t read it - don’t even skim it. Just pretend it doesn’t exist.
I also wouldn’t read the chapter on Test Automation. It’s dated, and it definitely doesn’t hold up.
The rest of this section is definitely still worth thumbing through. Remember, at the time, Microsoft had nearly 10,000 testers across the company, and those testers had a lot of ideas and tools that helped (or didn’t) with testing. We captured some of the more fruitful ideas and tools from this time, and a few of those approaches still hold up. Almost.
Readers Guide Suggestion (chapters 8-14): Skip Automation (chapter 10) and Services (chapter 14) - Skim the rest
About the Future
Predicting things is hard, especially about the future.
The final three chapters are me wondering what the future may be like at Microsoft. I was wrong. I was super wrong, but I don’t regret guessing. There were a few things I saw that I thought would blossom, and in re-reading (or re-skimming) the final chapters, I can see the seeds of some ideas I still hold today (learning and growth through community). I think there’s something here, even if it’s not accurate
Readers Guide Suggestion (chapters 15-16): Skim
The Summary
For those of you who skimmed the article about skimming HWTSAM, that’s my 2023 Reader’s Guide. In summary, read the chapter on Model Based Testing if you care at all about Model Based testing. Skip the chapters on Automation (my chapter in Beautiful Testing is a much better read) and Testing Services, and feel free to thumb through the rest if you want.
-A